DocsICP FilingConsole
Documentation
Home Page DataWorks User Guide Use cases Best practices Isolating development and production environments

Isolating development and production environments

Updated at:
Copy as MD
Product
My Favorites

This topic demonstrates how to isolate development and production environments using DataWorks, EMR, OSS, and DLF.

Background

Many organizations require isolated environments—such as development, testing, and production—for their big data development lifecycle. This practice involves separating physical data storage, compute clusters, and big data job scripts for each environment. It also enforces strict access control, where only operations teams can access the production environment, while developers are limited to the development environment.

This example uses the following services:

  • DataWorks manages the development, operations, and scheduling of big data jobs.

  • This example uses two EMR clusters: one for the development environment and one for the production environment.

  • OSS stores the data.

  • DLF stores and manages metadata.

Environment isolation in DLF

  1. In DLF, create two Data Catalog, one for development environment metadata (dev catalog) and one for production environment metadata (prod catalog), and set the Location to different OSS paths.

    For more information, see Data Catalog.

    On the Metadata Management page of the DLF console, click the Data Catalog tab to view the created catalogs. In this example, the dev catalog path is oss://emr-dlf-dfs/ and the prod catalog path is oss://oss-mc-test1/.

  2. In each catalog, create a database. To simplify future migration tasks, use the same database names in both the development and production catalogs but point them to different OSS paths.

    On the Create Database page, set Data Catalog to dev, enter a Database Name and Description. The default Path is oss://emr-dlf-dfs/. Click OK.

Environment isolation for EMR clusters

Configure the catalog settings for each engine in both EMR clusters. Ensure that engines in the development EMR cluster use the development catalog (dev catalog), and engines in the production EMR cluster use the production catalog (prod catalog).

For example, for the development cluster, modify the Hive engine's dlf.catalog.id parameter to point to the development catalog (dev catalog). For more information, see Manage configuration items.

On the Configurations tab of the Hive service in the EMR cluster, search for the dlf.catalog.id parameter and set its value to dev.

Important

  • The preceding example is for the Hive engine. Apply similar changes to all engine types in both EMR clusters.

  • After you modify the engine configurations, deploy the changes and restart the components to apply the new settings.

Environment isolation in DataWorks

Workspaces in basic mode

  1. Create two DataWorks workspaces in basic mode: one for the development environment, associated with the development EMR cluster, and the other for the production environment, associated with the production EMR cluster. For more information, see Create a workspace.

  2. In the development workspace, use Data Studio to develop tasks, configure scheduling properties, and create tables with SQL commands.

    Example CREATE TABLE statement:

    CREATE TABLE if NOT EXISTS db1.table1 ( id int, name String);
    Note

    You do not need to specify a storage path because it is already defined for the database in DLF.

  3. Use the cross-project cloning feature to promote the workflow from the development environment to the production workspace.

    In the top navigation bar, click the Cross-project Cloning tab. Select the workflow to clone and configure settings such as compute engine mappings and resource groups. For more information, see Deployment Center overview. After cloning, the corresponding tasks appear in the production workspace. You can then modify, validate, and deploy them according to your production requirements.

Workspaces in standard mode

In a standard mode workspace, you can use workspace-level parameters to select different EMR databases for the development and production environments.

  1. You can assign different database names for development and production to a workspace-level parameter. When you use this parameter in your task code, the task accesses the correct EMR database depending on the environment in which it runs.

    1. Define a workspace-level parameter.

      1. Log on to the DataWorks console. After selecting a region, in the left-side navigation pane, click Data Development and O&M > Operation Center. From the drop-down list, select the target workspace and click Operation Center.

      2. In the left navigation bar, click Scheduling Settings. Click Workspace Parameters, and then click the Create Workspace-level Parameter button on the right. In the Create Workspace-level Parameter dialog box, enter the following parameter information:

        Parameter

        Example

        Parameter Name

        Example: emr_env. This name is customizable.

        Owner

        Select a user from the drop-down list.

        Workspace

        Select your workspace.

        Parameter Type

        Select Constant (Plaintext).

        Parameter Value (Development Environment)

        emr_dev (the development database name)

        Parameter Value (Production Environment)

        emr_prod (the production database name)

        Description

        Enter a custom description.

    2. You need to use code variables to assign values to workspace-level parameters to access different EMR libraries for development and production. In Data Studio, when you develop a task and configure its properties, click Scheduling Settings on the right side of the task and configure the name of the workspace-level parameter in the Parameter Value field.

      In this example, the parameter name parsed from the code is emr_db. Set its Parameter Value to ${workspace.emr_env}.

    3. When the task is deployed, the variable is automatically assigned the value emr_dev in the development environment and emr_prod in the production environment.

Previous：Data modelingNext：Videos