Migrate from Offline Integration to a workspace

Step 1: Write or migrate your Spark SQL

The core migration task is to adapt your existing Offline Integration SQL logic to the AnalyticDB Spark syntax. For more information about how to develop and test SQL, see Build an open data lakehouse analysis solution with AnalyticDB Spark.

Step 2: Manually run tasks in the workspace

For one-time tasks or tasks that need to be triggered manually, follow these steps:

1. You can create a new Notebook file and execute the relevant code.

   You can create a Notebook file using one of the following two methods.

   Create a file manually

   1. In the workspace, click (Resource Manager) on the right.

   2. In the CODE area, right-click an empty area and select New Notebook File.

   3. Enter a file name and click OK.

   Import a file

   1. In the workspace, click (Resource Manager) on the right.

   2. In the CODE area, right-click a blank space and select Upload File.

   3. Enter the file name and click OK.

   Note

   After you create a Notebook file, if the file does not appear in the workspace, click to refresh the resource manager.

2. Double-click the file name to open the code page and develop the job.

3. Set Language to SQL, enter the code in the cell, and then click the Execute button.

   If the message "Run failed. You need to create and attach a Notebook session resource first." appears, click Create Session.

4. Create a Notebook session.

   1. Click Create Session and configure the following parameters.

      Parameter	Description
      Session Name	Enter a custom session name.
      Cluster	Select the destination cluster type. If you only need to use a CPU for execution, select the default DMS CPU cluster. This cluster is created automatically when you create the workspace. To develop Spark jobs, select a Spark cluster. You must create this cluster manually. In the Cluster drop-down list, you can click Create Cluster and select Create Spark cluster.
      Image	The page displays the corresponding image specifications based on the cluster type you select. Spark3.5_Scala2.12_Python3.9:1.0.9 Spark3.3_Scala2.12_Python3.9:1.0.9 Spark3.5_Scala2.12_Python3.13:1.0.9 Spark3.5_Scala2.12_Python3.11:1.0.9
      Specifications	The resource specifications for the driver. 1 core 4 GB 2 cores 8 GB 4 cores 16 GB 8 cores 32 GB 16 cores 64 GB
      Configuration	The profile resource. Edit the profile name, resource release duration, data storage location, PyPI package management, and environment variables. Note Resource release duration: Resources are automatically released when they are idle for longer than the specified duration. If you set the duration to 0, the resources are never automatically released.

   2. Click Finish and Create.

      The session creation is successful when the status changes to Running.

      Creating a session for the first time takes about 5 minutes. Subsequent creations or restarts take about 1 minute.

5. Run the code again.

   You can view the data objects created by the Spark SQL cell in the Data Catalog area.

Step 3: (Optional) Configure scheduled tasks

To schedule tasks to run automatically, use Airflow.

1. Learn how to build a complete data development and scheduling job with DMS Airflow and AnalyticDB Spark. For more information, see Build a data lakehouse workflow with AnalyticDB Spark and DMS Airflow.

2. In the Airflow DAG, use the DMSNotebookOperator to call the Notebook file that you created in Step 2. For more information about how to use the operator, see DMSNotebookOperator.