Configure a Databricks data source

Step 1: Prepare the Databricks data source

1. Prepare a Databricks personal access token.

   1. Create a personal access token. For more information, see Databricks personal access token authentication.

   2. Configure permissions. The personal access token for the data migration must have at least the following permissions on the catalogs and schemas that you want to migrate:

      1. BROWSE

      2. EXECUTE

      3. SELECT

      4. USE CATALOG

      5. USE SCHEMA

2. Make sure that an active compute resource is available in your Databricks workspace.

3. If you choose to read Databricks data from Azure Blob Storage, prepare either a SAS (shared access signature) for the Azure storage account or an Azure service principal. Choose one of the following authentication methods:

   1. Azure storage account SAS. The SAS must grant the Read, List, Write, and Add permissions on the relevant container.

   2. Azure service principal. The service principal must have the Storage Blob Data Contributor role for the relevant storage account.

      1. For more information about how to create a service principal and assign a role, see the official Azure documentation.

      2. The following command line shows how to use the Azure az command to create a Service Principal locally and assign it the Storage Blob Data Contributor role.

         az ad sp create-for-rbac \
             --name "blob_owner" \
             --role "Storage Blob Data Contributor" \
             --scopes /subscriptions/08****58-****-****-****-1f6d****48ff/resourcegroups/<resourcegroupName>/providers/Microsoft.Storage/storageAccounts/<storageAccountName>

         A sample response is shown below:

         Apple@B-2AXHMD6M-0228 ~(spark-3.5.4) % az ad sp create-for-rbac \
             --name "blob_contributor" \
             --role "Storage Blob Data Contributor" \
             --scopes /subscriptions/xxx/resourcegroups/xxx/providers/Microsoft.Storage/storageAccounts/xxx
         Creating 'Storage Blob Data Contributor' role assignment under scope '/subscriptions/xxx/resourcegroups/xxx/providers/Microsoft.Storage/storageAccounts/xxx'
         The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
         {
           "appId": "7f2c4503-2f3d-4ff6-a292-cda430be99ed",
           "displayName": "blob contributor",
           "password": "xxx",
           "tenant": "xxx2-14a2-4b7c-9430-3xxx61"
         }

   Note: If the storage account was automatically created by Databricks when the Databricks instance was created, the account can be accessed only by Databricks. To allow MMS to migrate data by reading from Azure Blob Storage, you must ask Azure to remove the deny assignment from the storage account.

Step 2: Add the data source

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Data Transfer > Migration Service.

3. On the Data Source tab, click Add Data Source.

   To create the service-linked role, click OK in the MaxCompute Service-linked Role dialog box. If this dialog box does not appear, the role already exists.

   On the Add Data Source page, configure the data source and click Add.

   • Data Source Basic Information

     Parameter	Required	Description
     Name	Yes	A custom name for the data source. It can contain letters, digits, and Chinese characters. Special characters are not allowed.
     Data Source Type	Yes	Select Databricks.
     Network Connection	Yes	Select the network connection to use. You can create a network connection in the MaxCompute console under Manage Configurations > Network Connection. MMS uses this connection to communicate with a VPC and connect to the data source.
     Databricks workspace URL	Yes	The URL of your Databricks workspace. You can find this URL by navigating to your Databricks instance in the Azure portal. On the Overview page of the instance, find the URL field in the Essentials section. The format is https://adb-{workspace-id}.azuredatabricks.net.
     Databricks Unity Catalog	Yes	The Databricks Unity Catalog that you want to migrate. Each MMS data source can migrate only one Databricks Unity Catalog.
     Databricks cluster ID	Yes	The ID of the Databricks cluster. This cluster is used to run SQL statements during metadata synchronization.
     Databricks access token	Yes	The Databricks personal access token. For more information, see Databricks personal access token authentication.
     Use Databricks JDBC for reading tables in Databricks Unity Catalog	No	Whether to use Databricks JDBC to read data from Databricks tables.
     Azure authentication	Yes	SAS: Access the storage account by using a SAS for the Azure Storage account. Azure storage account SAS token: To create a token, see the Azure documentation. Service Principal: Access the storage account by using an Azure service principal. Azure application ID Azure app client secret Azure tenant ID
     Default destination MaxCompute project	Yes	The default destination project for data migration. This parameter cannot be modified after the data source is created.
     Destination MaxCompute project list	No	If data from a single data source needs to be migrated to multiple destination projects, specify the projects in this list.
     MaxCompute project for migration jobs	Yes	The MaxCompute project where migration jobs, such as Spark and SQL jobs, are executed. The default compute quota associated with this project is used.

   • Additional Information

     These parameters are optional.

     Parameter	Description
     Jdbc page size	The default value is 100000.
     Scheduled metastore update	Whether to periodically synchronize metadata from the data source. Valid values: Update Cycle: Enables scheduled synchronization. You can set the Update Cycle to Daily or Hourly and specify an Update Started At time. Daily: The synchronization job runs once a day at the specified time (accurate to the minute). Hourly: The synchronization job runs once an hour at the specified minute. Disable: Disables scheduled synchronization.
     Metastore access concurrency	The number of concurrent connections to the metastore.
     Schema whitelist	The schemas to migrate. Separate multiple schema names with commas (,).
     Schema blacklist	The schemas to exclude from migration. Separate multiple schema names with commas (,).
     Table Blacklist	The Databricks tables to exclude from migration. The format for a single table is schema.table or table. Multiple tables are separated by a comma (,).
     Table Whitelist	The Databricks tables to migrate. The format for a single table is schema.table or table. Multiple tables are separated by a comma (,).
     Maximum migration task concurrency	The maximum number of concurrent MMS migration tasks. A high value may increase the compute load on Databricks. The default value is 20.
     Migration task SQL parameters	SQL parameters. For more information, see Flag parameters.
     Table name character mapping	A JSON-formatted mapping to convert special characters in source table names to characters that are supported by MaxCompute. Example: {"%": "_"}.
     Maximum partitions per task	The maximum number of partitions processed by a single migration task. Migrating partitions in batches reduces SQL submissions and saves time. You typically do not need to change this value. The default is 50.
     Maximum data size per task (GB)	The default is 5 GB. You typically do not need to change this value.
     Partition value character mapping	A JSON-formatted mapping to convert special characters in source partition values to characters that are supported by MaxCompute. Example: {"/": "_"}.

Step 3: Synchronize metadata

After you start the data source, MMS creates a job instance to connect to it and synchronize its metadata. You can then configure migration jobs.

1. On the Data Source tab, for the target data source, click Update metadata in the Actions column.

2. The Data Source tab shows the Status of the target data source.

3. If the metadata of the data source changes and you have enabled scheduled metastore updates, the system automatically updates the metadata at the configured interval. Manual synchronization is not required.