Solution 6: Network connectivity for public network data sources

This solution is recommended if your data source meets the following condition:

• The data source has a public endpoint.

Step 1: Get basic information

Step 2: Establish network connectivity

• Serverless resource group: A serverless resource group does not have internet access by default. You must configure an Internet NAT Gateway and bind an EIP for the VPC associated with the resource group before it can access data sources over the internet.

• Legacy exclusive resource group: Has internet access by default and can connect directly.

Step 3: (Optional) Add IP addresses to the allowlist

If the data source has allowlist controls, add the public IP address of the resource group to the data source allowlist.

The following example uses MySQL IP allowlist configuration, where a specific user can access the database only from the public IP address of the resource group.

1. Log on to the database as an administrator.

2. Create an account for DataWorks to access the data source and configure the required permissions.

   -- "dataworks_user" is the username, which you can customize.
   -- "StrongPassword123!" is the user password, which you can customize.
   CREATE USER 'dataworks_user'@'<Public IP address bound to the resource group>' IDENTIFIED BY 'StrongPassword123!';
   -- Grant the user access to the specified database (e.g., mydatabase) from the public IP address bound to the resource group.
   GRANT ALL PRIVILEGES ON mydatabase.* TO 'dataworks_user'@'<Public IP address bound to the resource group>' WITH GRANT OPTION;

3. Run the FLUSH PRIVILEGES; command to refresh permissions, and then exit the database (exit).

1. Basic information

2. Establish network connectivity

This procedure applies only to serverless resource groups. It uses an Internet NAT Gateway to enable internet access for the associated VPC. Legacy resource groups already have a bound EIP by default and do not require this configuration.

1. Go to the DataWorks Resource Group List page, find the target resource group, and click Network Settings in the Operation column.

2. Under the corresponding feature module, find the associated VPC and click the icon next to the VPC to go to the VPC Basic Information page.

   For example, if you want to connect a MySQL instance with a public endpoint to DataWorks for data synchronization, find the corresponding VPC under Task Scheduling & Data Integration and click the icon next to the VPC.

3. Switch to the Resource management tab. In the Internet Access Service section, click Create under Internet NAT Gateway to enable internet access for the VPC associated with the resource group.

   Configure the following key parameters:

   Parameter	Value
   VPC	Use the same VPC and vSwitch that are associated with the resource group.
   Associated vSwitch
   Access mode	Select Full VPC mode (SNAT).
   EIP Instance	Select Purchase New EIP.
   Associated Role	When you create a NAT gateway for the first time, you must create a service-linked role. Click Create Associated Role.

4. Click Buy Now, complete the payment, and create the NAT gateway instance.

   After the purchase is complete, a success message is displayed along with the results of three resource operations: Create EIP (created), Create NAT Gateway (created), and Bind EIP (bound).

5. After the NAT gateway instance is purchased, click the button to return to the console and create an SNAT entry.

   Note

   Only after an SNAT entry is configured can resource groups using this VPC access the internet.

   SNAT entries support four granularity levels: VPC, vSwitch, ECS/ENI, and custom CIDR block. The vSwitch granularity covers only the selected vSwitches. If the DataWorks resource group is deployed on a vSwitch that is not selected, it cannot access the internet through this SNAT entry. We recommend VPC granularity, where the source CIDR block is 0.0.0.0/0. This covers all CIDR blocks in the VPC and ensures that resource groups on all vSwitches can access the internet through the NAT gateway.

   1. Click Management in the Operation column of the newly purchased instance to go to the management page of the target NAT gateway instance, and switch to the SNAT Management tab.

   2. In the SNAT Entry List section, click Create SNAT Entry to create a NAT entry. The following are the key configurations:

      Parameter	Value
      SNAT entry granularity	Select VPC granularity to ensure that all resource groups in the VPC can access the internet through the configured EIP.
      Select EIP address	Select the EIP bound to the current NAT gateway instance.

      After you configure the SNAT entry parameters, click OK to create the SNAT entry.

   In the SNAT Entry List section, when the Status of the newly created SNAT entry changes to Enabled, the VPC associated with the resource group has internet access.

   If you previously configured a vSwitch-granularity SNAT entry, you can delete it after creating a VPC-granularity SNAT entry. The VPC-granularity entry uses a source CIDR block of 0.0.0.0/0, which already covers all vSwitch CIDR blocks in the VPC. Deleting the old vSwitch-granularity entry does not affect the VPC-granularity SNAT entry.

3. Add IP addresses to the allowlist

1. Obtain the public IP address of the resource group.

   Serverless resource group

   1. Go to the VPC console. In the left-side navigation pane, click NAT Gateway > Internet NAT Gateway to go to the Internet NAT Gateway list page.

   2. Find the Internet NAT Gateway that you created and view the EIP column to obtain the EIP address.

   Legacy resource group

   1. Go to the DataWorks Resource Group List page, find the target resource group, and click Details in the Operation column to go to the resource group details page.

   2. Obtain the EIP address.

2. Log on to the database as an administrator.

3. Create an account for DataWorks to access the data source and configure the required permissions.

   -- "dataworks_user" is the username, which you can customize.
   -- "StrongPassword123!" is the user password, which you can customize.
   CREATE USER 'dataworks_user'@'<Public IP address bound to the resource group>' IDENTIFIED BY 'StrongPassword123!';
   -- Grant the user access to the specified database (e.g., mydatabase) from the vSwitch CIDR block bound to the resource group.
   GRANT ALL PRIVILEGES ON mydatabase.* TO 'dataworks_user'@'<Public IP address bound to the resource group>' WITH GRANT OPTION;

4. Run the FLUSH PRIVILEGES; command to refresh permissions, and then exit the database (exit).

4. Test connectivity

1. Log on to the DataWorks console. In the target region, click Data Integration > Data Integration in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.

2. In the left-side navigation pane, click Data Sources to go to the Data Sources page, and then click Add Connection.

3. Select the MySQL data source type and configure the data source connection information.

   • Set Configuration Mode to User-created Data Store with Public IP Addresses.

   • Set Host Address to the public IP address of the server where MySQL resides (in this example, 47.117.XX.XX).

   • Set Port Number to 3306.

   • Set Database Name to the name of an existing database.

   • Set Username and Password to the dataworks_user user and password created in the Add IP addresses to the allowlist step.

4. In the Connection Configuration section, click Test Connectivity next to the resource group associated with the workspace and check whether the result is Connectable.

   Note

   If the test result shows Failed, use the Network Connectivity Diagnostic Tool to troubleshoot. If connectivity still fails, submit a ticket for assistance.

What do I do if a serverless resource group times out when connecting to a public FTP data source?

A serverless resource group does not have internet access by default. If a connection timeout occurs when accessing a public FTP data source, follow these steps to troubleshoot:

1. Verify that a NAT gateway and SNAT entry are configured for the VPC where the serverless resource group resides. If not, complete the NAT gateway binding and SNAT entry creation as described in "Step 2: Establish network connectivity" in this topic.

2. Verify that the NAT gateway is bound to the VPC and vSwitch actually used by the serverless resource group. Ensure that the binding is consistent with the resource group network settings.

3. In the VPC route table, add a route entry for the CIDR block corresponding to the FTP server IP address. For example, if the FTP server address is 121.4.x.x, add a route entry with the destination CIDR block 121.4.0.0/16 and set the next hop to the NAT gateway instance.

4. Check the allowlist on the FTP server side and add the egress IP of the serverless resource group (the EIP bound to the NAT gateway) to the FTP server allowlist.

Why does the table search feature fail to load the list but scripts can run when I connect to PostgreSQL over the internet?

When you connect to a PostgreSQL data source over the internet, the table search feature requires multiple interactions to retrieve metadata, which demands high network stability. In an unstable public network environment, metadata retrieval may time out or fail, causing the table search to fail to load. In contrast, script mode submits and runs synchronization tasks directly without relying on multiple metadata retrieval interactions, and is therefore unaffected by public network instability.

Recommended solutions:

• Use a private network connection preferentially: Access the PostgreSQL data source through a VPC connection to avoid public network instability issues. For more information, see Overview of network connectivity solutions.

• Use the SDK to create tasks: If you cannot switch to a private network connection, create data synchronization tasks by using the DataWorks OpenAPI or SDK to bypass the limitations of the console table search feature.

What do I do if the DataWorks console for an overseas region is inaccessible from a Chinese mainland network?

When you access the DataWorks console for an overseas region (such as US (Silicon Valley)) from the Chinese mainland, the page may fail to load or load slowly due to cross-border network latency or restrictions. Recommended steps:

1. Try a different computer or network environment (such as a mobile hotspot) to rule out local network issues.

2. If the issue is caused by cross-border network conditions, contact your network administration team to adjust network policies for accessing the DataWorks console in the overseas region.

3. For long-term stable access to an overseas console, use Alibaba Cloud Global Accelerator (GA) for low-latency, highly available cross-border network acceleration.

Can I use a proxy IP to disguise a resource group as an overseas IP?

No. Alibaba Cloud allocates and maintains the egress IP address for each resource group. You cannot modify or customize this address by using a proxy IP or other methods. Even if you configure proxy settings in your Python code or other programs, DataWorks tasks still use the IP address allocated to the resource group at runtime. To access geo-restricted data sources, use VPN Gateway or Express Connect to establish cross-border connections.