Overview
Before DataWorks can run data synchronization, scheduling, or other tasks, the virtual private cloud (VPC) attached to your resource group must be able to reach your data source. This topic describes the available network connectivity solutions for different network environments.
Background
Most DataWorks features — such as creating data sources, data synchronization, data analysis, metadata collection, and DataService — operate on connected data sources or compute resources. If the data source you want to access is not in the VPC attached to your DataWorks resource group (for example, the data source is in another VPC or an on-premises IDC), you must select an appropriate network connectivity solution to connect the VPC attached to the resource group to the network where the data source resides.
For example, in a data synchronization flow, the VPC attached to the resource group must be connected to the networks of both the source and the destination.
Prerequisites
You have purchased a resource group with the required specifications. For details, see Use a serverless resource group.
For more information about resource groups, see Overview of resource groups.
The solutions in this topic apply to serverless resource groups and the following legacy resource groups: exclusive resource groups for Data Integration, exclusive resource groups for scheduling, and exclusive resource groups for Data Service.
Notes
Serverless resource groups can access data sources in complex network environments over a private network by associating a VPC. However, serverless resource groups cannot access the public network by default. To reach a data source over the Internet, configure an Internet NAT Gateway and EIPs for the VPC attached to the resource group. For details, see Connect to a data source over the Internet.
Public network connections offer no speed or stability guarantees. For data synchronization, use a private network or Cloud Enterprise Network (CEN).
Network connectivity between a resource group and a data source is required before tasks can run.
Resource groups do not support classic network environments. Migrate your data sources and services from a classic network to a VPC before proceeding.
Choose a solution
Your choice depends on where your data source is relative to your DataWorks workspace resource group. Use the information below to identify the right solution based on your scenario:
Solution 1: Connect to an Alibaba Cloud data source (same account, same region)
Use cases
This solution is recommended if your data source and DataWorks workspace meet all of the following conditions:
The data source is an Alibaba Cloud product.
The data source and the DataWorks workspace belong to the same Alibaba Cloud account.
The data source and the DataWorks workspace are in the same region.
Solution description
For the same-account, same-region scenario, we recommend that you use a VPC (private network) connection. Deploy the resource group of your DataWorks workspace and the data source in the same VPC to enable network connectivity.
Network connectivity diagram
In the same-account, same-region scenario, configure network connectivity between a DataWorks serverless resource group and an ApsaraDB RDS database as follows:
On the Database Connection page of the RDS instance, confirm the VPC to which the instance belongs. For example, VPC 1 with the CIDR block 192.168.0.0/16.
On the Resource Groups page of the DataWorks console, associate the serverless resource group with the same VPC as the RDS instance.
In the VPC Binding section of the resource group details, obtain the vSwitch CIDR block. For example, 192.168.0.0/24.
Add the vSwitch CIDR block to the dataworks group on the Whitelist and Security Group page of the RDS instance. For example, 192.168.6.0/24.
Configure network connectivity
For the detailed solution description and configuration steps, see Connect to a data source in the same account and region.
Solution 2: Connect to an Alibaba Cloud data source (same account, different regions)
Use cases
This solution is recommended if your data source and DataWorks workspace meet all of the following conditions:
The data source is an Alibaba Cloud product.
The data source and the DataWorks workspace belong to the same Alibaba Cloud account.
The data source and the DataWorks workspace are in different regions.
Solution description
For the same-account, different-region scenario, we recommend that you use a VPC (private network) connection. Use a network connectivity tool (Cloud Enterprise Network or VPC Peering Connection) to connect the VPC of your DataWorks workspace resource group with the VPC where the data source resides.
Network connectivity diagram
In the same-account, different-region scenario, where DataWorks resides in the China (Shanghai) region and the database resides in the China (Hangzhou) region, configure network connectivity as follows:
View the VPC information of the database instance to obtain the VPC CIDR block, such as
192.168.0.0/16, and the internal endpoint.On the Resource Groups page of the DataWorks console, find the serverless resource group that you want to use.
Use a network connectivity tool, such as Cloud Enterprise Network, to connect the VPC on the DataWorks side to the VPC on the database side.
In the VPC Binding section of the resource group details, view the associated vSwitch CIDR block, such as
172.16.0.0/24.Add the vSwitch CIDR block
172.16.0.0/24to the dataworks group in the whitelist of the database.At the bottom of the resource group details page, click Custom Route and add a route for the CIDR block of the VPC on the database side.
Configure network connectivity
For the detailed solution description and configuration steps, see Connect to a data source in the same account but a different region.
Solution 3: Connect to an Alibaba Cloud data source (different accounts)
Use cases
This solution is recommended if your data source and DataWorks workspace meet all of the following conditions:
The data source is an Alibaba Cloud product.
The data source and the DataWorks workspace belong to different Alibaba Cloud accounts.
Solution description
For the different-account scenario, we recommend that you use a VPC (private network) connection. Use a network connectivity tool (Cloud Enterprise Network or VPC Peering Connection) to connect the data source under Account A with the DataWorks workspace resource group under Account B.
Network connectivity diagram
In the different-account scenario, where Account A owns the ApsaraDB RDS database and Account B owns DataWorks, configure cross-account network connectivity as follows:
On the Database Connection page of the RDS instance under Account A, obtain the VPC of the instance and its CIDR block. In this example, the CIDR block is
192.168.0.0/16.On the Resource Groups page of the DataWorks console under Account B, view the details of the serverless resource group and confirm the vSwitch CIDR block that is associated in the Data Scheduling & Data Integration section. In this example, the CIDR block is
172.16.66.0/24.Use a network connectivity tool, such as Cloud Enterprise Network, to connect the VPC of Account B to the VPC of Account A so that the two VPCs can communicate with each other.
In Whitelist Settings of the RDS instance, add a dataworks group and add the vSwitch CIDR block
172.16.66.0/24of the DataWorks resource group to the whitelist.On the resource group details page in the DataWorks console, click Custom Route and configure a route to the VPC where the RDS instance resides.
Configure network connectivity
For the detailed solution description and configuration steps, see Connect to a data source under a different account.
Solution 4: Connect to a data source deployed on ECS
Use case
This solution is recommended if your data source meets the following condition:
The data source is deployed on an Alibaba Cloud ECS instance.
Solution description
If the ECS instance hosting the data source and DataWorks are under the same account and in the same region, we recommend that you use a VPC (private network) connection. Deploy the resource group of your DataWorks workspace and the ECS instance in the same VPC to enable network connectivity.
If the ECS instance hosting the data source and DataWorks are under different accounts or under the same account but in different regions, we recommend that you use a VPC (private network) connection. Use a network connectivity tool (Cloud Enterprise Network or VPC Peering Connection) to connect the VPC of your DataWorks workspace resource group with the VPC where the ECS instance resides.
Network connectivity diagram
Same account, same region
If the data source is deployed on an ECS instance that belongs to the same Alibaba Cloud account and resides in the same region as DataWorks, configure network connectivity as follows:
In the ECS console, view the VPC and vSwitch information of the ECS instance that you want to use, such as MySQL_on_ECS.
On the Resource Groups page of the DataWorks console, find the serverless resource group that corresponds to your workspace.
In the VPC Binding section of the resource group details, view the vSwitch CIDR block that is associated with Data Scheduling & Data Integration, such as
172.16.0.0/24.Add the vSwitch CIDR block to an inbound rule of the ECS security group. Set the protocol type to Custom TCP, the port range to MySQL(3306), the authorization object to the vSwitch CIDR block of the resource group, such as
172.16.0.0/24, and the action to Allow.
Same account, different regions
If the data source is deployed on an ECS instance that belongs to the same Alibaba Cloud account but resides in a different region from DataWorks, configure network connectivity as follows:
In the ECS console, view the VPC and vSwitch information of the ECS instance that you want to use. In this example, the instance is MySQL_on_ECS, which resides in Hangzhou Zone H and whose primary private IP address is
192.168.6.172.On the resource group details page of the DataWorks console, view the association record of Data Scheduling & Data Integration.
Use Cloud Enterprise Network or a VPC peering connection to connect the VPC of DataWorks, which is VPC2 in the China (Shanghai) region, to the VPC of the ECS instance, which is VPC1 in the China (Hangzhou) region.
Confirm that the vSwitch CIDR block of the DataWorks resource group is
172.16.0.0/24.Add an inbound rule to the ECS security group to allow access. Set the protocol type to Custom TCP, the source to the CIDR block of the DataWorks resource group
172.16.0.0/24, and the port range to MySQL(3306).Record the vSwitch CIDR block of the ECS instance. You need this CIDR block to configure a route.
On the resource group details page of the DataWorks console, click Custom Route and add a route that points to the vSwitch CIDR block of the ECS instance.
Different accounts
If the data source is deployed on an ECS instance under Alibaba Cloud Account A and DataWorks belongs to Alibaba Cloud Account B, configure network connectivity as follows:
In the ECS console of Account A, obtain the VPC and vSwitch information of the MySQL_on_ECS instance. In this example, the instance resides in Hangzhou Zone H and its primary private IP address is
192.168.6.172.In the DataWorks console of Account B, view the details of the serverless resource group and confirm that the vSwitch CIDR block associated with Data Scheduling & Data Integration is
172.16.66.0/24.Establish a cross-account VPC connection between VPC2 on the DataWorks side and VPC1 on the ECS side.
Add an inbound rule to the ECS security group to allow access. Set the protocol type to Custom TCP, the authorization object to
172.16.66.0/24, which is the vSwitch CIDR block of the DataWorks resource group, and the port range to MySQL(3306).On the resource group details page in the DataWorks console, click Custom Route and add a route that points to the vSwitch CIDR block of the ECS instance.
Configure network connectivity
For the detailed solution description and configuration steps, see Connect to a data source deployed on ECS.
Solution 5: Connect to a data source in an on-premises IDC
Use case
This solution is recommended if your data source meets the following condition:
The data source is deployed in an on-premises IDC.
Solution description
For data sources deployed in an on-premises IDC, we recommend that you use a VPC (private network) connection. Use a network connectivity tool (Express Connect) to connect the on-premises network where the data source resides with the VPC of your DataWorks workspace resource group.
Network connectivity diagram
To connect to a data source that is deployed in an on-premises IDC, MySQL in this example, perform the following steps:
Confirm the CIDR block of the IDC, such as
172.16.10.0/24.In the DataWorks console, obtain the information about the exclusive resource group for Data Integration.
Use an Express Connect circuit to connect the VPC to the IDC.
View the vSwitch CIDR block that is associated with the resource group, such as
192.168.0.0/24.Configure whitelist and firewall rules on the MySQL database in the IDC to allow access from the vSwitch CIDR block of the resource group. Whitelist example:
CREATE USER 'dataworks_user'@'192.168.0.0/24' IDENTIFIED BY 'StrongPassword123';andGRANT ALL PRIVILEGES ON mydatabase.* TO 'dataworks_user'@'192.168.0.0/24' WITH GRANT OPTION;. Firewall example:sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.0.0/24" port="3306" protocol="tcp" accept'.On the resource group details page, click Custom Route and add a route for the CIDR block of the IDC.
Configure network connectivity
For the detailed solution description and configuration steps, see Connect to a data source in an on-premises IDC.
Solution 6: Connect to a data source over the Internet
Use case
This solution is recommended if your data source meets the following condition:
The data source has a public endpoint.
Solution description
Serverless resource groups cannot access the public network by default. You must configure an Internet NAT Gateway and EIPs for the VPC attached to the resource group before you can access data sources over the Internet.
Legacy resource groups have public network access by default and can connect directly.
NoteLegacy resource groups are being phased out. We recommend that you use serverless resource groups.
Network connectivity diagram
The diagram applies only to serverless resource groups. Legacy resource groups already have EIPs attached by default and can connect directly.
Configure network connectivity
For the detailed solution description and configuration steps, see Connect to a data source over the Internet.
References
For more information about resource groups, see Overview of resource groups.
To create and use a resource group, see Use a serverless resource group.
To associate a VPC with a resource group, see Associate a VPC.
To configure an Internet NAT Gateway for the VPC and vSwitch attached to a resource group, see Create and manage an Internet NAT Gateway.
For common questions about network connectivity, see FAQ about network connectivity.