Create an Elasticsearch data source

Limits

If you need to create a data source based on Alibaba Cloud products in Dataphin, before creating the data source, you must ensure that the IP address of Dataphin has been added to the database whitelist (or security group) to enable network connectivity between the data source and Dataphin. For more information, see Data source whitelist configuration.

Permission requirements

Only users with the Create Data Source permission point in a custom global role and users with the super administrator, data source administrator, domain architect, and project administrator system roles can create data sources.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Management Center > Datasource Management.

2. On the Datasource page, click +Create Data Source.

3. On the Create Data Source page, select Elasticsearch in the NoSQL section.

   If you have recently used Elasticsearch, you can also select Elasticsearch in the Recently Used section. You can also enter Elasticsearch keywords in the search box to quickly search for it.

4. On the Create Elasticsearch Data Source page, configure the connection parameters.

   1. Configure the basic information of the data source.

      Parameter	Description
      Datasource Name	Enter the name of the data source. The name must meet the following requirements: It can contain only Chinese characters, letters, digits, underscores (_), and hyphens (-). It cannot exceed 64 characters in length.
      Datasource Code	After you configure the data source code, you can directly access Dataphin data source tables in Flink_SQL tasks or by using the Dataphin JDBC client in the format of data source code.table name or data source code.schema.table name for quick consumption. If you need to automatically switch data sources based on the task execution environment, access them using the variable format ${data source code}.table or ${data source code}.schema.table. For more information, see Flink_SQL task development method. Important The data source code cannot be modified after it is configured. You can preview data on the object details page in the asset directory and asset checklist only after the data source code is configured. In Flink SQL, only MySQL, Hologres, MaxCompute, Oracle, StarRocks, Hive, SelectDB, and GaussDB data warehouse service (DWS) data sources are currently supported.
      Version	Supports Elasticsearch 6.x, Elasticsearch 7.x, and Elasticsearch 8.x versions.
      Data Source Description	A brief description of the data source, not exceeding 128 characters.
      Data Source Configuration	Select the data source to configure: If your business data source distinguishes between production and development data sources, select Production + Development Data Source. If your business data source does not distinguish between production and development data sources, select Production Data Source.
      Tag	You can categorize and tag data sources based on tags. To create tags, see Manage data source tags.

   2. Configure the connection parameters between the data source and Dataphin.

      If you selected Production + Development Data Source in the previous step, the configuration page for Production + Development Data Source is displayed. If you selected Production Data Source, only the configuration page for Production Data Source is displayed.

      Note

      Typically, production and development data sources should be configured as separate data sources to achieve environment isolation between development and production data sources, reducing the impact of development data sources on production data sources. However, Dataphin also supports configuring them as the same data source with identical parameter values.

      Parameter	Description
      ES URL	The connection address of Elasticsearch. We recommend that you use a private network connection address. The format is http://host:port. For example: http://192.168.*.212:9200.
      Username, Password	The username and password for accessing the Elasticsearch instance.
      Type	Supports Directly Connectable Database, ApsaraDB, and Self-managed Database On ECS (VPC). You can select and configure based on the database type and business requirements. Directly Connectable Database: Connect to the database directly through the default scheduling cluster or registered scheduling cluster. This is suitable for the following scenarios: ① public network databases, ② databases in the same network environment as the registered scheduling cluster. If you need to add an access whitelist, you can add the public network outbound IP of the Dataphin default scheduling cluster: 47.102.192.174. ApsaraDB: A database purchased on Alibaba Cloud. Supports access via VPC Proxy or Direct Connection. VPC Proxy: When the current database is an Alibaba Cloud database in a VPC network environment, specify the authorized IP whitelist: 100.104.0.0/16 for the connection. Region: The region where the database is located. Only databases in the same region as the Dataphin instance are supported. If your Dataphin instance is in China (Shanghai), you can only select the China (Shanghai) region. VPC ID: Enter the VPC ID of the VPC network where the database is located. You can log on to the VPC console to view it. The following figure shows the test results. VPC Instance ID: Enter the VPC instance ID of the database, which is VpcCloudInstanceId. You can obtain it by calling the DescribeDrdsInstance API. For more information, see DescribeDrdsInstance. Direct Connection: Connect to the database directly through the default scheduling cluster or registered scheduling cluster. If you need to add an access whitelist, you can add the public network outbound IP of the Dataphin default scheduling cluster: 47.102.192.174. Self-managed Database On ECS (VPC): When the current database is an Alibaba Cloud database in a VPC network environment, specify the authorized IP whitelist: 100.104.0.0/16 for the connection. Region: The region where the database is located. Only databases in the same region as the Dataphin instance are supported. If your Dataphin instance is in China (Shanghai), you can only select the China (Shanghai) region. VPC ID: Enter the VPC ID of the VPC network where the ECS is located. You can log on to the VPC console to view it. The following figure shows the test results. ECS ID: Enter the ECS ID of the ECS server where the database is deployed. You can log on to the ECS console to view it. The following figure shows the test results.

5. Select a Default Resource Group, which is used to run tasks related to the current data source, including database SQL, offline database migration, data preview, and more.

6. Perform a Test Connection or directly click OK to save and complete the creation of the Elasticsearch data source.

   Click Test Connection, and the system will test whether the data source can connect normally with Dataphin. If you directly click OK, the system will automatically test the connection for all selected clusters, but the data source can still be created normally even if all selected clusters fail to connect.

   Important

   • If the connection test fails, you can troubleshoot based on common network connectivity issues. For more information, see Network connectivity solutions.

   • When the test connection reports an error VPC_GRANT_ACCESS_API_ERROR, you can refer to Solution for VPC_GRANT_ACCESS_API_ERROR for resolution.