Create a custom real-time data source type

Limits

Only Ververica Flink compute engines are supported.

Notes on custom Ververica Flink connectors

After you configure a custom real-time data source type in Dataphin, you must also register the connector in the Ververica Flink real-time computing console. For more information, see .

To update the connector code, update it in both Dataphin and the Ververica Flink real-time computing platform. The JAR file uploaded to Ververica Flink is the version that takes effect.

Procedure

1. On the Dataphin home page, in the top menu bar, click Management Center > Datasource Management.

2. To navigate to the Create Offline Custom Source Type page, follow these steps.

   Go to Custom Source Type > Create Custom Source Type > Create Real-time Data Source Type.

   

3. On the Create Data Source Type page, configure the parameters.

   Parameter	Description
   Type Name	The name of the custom data source type. The name can be up to 128 characters long.
   Type Code	The code of the custom data source type. The code must follow these rules: Must be globally unique. Can be up to 64 characters long. Can contain only letters, digits, and underscores (_). Cannot start with a digit. Note This parameter cannot be modified after the type is created.
   JAR Package	Upload the JAR package for the data source type connector. Only files with the .jar extension are supported. The extension is case-insensitive. The file size cannot exceed 500 MB. The connector cannot be the same as an existing or reserved data source connector. The file can contain only data source-level encryption parameters. Note For more information about custom real-time connectors, see the Flink official website.
   Configuration File	Upload the configuration file for the data source type connector. Click Download Configuration File Template to configure the file. Only files with the .yaml extension are supported. The extension is case-insensitive. The connector cannot be the same as an existing or reserved data source connector. The file can contain only data source-level encryption parameters. The following example shows the configuration file structure: # Declare the usage type of the connector (source table, sink table, or dimension table). kind: source sink dim # Define the connector name. The format is CUSTOM_([A-Z|0-9]+). connector: CUSTOM_SOCKET # hostname is a property of the connector. It is displayed in the WITH clause of the DDL statement. # isSensitive: If set to false, the target property (for example, hostname) is not encrypted on the page. # defaultValue: Specifies the default value of the target property. # isRequired: If set to true, the property is required and marked with an asterisk (*) on the page. # module: Specifies where the property is configured. datasource = on the data source page; table = on the real-time metatable page. hostname: isSensitive: false defaultValue: localhost isRequired: true module: datasource port: isSensitive: false # format: Defines the serialization format for data input or output. The module for this property is usually set to table. format: name: changelog-csv module: table format_property: line-delimiter: isSensitive: false defaultValue: '/n' isRequired: true column-delimiter: isSensitive: false defaultValue: ','
   Description	Enter a brief description of the custom real-time source type. The description can be up to 128 characters long.

4. Click OK to create the custom real-time data source type.

What to do next

After you create a custom real-time data source type, its configuration items are automatically generated. You can then add a data source of this type to Dataphin. For more information, see Data Source Management.