DocsICP FilingConsole
Documentation
Home Page Dataphin Intelligent Data construction and governance Dataphin-sharing mode (fully managed version) User Guide Management Center Manage data sources Custom data source type Example of Real-Time Development by Customizing Real-Time Data Sources

Example of Real-Time Development by Customizing Real-Time Data Sources

Updated at:
Copy as MD
Product
My Favorites

This topic introduces how to use custom real-time data sources for real-time development in Dataphin through examples.

Procedure

Step 1: Create a real-time data source

  1. On the Dataphin home page, navigate to the top menu bar and select Management Center > Datasource Management.

  2. In the left-side navigation pane, select Datasource > Custom Source Type.

  3. On the Custom Source Type page, click Create Custom Source Type, select Create Real-time Custom Source Type, and open the Create Real-time Custom Source Type page.

  4. On the Create Real-time Custom Source Type page, configure the following parameters:

    Parameter

    Pattern

    Type Name

    Enter Oracle_cdc_test.

    Type Encoding

    Enter Oracle_cdc_test.

    JAR Package

    Upload the JAR package of the data source type connector. This example uses the Community Edition of the Oracle CDC Connector. For JAR package download, see Oracle | Apache Flink CDC.

    Configuration File

    Enter the following configuration file:

    Note

    If you use another JAR package, make the corresponding modifications.

    #delclare the use type of this connector
kind:
  source
# the name of this connector
# if you use vvp flink,you also need to follow this match pattern [a-z0-9]([-a-z0-9\.]*[a-z0-9])?
connector: oracle-cdc
# for example, hostname is one property of this connector, you can see it in with clause
# when isSensitive is true, usually used for password, it is mean that the value of your key will be encrypted in page
# defaultValue is the default value of this key
# when isRequired is true, you must set value to the key, and the key in page is marked begin with start *
# when module is datasource it is mean that you can set the config in datasource page
# when module is table it is mean that you can set the config in table page
hostname:
  isSensitive: false
  isRequired: true
  module: datasource
port:
  isSensitive: false
  isRequired: true
  module: datasource
username:
  isSensitive: false
  isRequired: true
  module: datasource
password:
  isSensitive: true
  isRequired: true
  module: datasource
database-name:
  isSensitive: false
  isRequired: true
  module: table
schema-name:
  isSensitive: false
  isRequired: true
  module: table
table-name:
  isSensitive: false
  isRequired: true
  module: table
table-name:
  isSensitive: false
  isRequired: true
  defaultValue: latest-offset
  module: table
#format define the input or sink format of the data, it always with module table

    Configuration Item Description:

    The configuration item parameters in the configuration file will eventually become the WITH parameters in Flink DDL when the system is executed. As shown below:

    CREATE TABLE MyUserTable (
  ID INT NOT NULL,
  NAME STRING,
  DESCRIPTION STRING,
  WEIGHT DECIMAL(10, 3),
  PRIMARY KEY(id) NOT ENFORCED
  ) WITH (
  'connector' = 'oracle-cdc',
  'hostname' = 'localhost',
  'port' = '1521',
  'username' = 'Dataphin',
  'password' = 'fli*****ssword',
  'database-name' = 'ORCLCDB',
  'schema-name' = 'dataphin',
  'table-name' = 'myusertable');

    Description

    Enter Custom Real-Time Data Source Test.

  5. Click Confirm to complete the custom real-time data source type.

Step 2: Customize Ververica Flink Connector

If your Dataphin tenant uses the open-source Flink real-time computing engine, this step is not required. For more information, see the description of customizing Ververica Flink Connector.

  1. Log on to the Real-Time Compute Console.

  2. Click the Console under the Actions column of the target workspace.

  3. In the left-side navigation pane, click Connector.

  4. On the Connector page, click Create Custom Connector.

  5. Upload the JAR file of the custom connector.

    You can use one of the following methods to upload the JAR file of a custom connector:

    • Upload File: Click Select File and then select your target connector JAR file.

    • External URL: When you need to use a JAR file that exists on another service, you can use the external URL feature to obtain the JAR file. For example, https://ossbucket/artifacts/namespaces/flink-default/flink-jobs-1.0-SNAPSHOT.jar

      Note

      Only the following two types of external URLs are supported:

      • The OSS Bucket address selected when the Flink workspace is created. You can view the attached OSS Bucket in the details of the target workspace in the Real-Time Compute Management Console.

      • Other external storage system addresses that the Real-Time Compute Flink Edition can access and is allowed to access (public-read or granted permission).

  6. After the upload is complete, click Next.

    The system parses the content of the JAR file that you uploaded. If file parsing is successful, proceed to the next step. If file parsing fails, check whether the code of your custom connector complies with the standards that are defined by the Apache Flink community.

  7. Click Finish.

    The custom connector that you create appears in the connector list.

    Note

    The value of the connector parameter in the WITH clause is the value of the identifier parameter of DynamicTableFactory in the JAR file of the custom connector. Other parameters in the WITH clause and their definitions vary based on the custom connector that you create.

Step 3: Create a custom real-time data source

After the creation of the real-time data source is complete, Dataphin's data source management will generate a custom real-time data source. You can configure the connection information of the real-time data source in the data source management to integrate the data source into Dataphin, supporting subsequent metatable creation.

  1. In the top menu bar on the Dataphin home page, select Management Center > Datasource Management.

  2. On the data source page, click +create Datasource to open the Create Datasource dialog box.

  3. In the Create Datasource dialog box, in the Custom Datasource area, select Oracle_cdc_test.

  4. In the Create Oracle-cdc-test Datasource dialog box, configure the connection data source parameters.

    The configuration items of the custom real-time data source are determined by the YAML configuration file in Step 1. Different configuration parameters in the YAML configuration file will result in different options appearing on the data source interface, and the display form of the options will also vary.

    Parameter

    Description

    Datasource Name

    Enter Oracle-cdc-test.

    Datasource Code

    Enter Oracle_cdc_test.

    Datasource Description

    Enter Custom Real-Time Data Source Access Test.

    Datasource Config

    Select Production Datasource.

    Tag

    Default is Empty.

    hostname

    Enter the host address of the Oracle service.

    port

    Enter the port of the Oracle service.

    username

    Enter the username of the Oracle service.

    password

    Enter the password of the Oracle service.

  5. Click Confirm to complete the creation of the custom data source Oracle-cdc-test.

    Note

    Custom connectors (real-time data sources) do not support connectivity checks.

Step 4: Create a metatable for the custom data source

After configuring the connection information for the custom data source in data source management, you can use the configured data source to create a metatable to support subsequent real-time development.

  1. In the top menu bar on the Dataphin home page, select Development > Data Development.

  2. In the top menu bar, select Project (Dev-Prod mode requires selecting Environment).

  3. In the left-side navigation pane, select Data Processing > Tables.

  4. In the right-side table management list, click image the new icon, select Real-time Compute Table, and open the Create Table dialog box.

  5. In the Create Table dialog box, configure the parameters.

    The configuration items in the metatable are determined by the YAML configuration file in Step 1. Different configuration parameters in the YAML configuration file will result in different options appearing in the metatable, and the display form of the options will also vary.

    Parameter

    Description

    Table Type

    Select Metatable.

    Metatable Name

    Enter Oracle_cdc_test.

    Data Source

    Select the created Oracle-cdc-test.

    database-name

    Enter Database Name.

    schema-name

    Enter Schema.

    table-name

    Enter Table Name.

    Select Directory

    Default is Real-time Compute Table.

    Description

    Enter Custom Real-Time Data Source Metatable Test.

  6. Click Confirm to complete the creation of the metatable.

    After the metatable is created, Dataphin will automatically assemble the Flink SQL for the metatable. The parameter names in the WITH clause of the code are defined in the YAML configuration. The specific parameter values are configured on the data source page and the metatable page. The fields in the code are configured in the metatable structure. The final generated code for the metatable Oracle_cdc_test is as follows:

    create table oracle_cdc_test ( 
	`id`	INT	comment '',
	`name`	VARCHAR	comment '',
	PRIMARY KEY(id) NOT ENFORCED 
)
 with ( 
'hostname'='47.***.***.217'
,'connector'='oracle-cdc'
,'port'='1511'
,'database-name'='dataphin'
,'schema-name'='dataphin'
,'table-name'='dataphin-tables'
,'username'='flink_demo'

);
Previous：Create a custom real-time data source typeNext：Develop custom component examples