Create a custom offline source type

Dataphin lets you create a custom offline source type to meet various data synchronization needs.

Prerequisites

For an RDBMS database, you must have the database driver file.
For other database types, you must have the JSON configuration for the data source connection and the reader/writer plugin.

Background information

An RDBMS database, or relational database, includes types such as MySQL, Oracle, and SQL Server. These databases typically share a common connection method that requires a database driver and connection parameters, such as a connection address, a username, and a password. Dataphin provides default data source settings for RDBMS databases. You only need the database driver to create an RDBMS custom offline source type.

For non-RDBMS databases, connection methods and parameters can vary widely. You can define the required connection parameters in JSON and upload a reader/writer plugin package to create a non-RDBMS custom offline source type.

Integration component generation

After you create a custom offline source type, Dataphin automatically generates an integration component of this type in the integration pipeline's component library. You can use this component for integration development. The generation process is as follows:

When you create a custom offline source type for an RDBMS database, Dataphin automatically generates reader and writer components in the public directory of the integration pipeline's component library.
For other custom source types, Dataphin generates reader or writer components in the public directory of the component library based on the reader/writer plugin that you upload.

Create an RDBMS source type

On the Dataphin homepage, choose Management Center > Data Source Management from the top navigation bar.
In the left-side navigation pane, choose Data Source > Custom Source Type.
On the Custom Source Type page, click New Custom Source Type, and select New Offline Custom Source Type from the drop-down list.

On the New Offline Custom Source Type page, configure the following parameters.

Parameter		Description
Basic settings	Type	Select RDBMS database.
	Name	The name of the custom component. The name can contain Chinese characters, uppercase and lowercase letters, underscores (_), and digits. The name can be up to 64 characters long.
	Type code	The type code is a backend identifier and cannot be changed after creation. The type code can contain only uppercase and lowercase letters, digits, and underscores (_). It cannot start with a digit.
Resource settings	Driver name	Enter the database driver name. The name can contain uppercase and lowercase letters, underscores (_), digits, and periods (.). The name can be up to 64 characters long. For example, `com.mysql.jdbc.Driver`.
Resource settings	Driver upload	Upload the database driver file. You can upload up to 20 files. Note Only `.jar` files are supported. The total size of all files cannot exceed 200 MB.
Description	Description	A brief description of the custom data source. The description can be up to 128 characters long.

Click Create.

Create a source type for other databases

On the New Offline Custom Source Type page, configure the following parameters.

Parameter		Description
Basic settings	Type	Select Other databases.
	Name	The name of the custom component. The name can contain Chinese characters, uppercase and lowercase letters, underscores (_), and digits. The name can be up to 64 characters long.
	Type code	The type code is a backend identifier and cannot be changed after creation. The type code can contain only uppercase and lowercase letters, digits, and underscores (_). It cannot start with a digit.
	Data source JSON	Enter the data source JSON code to define the connection parameters. The following code provides an example. [ { "columnName": "url", "columnType": "NORMAL", "text": { "zh_CN": "链接地址", "en_US": "address", "zh_TW": "Address (Traditional Chinese)" }, "placeholder": { "zh_CN": "请输入链接地址", "en_US": "input address", "zh_TW": "Enter Address (Traditional Chinese)" } }, { "columnName": "username", "columnType": "NORMAL", "text": { "zh_CN": "用户名", "en_US": "username", "zh_TW": "Username (Traditional Chinese)" }, "placeholder": { "zh_CN": "请输入用户名", "en_US": "input username", "zh_TW": "Enter Username (Traditional Chinese)" } }, { "columnName": "password", "columnType": "ENCRYPT", "text": { "zh_CN": "密码", "en_US": "password", "zh_TW": "Password (Traditional Chinese)" }, "placeholder": { "zh_CN": "请输入密码", "en_US": "input password", "zh_TW": "Enter Password (Traditional Chinese)" } } ]
Resource settings	Reader/writer plugin	Select a plugin type, enter its ClassName, and upload the plugin package. Only `.jar` files are supported. Each file cannot exceed 200 MB. You must configure a reader plugin, a writer plugin, or both. Dataphin generates a reader component for a reader plugin and a writer component for a writer plugin.
Description	Description	A brief description of the custom data source. The description can be up to 128 characters long.

Click Create.

Next steps

After you create the custom offline source type, Dataphin generates the data source settings for that type. You can then add a data source of this type to Dataphin. For more information, see Add a custom data source.