DingTalk Wiki CDC (Beta)

Overview

The DingTalk CDC connector periodically polls the DingTalk Open Platform API to continuously capture new and updated online documents in DingTalk Wiki workspaces based on document modification time. The connector outputs metadata such as the title, URL, workspace, parent node, and update time of each document, along with the document body in Markdown format. You can also obtain the raw block structure of documents. This connector applies to scenarios such as RAG knowledge base building, document search indexing, and lakehouse content synchronization.

How It Works (Polling)

DingTalk Wiki does not provide a change event stream that can be consumed directly. DingTalk CDC implements full and incremental reads by polling the OpenAPI:

1. Each workspace is an independent read shard. If workspace-id is not configured, the connector calls the workspace list API at the interval specified by scan.discovery.interval to discover all workspaces visible to the operator. After it is configured, only the specified workspaces are synchronized.

2. On each pull, the connector traverses the workspace node tree according to scan.discovery.max-depth and scan.discovery.max-nodes, filters DingTalk online document nodes (type=FILE and category=ALIDOC), and generates the cursor position for this pull based on document modification time.

3. Documents after the cursor are sorted by modification time. A single shard outputs at most scan.prefix-size documents per pull. For each changed document, the connector reads its block structure through a consistency check mechanism and renders it into a Markdown body.

4. When there are no incremental changes, the connector repeats the preceding checks at the interval specified by scan.poll-interval.

Changelog Semantics

The connector outputs a changelog with upsert semantics keyed by the primary key doc_id. The changelog contains INSERT and UPDATE_AFTER records, but does not contain UPDATE_BEFORE:

The current implementation does not detect document deletions. After a document is deleted, it no longer appears in the node list, and the connector does not output DELETE events. The corresponding row in the downstream system retains the state of the last synchronization. To clean up residual data, submit a batch job that performs an anti-join between the sink table and the source table on the primary key, and deletes the rows whose primary key no longer exists in the source table.

Supported Capabilities

Network Requirements

A Flink job needs to access the DingTalk Open Platform API (the default domain is api.dingtalk.com). If the job is deployed in a VPC, make sure that it has Internet access. For more information, see Network connection options in the Realtime Compute for Apache Flink documentation.

Application Creation and API Authorization

Regardless of the startup mode, you must first create an application and complete API authorization on the DingTalk Open Platform.

1. Log on to the DingTalk Developer Console and create an enterprise internal application. On the Credentials and Basic Information page of the application, obtain the Client ID (AppKey) and Client Secret (AppSecret). The connector uses the AppKey and AppSecret to call POST /v1.0/oauth2/accessToken to obtain the access token of the enterprise internal application and automatically renews the token.

2. Grant the API permissions required by this connector to the application. In the developer console, go to the Permissions page of the target application, and search for and apply for the permissions listed in the following table. Some permissions require approval by an organization administrator. The connector calls the following DingTalk OpenAPI operations:

Permission names are subject to the official documentation of the DingTalk Open Platform. The following table summarizes the required permissions:

Operator and Workspace Data Authorization

Set operator-id to the Union ID of an enterprise member. You can obtain the Union ID by calling the Query User Details API of the DingTalk Open Platform. All API calls of the connector are made as this operator, and the visible scope is determined by the permissions of the operator:

• The operator must have at least the View Only permission on the target workspace or document. The View/Download permission or a higher permission is also acceptable.

• If workspace-id is not configured for the job, the connector can discover only the workspaces visible to the operator.

DingTalk Wiki supports three permission levels: workspace level, folder level, and document level. Documents inherit the permissions of their parent directory by default. The following table describes the permission types and the minimum requirements of this connector:

Authorization methods: The workspace administrator adds the operator in Settings > Members and Permissions of the workspace. Authorization by member, department, or group is supported. You can also grant permissions to the operator in the sharing settings of a single document. New documents inherit workspace permissions by default. After you grant permissions on the whole workspace, newly added documents do not require separate configuration. The connector is a read-only pipeline and does not require any write permission.

Authorization Troubleshooting

The connector does not retry on authorization errors. It fails directly or skips the corresponding document. Troubleshoot based on the following table:

Key Features

• Fixed-schema source table: outputs the document title, URL, body (Markdown), and metadata, and outputs a changelog with upsert semantics keyed by the primary key doc_id.

• Startup modes: select the read method by using scan.startup.mode.

• Raw block structure output: outputs the JSON text of the raw document block structure through the blocks_json METADATA column. You must also enable scan.block.include-raw-json.

Syntax

The Flink SQL DDL must declare all seven physical fields listed in the following table. The field order, names, and case must match exactly. Field names are case-sensitive. You must also declare PRIMARY KEY (doc_id) NOT ENFORCED.

In addition, the DDL can declare the following read-only metadata by using METADATA. Metadata columns do not change the physical schema described above.

CREATE TEMPORARY TABLE dingtalk_wiki_source (
  doc_id STRING NOT NULL,
  workspace_id STRING,
  parent_node_id STRING,
  title STRING,
  url STRING,
  content STRING,
  modified_time TIMESTAMP_LTZ(3),
  blocks_json STRING METADATA VIRTUAL,
  PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
  'connector' = 'dingtalk-cdc',
  'object-type' = 'document',
  'app-key' = '<yourAppKey>',
  'app-secret' = '<yourAppSecret>',
  'operator-id' = '<yourOperatorUnionId>',
  'workspace-id' = '<yourWorkspaceId>',
  'scan.startup.mode' = 'initial',
  'scan.block.include-raw-json' = 'true'
);

WITH Parameters

Examples

Synchronize workspace documents in full + incremental mode and write them to a Paimon table:

CREATE TEMPORARY TABLE dingtalk_wiki_source (
  doc_id STRING NOT NULL,
  workspace_id STRING,
  parent_node_id STRING,
  title STRING,
  url STRING,
  content STRING,
  modified_time TIMESTAMP_LTZ(3),
  blocks_json STRING METADATA VIRTUAL,
  PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
  'connector' = 'dingtalk-cdc',
  'object-type' = 'document',
  'app-key' = '<yourAppKey>',
  'app-secret' = '<yourAppSecret>',
  'operator-id' = '<yourOperatorUnionId>',
  'workspace-id' = '<yourWorkspaceId>',
  'scan.startup.mode' = 'initial',
  'scan.block.include-raw-json' = 'true'
);

CREATE TEMPORARY TABLE paimon_sink (
  doc_id STRING,
  workspace_id STRING,
  parent_node_id STRING,
  title STRING,
  url STRING,
  content STRING,
  modified_time TIMESTAMP_LTZ(3)
) WITH (
  'connector' = 'paimon',
  ...
);

INSERT INTO paimon_sink
SELECT doc_id, workspace_id, parent_node_id, title, url, content, modified_time
FROM dingtalk_wiki_source;

snapshot performs only one full read. It is suitable for content migration or one-time export:

CREATE TEMPORARY TABLE dingtalk_wiki_snapshot (
  doc_id STRING NOT NULL,
  workspace_id STRING,
  parent_node_id STRING,
  title STRING,
  url STRING,
  content STRING,
  modified_time TIMESTAMP_LTZ(3),
  PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
  'connector' = 'dingtalk-cdc',
  'object-type' = 'document',
  'app-key' = '<yourAppKey>',
  'app-secret' = '<yourAppSecret>',
  'operator-id' = '<yourOperatorUnionId>',
  'workspace-id' = '<yourWorkspaceId>',
  'scan.startup.mode' = 'snapshot'
);

To exclude specific documents, add 'scan.document.exclude-node-ids' = '<nodeId1>,<nodeId2>' to the WITH parameters.

Key Features

• Automatic table creation: Each workspace corresponds to a table named <workspaceId>.documents. When the connector discovers a workspace for the first time, it automatically creates the table and synchronizes the schema.

• Multi-workspace synchronization: workspace-id supports multiple comma-separated values. If not configured, all workspaces visible to the operator are automatically discovered, and each workspace is automatically routed to its own table.

• Raw block structure output: After you enable scan.block.include-raw-json: true, the output schema contains the blocks_json physical column.

Usage Notes

• Data ingestion does not support the BATCH execution mode. To export all data, use the snapshot startup mode: the read is bounded, and the job ends automatically after the read completes in a streaming deployment.

• The full phase (snapshot shards) outputs INSERT events. The incremental phase outputs REPLACE (upsert) events. The downstream system only needs to perform idempotent writes based on the primary key doc_id.

• Workspaces newly discovered during execution are always read in the initial manner: the current full content is read first, and then subsequent changes are followed.

• You cannot change scan.block.include-raw-json when a job is restored from a checkpoint or savepoint. Otherwise, the job reports an error and refuses to start.

• Only DingTalk online document nodes are supported (type=FILE and category=ALIDOC). Other objects are skipped. Document deletions are not detected.

• The connector continuously calls the DingTalk Open Platform API. Use the rate limiting parameters to control the API call costs. For more information, see the Control Synchronization Speed and API Call Costs example below.

• app-secret is sensitive information. Reference it by using a project variable, such as ${secret_values.dingtalk_app_secret}, instead of writing plaintext credentials in the job configuration. For more information, see Manage variables.

Syntax

The fixed YAML schema is as follows. The primary key is doc_id. Compared with Flink SQL, the YAML schema outputs blocks_json directly as a physical column instead of a METADATA column, and workspace_id is NOT NULL.

Minimal syntax example:

source:
  type: dingtalk-cdc
  name: DingTalk Wiki Source
  object-type: document
  app-key: <yourAppKey>
  app-secret: <yourAppSecret>
  operator-id: <yourOperatorUnionId>
  workspace-id: <yourWorkspaceId>
  scan.startup.mode: initial

route:
  - source-table: <yourWorkspaceId>.documents
    sink-table: dingtalk_wiki.documents

sink:
  type: paimon
  catalog.properties.metastore: rest
  catalog.properties.token.provider: dlf
  catalog.properties.uri: <yourDlfCatalogEndpoint>
  catalog.properties.warehouse: <yourWarehouse>

The preceding example performs a full read of documents in the specified workspace, continuously synchronizes incremental changes, and writes them to Paimon. To synchronize multiple workspaces, set workspace-id to multiple comma-separated IDs. To preserve the raw block structure, add scan.block.include-raw-json: true.

Examples

Each of the following examples is a complete configuration of a data ingestion YAML job. Copy an example and modify it based on your job scenario. For more information about the meaning and the method of obtaining app-key, app-secret, and operator-id, see Prerequisites. The source connector is compatible with all data ingestion sink connectors, such as Paimon, Hologres, and MaxCompute. For the sink configuration, see the documentation of the corresponding connector. All the following examples use Paimon.

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator. All API operations are called as this identity, which must have permission to read the workspaces to synchronize
  workspace-id: <yourWorkspaceId>       # Optional. The ID of the workspace to synchronize. If you leave this parameter empty, all workspaces visible to the operator are automatically discovered
  scan.startup.mode: initial            # Optional. The startup mode. Valid values: initial (default, full + incremental), snapshot (full only), and latest-offset (incremental only)

route:
  # The connector automatically creates a source table named <workspaceId>.documents for each workspace,
  # and you can use route to map it to a specific sink table. If you do not configure route, the data is written to the downstream as <workspaceId>.documents.
  - source-table: <yourWorkspaceId>.documents   # The source table. The format is fixed as <workspaceId>.documents
    sink-table: dingtalk_wiki.documents         # The name of the sink table, which can be changed

sink:
  type: paimon                                  # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest            # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server, in the format of http://<region-id>-vpc.dlf.aliyuncs.com
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog
  # (Optional) Enable deletion vectors to improve the read performance of downstream systems
  table.properties.deletion-vectors.enabled: true

pipeline:
  name: DingTalk Wiki to Paimon Pipeline        # The job name

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator, which must have read permissions on all workspaces to synchronize
  workspace-id: <workspaceId1>,<workspaceId2>   # Optional. The IDs of multiple workspaces, separated by commas (,)
  # (Optional) The discovery interval for new workspaces. Default value: 1min. If workspace-id is configured, this parameter affects only the node scan cycle of the specified workspaces
  scan.discovery.interval: 1min

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog
  table.properties.deletion-vectors.enabled: true     # (Optional) Enable deletion vectors to improve the read performance of downstream systems

# route is not configured: the two workspaces are written to the <workspaceId1>.documents and <workspaceId2>.documents sink tables

pipeline:
  name: DingTalk Wikis to Paimon Pipeline     # The job name
  # (Optional) The parallelism of the job. Each workspace is an independent read shard. When you synchronize multiple workspaces, a higher parallelism speeds up the full read
  parallelism: 2

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator, which must have read permissions on all workspaces to synchronize
  workspace-id: <workspaceId1>,<workspaceId2>   # Optional. The IDs of multiple workspaces, separated by commas (,)

route:
  # Point multiple route rules to the same sink table to merge multiple workspaces.
  # To distinguish the source of each document, keep the workspace_id field in the sink table.
  - source-table: <workspaceId1>.documents      # The source table of the first workspace
    sink-table: dingtalk_wiki.documents         # The sink table that receives the merged data
  - source-table: <workspaceId2>.documents      # The source table of the second workspace
    sink-table: dingtalk_wiki.documents         # Written to the same table as the preceding rule

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog
  table.properties.deletion-vectors.enabled: true     # (Optional) Enable deletion vectors to improve the read performance of downstream systems

pipeline:
  name: Merge DingTalk Wikis to One Paimon Table      # The job name

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator. The synchronization scope covers the workspaces visible to this operator
  # workspace-id is not configured: all workspaces visible to the operator are automatically discovered
  # (Optional) The discovery interval for new workspaces. Default value: 1min
  scan.discovery.interval: 1min
  # (Optional) The IDs of the document or folder nodes to exclude, separated by commas (,). The excluded nodes are skipped in both the discovery and pull phases
  scan.document.exclude-node-ids: <excludeNodeId1>,<excludeNodeId2>

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog
  table.properties.deletion-vectors.enabled: true     # (Optional) Enable deletion vectors to improve the read performance of downstream systems

pipeline:
  name: Sync All DingTalk Wikis to Paimon       # The job name
  # (Optional) The parallelism of the job. If there are many workspaces, a higher parallelism speeds up the full read
  parallelism: 4

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator, which must have permission to read the workspaces to synchronize
  workspace-id: <yourWorkspaceId>       # Optional. The ID of the workspace to export. If you leave this parameter empty, all workspaces visible to the operator are exported
  scan.startup.mode: snapshot           # Full read only. The job ends automatically after the read completes (bounded)

route:
  - source-table: <yourWorkspaceId>.documents   # The source table. The format is fixed as <workspaceId>.documents
    sink-table: dingtalk_wiki.documents         # The name of the sink table, which can be changed

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog

pipeline:
  name: DingTalk Wiki One-Time Export   # The job name

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator, which must have permission to read the workspaces to synchronize
  workspace-id: <yourWorkspaceId>       # Optional. The ID of the workspace to synchronize. If you leave this parameter empty, all workspaces visible to the operator are automatically discovered
  scan.startup.mode: latest-offset      # Skip the full phase and synchronize only the documents that change after the job starts

route:
  - source-table: <yourWorkspaceId>.documents   # The source table. The format is fixed as <workspaceId>.documents
    sink-table: dingtalk_wiki.documents         # The name of the sink table, which can be changed

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog
  table.properties.deletion-vectors.enabled: true     # (Optional) Enable deletion vectors to improve the read performance of downstream systems

pipeline:
  name: DingTalk Wiki Incremental Sync  # The job name

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator, which must have permission to read the workspaces to synchronize
  workspace-id: <yourWorkspaceId>       # Optional. The ID of the workspace to synchronize
  scan.startup.mode: initial            # Optional. The startup mode. Default value: initial (full + incremental)
  scan.block.include-raw-json: true     # Optional. Appends the blocks_json physical column to the output schema, which stores the raw block structure of the document as JSON text

route:
  - source-table: <yourWorkspaceId>.documents   # The source table. The format is fixed as <workspaceId>.documents
    sink-table: dingtalk_wiki.documents         # The name of the sink table, which can be changed

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog
  table.properties.deletion-vectors.enabled: true     # (Optional) Enable deletion vectors to improve the read performance of downstream systems

pipeline:
  name: DingTalk Wiki with Raw Blocks   # The job name

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator, which must have permission to read the workspaces to synchronize
  workspace-id: <yourWorkspaceId>       # Optional. The ID of the workspace to synchronize

transform:
  - source-table: <yourWorkspaceId>.documents   # The source table of the workspace to which the transform rule applies
    # Projection: keeps only the fields required by the RAG index and drops the columns that are not required, such as parent_node_id. The primary key doc_id must be kept.
    projection: doc_id, workspace_id, title, url, content, modified_time
    description: keep only fields required by the RAG index

route:
  - source-table: <yourWorkspaceId>.documents   # The source table. The format is fixed as <workspaceId>.documents
    sink-table: dingtalk_wiki.documents         # The name of the sink table, which can be changed

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog

pipeline:
  name: DingTalk Wiki to Paimon with Transform  # The job name

source:
  type: dingtalk-cdc                    # Fixed value. The type of the DingTalk Wiki CDC connector
  name: DingTalk Wiki Source            # The source name
  object-type: document                 # Required. The type of the object to synchronize. Only document is supported
  app-key: <yourAppKey>                 # Required. The Client ID (AppKey) of the DingTalk internal app
  app-secret: <yourAppSecret>           # Required. The Client Secret (AppSecret) of the DingTalk internal app
  operator-id: <yourOperatorUnionId>    # Required. The union ID of the operator, which must have permission to read the workspaces to synchronize
  workspace-id: <yourWorkspaceId>       # Optional. The ID of the workspace to synchronize
  # (Optional) The polling interval when no incremental changes occur. Default value: 10s. A larger value reduces API calls during idle periods, at the cost of a higher synchronization latency for new content
  scan.poll-interval: 30s
  # (Optional) The maximum number of documents whose content a single workspace emits in one scan. Default value: 100. A larger value synchronizes more changed documents in each round
  scan.prefix-size: 100
  # (Optional) The following three rate limiting parameters are process-level and shared by all jobs that use the same app-key. Evaluate them against the API quota of your organization
  # The rate limit on the discovery requests that list workspaces and nodes. Default value: 5 QPS
  scan.discovery.rate-limit.requests-per-second: 5
  # The rate limit on the requests that resolve node metadata. Default value: 20 QPS
  scan.metadata.rate-limit.requests-per-second: 20
  # The rate limit on the Block API requests that read document content. Default value: 10 QPS
  scan.content.rate-limit.requests-per-second: 10

route:
  - source-table: <yourWorkspaceId>.documents   # The source table. The format is fixed as <workspaceId>.documents
    sink-table: dingtalk_wiki.documents         # The name of the sink table, which can be changed

sink:
  type: paimon                          # The type of the sink. This example writes data to Paimon
  catalog.properties.metastore: rest    # The metastore type. Fixed value rest for DLF
  catalog.properties.token.provider: dlf        # The token provider. Fixed value dlf for DLF
  catalog.properties.uri: <yourDlfCatalogEndpoint>    # The endpoint of the DLF Rest Catalog Server
  catalog.properties.warehouse: <yourWarehouse>       # The name of the DLF catalog

pipeline:
  name: DingTalk Wiki Rate-Limited Sync # The job name

Configuration Parameters