Feishu CDC

Background information

The Feishu CDC connector periodically polls the Feishu Open Platform OpenAPI and continuously captures new and updated online documents in Feishu wiki spaces based on document edit time. The connector outputs document metadata such as the title, URL, and update time, together with the document body in Markdown format. The raw block structure of each document is also available. Typical scenarios include RAG knowledge base building, document search indexing, and lakehouse content synchronization.

The following table lists the capabilities of the Feishu CDC connector.

How it works

Feishu wikis do not expose a change event stream that can be consumed directly. The Feishu CDC connector implements full and incremental reads by polling the OpenAPI:

1. Each wiki space is an independent read split. The connector discovers newly authorized wiki spaces at the interval specified by scan.discovery.interval.

2. On each poll, the connector lists all nodes in a space, filters the doc and docx document nodes within the configured scope, and generates a cursor position for the poll based on document edit time.

3. Documents after the cursor are sorted by edit time. A single split emits at most scan.prefix-size changed documents per poll. For each changed document, the connector reads its block structure and renders it into a Markdown body.

4. When no incremental change exists, the connector repeats the check at the interval specified by scan.poll-interval.

Prerequisites

The Flink job must access the Feishu Open Platform OpenAPI (default domain: open.feishu.cn). If the job is deployed in a VPC, make sure that public network access is enabled.

Regardless of the startup mode, you must first create an app on the Feishu Open Platform and authorize it to access the target wiki spaces.

1. Log on to the Feishu Open Platform and create an enterprise custom app to obtain the App ID and App Secret of the app. The connector uses the App ID and App Secret to obtain and automatically renew the tenant_access_token.

2. Grant the app the API permissions required by this connector and publish an app version. The connector calls the following Feishu OpenAPI operations:

   API	Purpose
   GET /open-apis/wiki/v2/spaces	Lists the wiki spaces that the app can access
   GET /open-apis/wiki/v2/spaces/{space_id}/nodes	Lists the document nodes in a wiki space
   GET /open-apis/wiki/v2/spaces/get_node	Resolves the metadata of a document node
   GET /open-apis/docx/v1/documents/{document_token}/blocks	Reads the block content of a document

3. Add the app as a member of the target wiki spaces so that the app can read the documents to be synchronized. For more information, see the topic "How to authorize an app to access wiki documents" in Wiki FAQ on the Feishu Open Platform. The connector synchronizes only the doc and docx nodes that the app is authorized to access.

Limitations

• Only source tables are supported. Dimension tables, sink tables, and watermarks are not supported.

• Only doc and docx nodes that the app is authorized to read in wiki spaces are synchronized. Other objects such as spreadsheets, bitable tables, and files are skipped.

• The current implementation does not detect document deletions and does not emit DELETE messages.

• A Flink SQL DDL must declare the six fixed physical fields and PRIMARY KEY (node_id) NOT ENFORCED. Adding or removing physical columns is not supported. blocks_json can only be declared as a METADATA column.

• A Flink SQL source table reads only one wiki space. wiki.space-id accepts only a single value. A data ingestion YAML job supports multiple comma-separated spaces, or automatic discovery when this parameter is not configured.

• The incremental pipeline is based on OpenAPI polling and generates continuous API calls. If a space contains many documents, tune scan.poll-interval, scan.prefix-size, and the rate limiting parameters to balance API call cost against the synchronization latency of new content.

• content is Markdown text rendered from the document block structure. It is not guaranteed to preserve all rich media and structural elements. To access the raw block structure, use the blocks_json METADATA column in Flink SQL or read the blocks_json physical column in a data ingestion YAML job.

• Resource discovery is cumulative. Shrinking wiki.space-id, wiki.document-url, or wiki.node-token does not remove wiki spaces or documents that have already been discovered.

• Failure recovery may emit duplicate INSERT and UPDATE_AFTER messages. We recommend that the downstream write idempotently by the primary key node_id.

• The YAML schema differs from the SQL schema: the YAML schema contains an additional space_id physical column, and blocks_json is a physical column rather than a METADATA column. Choose the corresponding schema definition based on the access method.

Startup modes

Use scan.startup.mode to select the read behavior.

For wiki spaces or documents newly discovered while the job is running, scan.new-split.startup.mode determines the read behavior: initial first reads the current full content and then follows subsequent changes; latest-offset processes only changes that occur after discovery.

Synchronization scope

Flink SQL and data ingestion YAML differ in how the synchronization scope can be configured:

If no document selector is configured, all doc and docx documents that the app is authorized to read in a space are synchronized.

To narrow the scope of a Flink SQL source table to a single document in the space, configure one of the following parameters:

• wiki.document-url: the URL of a Feishu document. The connector resolves the document identifier from the URL. Common URL forms such as wiki, docx, and docs are supported.

• wiki.node-token: the token of a node in the wiki space.

At most one of the two parameters can be configured, and the selected document must belong to the space configured in wiki.space-id. Once configured, both the full and incremental phases process only this document. The YAML document selector supports multiple documents. For more information, see the Data ingestion section.

Resource discovery is cumulative: once a wiki space or document has been discovered by the job, shrinking the scope configuration does not remove it from the running job. For jobs that synchronize an entire space, documents added to the space later are discovered by normal polling without configuration changes. To remove resources completely, stop the job and restart it without state.

SQL

CREATE TEMPORARY TABLE <yourTableName> (
  node_id STRING NOT NULL,
  title STRING,
  url STRING,
  `type` STRING,
  content STRING,
  updated_time TIMESTAMP_LTZ(3),
  PRIMARY KEY (node_id) NOT ENFORCED
) WITH (
  'connector' = 'feishu-cdc',
  'object-type' = 'document',
  'app-id' = '<yourAppId>',
  'app-secret' = '<yourAppSecret>',
  'wiki.space-id' = '<yourWikiSpaceId>'
);

CREATE TEMPORARY TABLE feishu_wiki_source (
  node_id STRING NOT NULL,
  title STRING,
  url STRING,
  `type` STRING,
  content STRING,
  updated_time TIMESTAMP_LTZ(3),
  blocks_json STRING METADATA VIRTUAL,
  PRIMARY KEY (node_id) NOT ENFORCED
) WITH (
  'connector' = 'feishu-cdc',
  'object-type' = 'document',
  'app-id' = '<yourAppId>',
  'app-secret' = '<yourAppSecret>',
  'wiki.space-id' = '<yourWikiSpaceId>',
  'scan.startup.mode' = 'initial'
);

CREATE TEMPORARY TABLE print_sink (
  node_id STRING,
  title STRING,
  url STRING,
  `type` STRING,
  content STRING,
  updated_time TIMESTAMP_LTZ(3)
) WITH (
  'connector' = 'print'
);

INSERT INTO print_sink
SELECT node_id, title, url, `type`, content, updated_time
FROM feishu_wiki_source;

CREATE TEMPORARY TABLE feishu_wiki_snapshot (
  node_id STRING NOT NULL,
  title STRING,
  url STRING,
  `type` STRING,
  content STRING,
  updated_time TIMESTAMP_LTZ(3),
  PRIMARY KEY (node_id) NOT ENFORCED
) WITH (
  'connector' = 'feishu-cdc',
  'object-type' = 'document',
  'app-id' = '<yourAppId>',
  'app-secret' = '<yourAppSecret>',
  'wiki.space-id' = '<yourWikiSpaceId>',
  'scan.startup.mode' = 'snapshot'
);

Data ingestion

The type of the data ingestion connector is fixed to feishu-cdc. Each wiki space corresponds to a table named <wiki space ID>.documents. The connector creates the table automatically and synchronizes the schema when the space is first discovered.

source:
  type: feishu-cdc
  name: <yourSourceName>
  app-id: <yourAppId>
  app-secret: <yourAppSecret>
  wiki.space-id: <yourWikiSpaceId>
  scan.startup.mode: initial

source:
  type: feishu-cdc
  name: Feishu Wiki Source
  app-id: <yourAppId>
  app-secret: <yourAppSecret>
  wiki.space-id: <yourWikiSpaceId>
  scan.startup.mode: initial

route:
  - source-table: <yourWikiSpaceId>.documents
    sink-table: feishu_wiki.documents

sink:
  type: paimon
  catalog.properties.metastore: rest
  catalog.properties.uri: <yourDlfCatalogEndpoint>
  catalog.properties.warehouse: <yourWarehouse>