Dataset

Datasets (formerly Data Connections) are the unified entry point for Alibaba Cloud Model Studio RAG to manage external data sources. By creating a dataset, applications can securely access data in enterprise databases, document systems, and object storage, and query and reference the data in real time during conversations.

Datasets are the unified entry point for RAG to manage external data sources. When you create a dataset, you select a dataset type based on the data source. Applications can then securely access data in enterprise databases, document systems, and object storage, and query and reference the data in real time during conversations.

Dataset types

Datasets fall into two main categories based on how data is stored and accessed — platform-hosted and stream processing — with three dataset types in total:

Prerequisites

Before you create a dataset, ensure you meet the following prerequisites:

• Account permissions: You must use a root user or a RAM user with permissions to manage datasets. A root user must authorize a RAM user before they can use the dataset feature. For details, see Permission Management.

• Data source preparation:
  • File/Table dataset: You must have documents or spreadsheets ready for upload, or have an existing OSS Bucket.
  • MySQL database: You must have a MySQL database instance (Alibaba Cloud RDS or self-managed). Ensure the instance is accessible via a public or private network.
  • PostgreSQL database: You must have a PostgreSQL database instance with its wal_level parameter set to logical.
  • PolarDB-X 2.0 database: You must have an Alibaba Cloud PolarDB-X 2.0 instance in a region that supports private network access. If you plan to import the data source using DMS, you must first register the PolarDB-X instance in DMS.
  • Yuque sync source: You must have a Yuque Knowledge Base and a Personal Access Token. Only the public version of Yuque is supported.
  • OSS sync source: You must have an OSS Bucket and have activated Alibaba Cloud Vector Search.

Create a dataset

1. Go to the Dataset page and click Create in the upper-right corner.

2. Enter a dataset name (up to 20 characters) and a description (up to 200 characters). The description guides how accurately agents call the dataset, so include the data content and purpose.

3. Select a dataset type (File, Table, or Database), and then configure the settings for that type:

   File

   A file dataset is used to manage unstructured documents (such as PDF and Word) and supports local upload and continuous synchronization from external sources such as Lark, DingTalk, SharePoint, Yuque, and OSS.

   1. Select File for the dataset type.

   2. Select a storage location:

      • Use Platform Storage: Data is stored in the storage space provided by Alibaba Cloud Model Studio. This option offers a limited-time free quota of up to 200,000 files and 1 TB of storage.

      • Use Your Own OSS Storage: Data is stored in your own OSS bucket. This option is suitable for large-scale data storage.

        Note

        • If you are using this option for the first time, follow the on-screen instructions to complete the authorization.
        • The target bucket must have the bailian-connector-access tag with the value ReadAndWrite to allow access from Alibaba Cloud Model Studio. Add tag

   Table

   A table dataset is used to manage structured data, such as CSV and Excel files.

   1. Select Table for the dataset type.

   2. Select a storage location:

      • Use Platform Storage: Data is stored in the storage space provided by Alibaba Cloud Model Studio. This option provides a 1 TB free quota. After the quota is used up, billing automatically switches to pay-as-you-go. This option is suitable for small-scale data storage.

      • Use Your Own OSS Storage: Data is stored in your own OSS bucket. This option is suitable for large-scale data storage.

        Note

        • If you are using this option for the first time, follow the on-screen instructions to complete the authorization.
        • The target bucket must have the bailian-connector-access tag with the value ReadAndWrite to allow access from Alibaba Cloud Model Studio. Add tag

   MySQL

   A MySQL database dataset is a stream processing dataset. It connects to a MySQL database, allowing applications to execute SQL queries and retrieve real-time data.

   NoteOnly MySQL datasets created using the import from DMS data source method support SQL query execution; those created with the create custom data source method do not.

   1. Select Database for the dataset type, and select MySQL for the database type.

   2. Configure Data Source Information:

      • Create custom data source: Manually configure the connection to an Alibaba Cloud RDS for MySQL instance or a self-managed MySQL database. You can connect over a public or private network.

        • Alibaba Cloud RDS for MySQL: Authorize access by using a Service-Linked Role (SLR) to associate the dataset with a MySQL database in Alibaba Cloud RDS. After you select this option, the database address and port are automatically populated based on the instance ID.
        • Self-managed MySQL: Manually configure the connection details for a remote self-managed MySQL database. You must enter the database address and port.

      • Import from DMS data source: Quickly import a data source that is already configured in Data Management Service (DMS). The first time you use this option, you must complete the SLR authorization for the EventBridge, RDS, and DMS service management roles.

   3. Select network type:

      • Public network (Default): Connect to the database over the public network. When you select this option, you must add the specified IP address range to the allowlist of your database instance.

      • Private network: Connect to the database over an internal network. You must also select the region. This option is recommended for production environments because it provides a more stable and secure connection.

      • Enter database connection information:

        Parameter	Description
        Database instance	Displayed only for Alibaba Cloud RDS for MySQL. Enter the RDS instance ID, for example, rm-******adq7.
        Database address	For Alibaba Cloud RDS for MySQL, this parameter is automatically populated and the input is disabled. For self-managed MySQL, you must enter it manually.
        Database port	For Alibaba Cloud RDS for MySQL, this parameter is automatically populated with 3306 and the input is disabled. For self-managed MySQL, you must enter it manually.
        Database username	Required. Enter the username for the database connection.
        Database password	Required. Enter the database password. The corresponding user must have read permissions on the database.

      • (Optional) Click Start Test to verify the database connectivity.

        The system uses EventBridge to test connectivity, and the test is free of charge. After the test is successful, you can select the target database from the Select DB drop-down list.

   NoteAfter a dataset is created, its database connection information, including the database password, cannot be modified. If you change the database password in the RDS console, the password saved in the dataset becomes invalid, data synchronization stops due to an authentication failure, and the knowledge base data is no longer updated. In this case, delete the original dataset and create a new one with the new password. Because a knowledge base cannot change its associated dataset, you must also re-create the associated knowledge base.

   PostgreSQL

   A PostgreSQL database dataset supports connections to Alibaba Cloud RDS for PostgreSQL instances and self-managed PostgreSQL databases.

   NoteOnly PostgreSQL datasets created using the import from DMS data source method support SQL query execution; those created with the create custom data source method do not.

   Prerequisites

   • The database account must have high-level permissions (Superuser or REPLICATION permissions).

   • The instance system parameter wal_level must be set to logical (the default is 'replica').

   • (For self-managed instances only) You have configured the listen_addresses parameter to allow access from the 100.64.0.0/16 CIDR block. To do this:

     1. Edit the configuration file: sudo vim /etc/postgresql/[version]/main/pg_hba.conf
     2. Add the following rule to the top of the file: host [database_name] [username] 100.64.0.0/16 md5
     3. Reload the configuration: sudo systemctl reload postgresql

   1. Select Database for the dataset type, and select PostgreSQL for the database type.

   2. Configure database connection:
      • Host Address: The connection address (public or private) of the database instance.
      • Port: The default is 5432.
      • Database Name (dbName): Required. Specify the database to connect to.
      • Username: A database account with high-level permissions.
      • Password: The database password.

   3. Click Test Connectivity to ensure the configuration is correct. The PostgreSQL dataset uses Data Transmission Service (DTS) to test connectivity.

   Comparison between MySQL and PostgreSQL datasets:

   Difference	MySQL	PostgreSQL
   Default port	3306	5432
   Additional required fields	None	Requires dbName (database name)
   Connectivity test service	EventBridge	Data Transmission Service (DTS)
   Special configuration requirements	None	Requires setting the instance system parameter wal_level to logical

   PolarDB-X 2.0

   A PolarDB-X 2.0 database dataset is a stream processing dataset. It connects to an Alibaba Cloud PolarDB-X 2.0 distributed database, allowing applications to execute SQL queries and retrieve real-time data.

   NoteOnly PolarDB-X 2.0 datasets created using the import from DMS data source method support SQL query execution; those created with the create custom data source method do not.

   1. Select Database for the dataset type, and select PolarDB-X 2.0 for the database type.

   2. Configure Data Source Information:

      • Create custom data source: Authorize access using a Service-Linked Role (SLR) to associate the dataset with an Alibaba Cloud PolarDB-X 2.0 database. The database address and port are automatically populated based on the selected instance and cannot be manually entered.

        The first time you use this option, you must authorize two Service-Linked Roles in the pop-up window: the DTS service management role (AliyunServiceRoleForSFMConnectorAccessDTS) and the PolarDB-X service management role (AliyunServiceRoleForSFMAccessPolarDBX).

      • Import from DMS data source: Quickly import a PolarDB-X data source that is already configured in Data Management Service (DMS). The first time you use this option, you must complete the SLR authorizations for the DMS, DTS, and PolarDB-X service management roles.

   3. Select network type: Only private network is supported. You must select a region to connect to the PolarDB-X instance over an internal network. This option is suitable for production environments.

   4. Enter database connection information:

      Parameter	Description
      Database instance	Displayed only for the create custom data source method. From the drop-down list, select a PolarDB-X 2.0 instance under the current account in the selected region.
      Select Data Source	Displayed only for the import from DMS data source method. From the list of DMS data sources, select one PolarDB-X data source. DMS ensures the connectivity of the data source.
      Database address	This field is automatically populated based on the selected instance and input is disabled when using the create custom data source method. When using the import from DMS data source method, you must manually enter or confirm the address.
      Database port	This field is automatically populated based on the selected instance and input is disabled when using the create custom data source method. When using the import from DMS data source method, you must manually enter or confirm the port.
      Database username	Required. Enter the username for the database connection.
      Database password	Required. Enter the database password. The corresponding user must have read permissions on the database.

   5. (Optional) Click Start Test to verify the database connectivity. After the test is successful, you can select the target database from the Select DB drop-down list.

   Key differences from the MySQL dataset:

   • Network type: Only private network connections are supported.
   • Data source: Supports only Alibaba Cloud PolarDB-X 2.0 instances, not self-managed databases.
   • SLR Authorization: The first time you use this dataset, you must explicitly authorize the DTS and PolarDB-X service management roles (and the DMS role if using the DMS method) in the pop-up window.

4. Click Confirm to complete the creation.

Import data

-- Create a user. Replace user1 with your actual database username.
CREATE USER 'user1'@'%' IDENTIFIED BY 'your_password_for_user1';
-- Grant privileges. Replace user1 with your actual database username.
GRANT ALL PRIVILEGES ON *.* TO 'user1'@'%' WITH GRANT OPTION;
-- Flush privileges (this only needs to be done once).
FLUSH PRIVILEGES;

[mysqld]
log-bin=mysql-bin
server-id=1
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON

-- Create a user. Replace user1 with your actual database username.
CREATE USER 'user1'@'%' IDENTIFIED BY 'your_password_for_user1';
-- Grant privileges. Replace user1 with your actual database username.
GRANT ALL PRIVILEGES ON *.* TO 'user1'@'%' WITH GRANT OPTION;
-- Flush privileges (this only needs to be done once).
FLUSH PRIVILEGES;

[mysqld]
log-bin=mysql-bin
server-id=1
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON

Dataset details

On the dataset list page, click the Details button for a dataset to view its details page. The details page contains the following tabs:

• Overview: Shows basic information about the dataset, such as its name, description, type, creation time, and storage quota, and lists its auto-generated tools.
• Files/Tables (platform-hosted type only): Manage the files or table data within the dataset.
• Tools: View the details of the dataset's auto-generated tools, including parameter descriptions and online testing.

On the dataset details page, you can expand a tool, fill in the parameters, and click the Run button to test the tool's response online.

Data synchronization rules

On the Files tab of a file dataset, click Sync Data Rules > Create Sync Rule in the upper-right corner to open the rule creation page.

1. Select a Category for the data synchronization.

2. Select a Sync Source:

   OSS

   Model Studio uses a service-linked role (SLR) to automatically synchronize data from OSS by accessing your buckets.

   Important

   • OSS synchronization uses a service-linked role (SLR) to access user buckets, eliminating the need to configure an AccessKey. Before first use, you must initialize the service-linked role and complete the authorization as prompted.
   • The object path must include the bucket name, for example, my-bucket/docs/ or my-bucket/docs/foo.md.
   • The current account must have read permissions for the target bucket. To grant access to Model Studio, you must also add the bailian-datahub-access tag with the value read to the target bucket. For more information, see Manage bucket tags.

   Fill in the following OSS object information:

   Parameter	Required	Description
   OSS region	Yes	Select the region where the OSS bucket is located. For a list of supported regions, see OSS Regions Overview.
   OSS object path	Yes	Enter the path of the OSS object to synchronize. The path must include the bucket name. For example, my-bucket/docs/ synchronizes all files in the directory, while my-bucket/docs/foo.md synchronizes a single file.
   Synchronization period	Yes	Select the frequency for automatic data synchronization, for example, every minute.
   Connection test	Yes	Click Connection Test to verify that the configured OSS path is accessible. You can submit the rule only after the test passes.
   Data tags	No	Configure tags for the synchronized data to help categorize it. Each tag can be up to 32 characters long and can contain Chinese characters, uppercase and lowercase letters, numbers, underscores (_), and hyphens (-). Press Enter after typing to add a tag.

   Lark

   Access Lark knowledge bases and documents through Lark application credentials to synchronize content to the Bailian platform. For instructions on creating a Lark custom app, see Lark custom app development process.

   Important

   • Before synchronizing Lark documents, you must create a custom app on the Lark Open Platform and obtain the App ID and App Secret.
   • You must enable read-only permissions for knowledge base, cloud document, and spreadsheet features, as well as the contact user ID read permission for the app.
   • You must fill in one of the following: Lark Knowledge Base ID, Lark Folder ID, or Lark Document ID.
   • Synchronization is supported for documents (doc), spreadsheets (xls), bitable, PPT, and other nodes in Lark knowledge bases. It is not supported for locally uploaded files.

   1. Create a Lark application and obtain credentials

   1. Log in to the Lark Open Platform and click Developer Console in the upper-right corner. In the console, click Create Custom App, fill in the App Name, App Description, and App Icon, and then click Create.
   2. On the app management page, find the Bot feature and click Add.
   3. In the left-side navigation pane, click Permission Management. On the tab that opens, click Bulk Import/Export Permissions. Copy the following JSON-formatted permission description code into the Import tab, click Next, Confirm New Permissions, and then click Request to Enable.

   {
     "scopes": {
       "tenant": [
         "aily:file:read",
         "aily:file:write",
         "application:application.app_message_stats.overview:readonly",
         "application:application:self_manage",
         "application:bot.menu:write",
         "cardkit:card:write",
         "contact:contact.base:readonly",
         "contact:user.employee_id:readonly",
         "corehr:file:download",
         "docs:document.content:read",
         "event:ip_list",
         "im:chat",
         "im:chat.access_event.bot_p2p_chat:read",
         "im:chat.members:bot_access",
         "im:message",
         "im:message.group_at_msg:readonly",
         "im:message.group_msg",
         "im:message.p2p_msg:readonly",
         "im:message:readonly",
         "im:message:send_as_bot",
         "im:resource",
         "sheets:spreadsheet",
         "wiki:wiki:readonly"
       ],
       "user": [
         "aily:file:read",
         "aily:file:write",
         "im:chat.access_event.bot_p2p_chat:read"
       ]
     }
   }
   

   4. Go to the Credentials and Basic Information page and copy the App ID and App Secret for later use.

   2. Configure knowledge base access permissions for the Lark application

   • In the Lark client, create a group chat and add the application to it.
   • As a knowledge base administrator, navigate to Knowledge Base Settings > Member Settings > Add Admin.
   • Search for the group chat containing the bot and add the group as an administrator.

   Fill in the following Lark document information:

   Parameter	Required	Description
   Lark App ID	Yes	The App ID of your Lark custom app. View instructions.
   Lark App Secret	Yes	The App Secret of your Lark custom app. Obtain it from the Credentials and Basic Information page on the Lark Open Platform.
   Lark Knowledge Base ID	One of three	You must fill in one of the following: Lark Knowledge Base ID, Lark Folder ID, or Lark Document ID. Obtain it from the Lark knowledge base URL.
   Lark Folder ID	One of three	You must fill in one of the following: Lark Knowledge Base ID, Lark Folder ID, or Lark Document ID. Obtain it from the Lark folder URL.
   Lark Document ID	One of three	You must fill in one of the following: Lark Knowledge Base ID, Lark Folder ID, or Lark Document ID. Obtain it from the Lark document URL.
   Exclude Document IDs	No	A list of Lark document IDs that you do not want to synchronize.
   Lark Document Domain	No	If your document domain is xxx.feishu.cn/wiki/..., enter xxx. This is used to populate the document source URL during upload.
   Sync Period	Yes	Select the frequency for automatic data synchronization. Options: 1 minute, 1 hour (default), and 1 day.
   Connection Test	Yes	Click Connection Test to verify that the Lark application credentials and knowledge base access are working properly. You can submit the rule only after the test passes.
   Data Tags	No	Configure tags for the synchronized data for easier categorization. Each tag can be up to 32 characters and supports Chinese, English letters (upper and lower case), numbers, underscores (_), and hyphens (-). Press Enter to confirm after typing.

   After filling in the form, click Connection Test to verify connectivity. Once the test passes, configure data tags.

   Important

   • You must set export permissions for Lark.
   • Lark does not support exporting MindNote documents.

   DingTalk

   To create a DingTalk document synchronization rule, obtain the necessary credentials by following these steps.

   1. Create a DingTalk application

   1. Log in to the DingTalk Open Platform. Select an organization for which you have developer permissions, or select an organization and then obtain developer permissions.
   2. In the top menu bar, select App Development.
   3. On the right side of the page, click Create App, fill in the application name and description, and then click Save. You are automatically taken to the application details page.

   2. Add permissions to the DingTalk application

   Before synchronizing data, on the application details page, navigate to Development Configuration > Permission Management, then search for and enable the following permissions:

   • Contact.User.Read
   • Wiki.Workspace.Read
   • Wiki.Node.Read
   • Document.Workbook.Read
   • Notable.Base.Read_All
   • qyapi_get_member

   3. Publish the application

   1. In the left-side navigation pane of the application details page, click Version Management & Release.
   2. On the right side of the page, click Create New Version and enter a version number (for example, 1.0.0) and a version description.
   3. Set the visibility range, for example, to All Employees.
   4. Click Save, and then confirm the publication.

   4. Obtain the application Client ID and Client Secret

   1. In the left-side navigation pane, click Credentials and Basic Information.
   2. Obtain the Client ID and Client Secret.

   5. Obtain the DingTalk User ID

   1. Go to the DingTalk Admin Console and log in as an organization administrator.
   2. In the left-side navigation pane, click Contacts > Member Management.
   3. In the member list, find the Employee UserID column and copy the UserID of the target user.

   NoteYou must have administrative permissions for the organization to view employee UserIDs.

   6. Enable DingTalk MCP

   Before creating a DingTalk synchronization rule, you must enable the following MCP services in the DingTalk MCP Marketplace:

   • DingTalk Docs MCP: Required to synchronize DingTalk documents.
   • DingTalk Sheets MCP: Required to synchronize DingTalk spreadsheets.

   Steps to obtain an MCP Endpoint:

   1. Go to the DingTalk MCP Marketplace, find DingTalk Docs and DingTalk Sheets, and click to open their details pages.
   2. On the details page, click to get the MCP server configuration and copy the corresponding MCP Endpoint link.
   3. Paste the copied links into the DingTalk Docs MCP Link and DingTalk Sheets MCP Link fields in the form.

   7. Obtain the DingTalk Knowledge Base ID, Folder ID, or Document ID

   You must fill in one of the following three IDs in the form to specify which content to synchronize: DingTalk Knowledge Base ID, DingTalk Folder ID, or DingTalk Document ID. You can obtain them directly from the URL:

   • DingTalk Knowledge Base ID: Go to the DingTalk knowledge base homepage, click the knowledge base you want to synchronize, and find the ID at the end of the URL in your browser's address bar.
   • DingTalk Folder ID: In the knowledge base, click the corresponding folder. The ID is at the end of the URL.
   • DingTalk Document ID: In the knowledge base, click the corresponding document. The ID is at the end of the URL.

   NoteThese three IDs determine the synchronization scope. Providing the knowledge base ID synchronizes the entire knowledge base. Providing the folder ID synchronizes that folder and its sub-documents. Providing the document ID synchronizes only that single document.

   After completing the preceding steps, fill in the DingTalk Document Information:

   Parameter	Required	Description
   DingTalk user ID	Yes	Obtain this from Contacts > Member Management in the DingTalk Admin Console.
   DingTalk application ID	Yes	Obtain this from the app management page on the DingTalk Open Platform.
   DingTalk application secret	Yes	Obtain this from the app management page on the DingTalk Open Platform.
   DingTalk knowledge base ID	One of three	You must enter one of the following: DingTalk Knowledge Base ID, DingTalk Folder ID, or DingTalk Document ID.
   DingTalk folder ID	One of three	You must enter one of the following: DingTalk Knowledge Base ID, DingTalk Folder ID, or DingTalk Document ID.
   DingTalk document ID	One of three	You must enter one of the following: DingTalk Knowledge Base ID, DingTalk Folder ID, or DingTalk Document ID.
   DingTalk Docs MCP link	Conditionally required	This value is required if you need to synchronize DingTalk documents.
   DingTalk Sheets MCP link	Conditionally required	This value is required if you need to synchronize DingTalk spreadsheets.
   Synchronization period	Yes	Select the frequency for automatic data synchronization.

   After filling in the information, click Connection Test to verify connectivity. After the test passes, configure the data tags.

   Important

   • DingTalk document synchronization consumes DingTalk development resources. View DingTalk development resource consumption and expansion options.
   • Synchronization is supported for DingTalk documents, spreadsheets, and AI sheets created in a DingTalk knowledge base. It is not supported for documents and spreadsheets uploaded from local files.

   DingTalk API call consumption

   Each synchronization consumes the API quota of the DingTalk Open Platform. We recommend that you assess your API quota in advance to ensure it is sufficient. Consumption occurs in two phases: pre-synchronization check and synchronization execution.

   Pre-synchronization check (checks connectivity, permissions, and resource existence when a task starts):

   • Account authentication: 1 call.
   • Batch document check: 1 call for every 30 documents (rounded up), plus 1 additional call.
   • AI sheet existence check: 1 call per AI sheet.
   • Knowledge base existence check: 1 call per knowledge base.

   Synchronization execution (pulls content into Model Studio):

   • Account authentication: 1 call.
   • Knowledge base content retrieval: 1 call per knowledge base.
   • AI sheet content retrieval: (Number of sheets + Number of unidirectional/bidirectional linked fields) calls per AI sheet.

   Example

   To synchronize 1 knowledge base (containing 50 documents) and 2 AI sheets (each with 3 sheets and 2 linked fields), the total API calls are calculated as follows:

   • Pre-synchronization check: 1 + (50/30 + 1) + 2 + 1 = 7 calls.
   • Synchronization execution: 1 + 1 + 2 * (3 + 2) = 12 calls.
   • Total: approximately 19 calls.

   NoteThis is a typical estimate. The actual consumption may fluctuate slightly due to factors such as API retries and error rollbacks.

   SharePoint

   Access SharePoint documents and folders through Azure AD application registration credentials to synchronize content to the Bailian platform.

   ImportantSharePoint synchronization is currently available only to whitelisted users. To request access, contact the Alibaba Cloud Bailian team.

   Prerequisites: Register an Azure AD app and obtain credentials

   1. Log in to the Azure portal, search for Microsoft Entra ID (formerly Azure Active Directory) in the top search bar, or click Manage Microsoft Entra ID on the home page.

   2. Under Manage in the left navigation pane, click App registrations, then click New registration. Fill in the application name and select the supported account type to create the app. For detailed instructions, see Microsoft documentation: Register an application.

   3. On the app Overview page, in the Essentials section, copy the Directory (tenant) ID. This is the Tenant ID in the form.

   4. In the same Essentials section, copy the Application (client) ID. This is the Client ID in the form.

   5. Under Manage in the left navigation pane, click Certificates & secrets, switch to the Client secrets tab, and click New client secret. Fill in the description and set the expiration. Copy the Value immediately after creation (it is only displayed once and cannot be viewed after leaving the page). This is the Application Secret in the form.

   6. Under Manage in the left navigation pane, click API permissions, click Add a permission, select Microsoft Graph -> Application permissions, and add the following permissions:

      • Sites.Read.All: Read files in SharePoint sites.
      • Files.Read.All: Read file content.

   7. After adding permissions, click the Grant admin consent for [tenant name] button at the top of the page to complete authorization. The permission status will change to "Granted for [tenant name]".

   Obtain SharePoint document or folder share links

   1. Open the document or folder you want to synchronize in SharePoint.
   2. Click the Share button in the upper-right corner, configure the sharing permissions, and obtain the share link.
   3. Copy the share link and paste it into the SharePoint document or folder share link list field in the form. Separate multiple links with line breaks.

   Fill in the following SharePoint document information:

   Parameter	Required	Description
   Tenant ID	Yes	The Directory (tenant) ID of your Azure AD app. Obtain it from the Essentials section on the app registration Overview page in the Azure portal.
   Client ID	Yes	The Application (client) ID of your Azure AD app. Obtain it from the Essentials section on the app registration Overview page in the Azure portal.
   Application Secret	Yes	The client secret value of your Azure AD app. Create it on the Certificates & secrets page in the Azure portal, under the Client secrets tab. Note: The secret value is only displayed once at creation time and cannot be viewed after leaving the page. Save it promptly.
   SharePoint document or folder share link list	Yes	Enter the share links of the SharePoint documents or folders you want to synchronize. Separate multiple links with line breaks. Obtain share links through the Share feature in SharePoint.
   Cloud Type	No	Select the cloud environment of your SharePoint account. Options: Global Public Cloud (default) and China 21Vianet. If your SharePoint account is deployed on the China 21Vianet cloud, select the corresponding option.
   Sync Period	Yes	Select the frequency for automatic data synchronization. Options: 1 minute, 1 hour (default), and 1 day.
   Connection Test	Yes	Click Connection Test to verify that the SharePoint credentials and share links are accessible. You can submit the rule only after the test passes.
   Data Tags	No	Configure tags for the synchronized data for easier categorization. Each tag can be up to 32 characters and supports Chinese, English letters (upper and lower case), numbers, underscores (_), and hyphens (-). Press Enter to confirm after typing.

   After filling in the form, click Connection Test to verify connectivity. Once the test passes, configure data tags.

   Yuque Sync

   Access Yuque documents and knowledge bases through a Yuque Token to synchronize content to the Bailian platform.

   Obtain a Yuque Token

   • Personal Token: Log in to the Yuque settings page, create a new token, and copy it.
   • Team Token: Navigate to Team Space -> Team Settings -> Developer -> Tokens, create a new token, and copy it.

   Fill in the following Yuque document information:

   Parameter	Required	Description
   Yuque Document/Knowledge Base URL	Yes	A list of complete Yuque document or knowledge base URLs. Separate multiple URLs with commas (,). Example: https://www.yuque.com/xionghuyi/xr0xqi,https://www.yuque.com/xionghuyi/xr0aax.
   Yuque Token	Yes	A Yuque access token. Both personal tokens and team tokens are supported. See the instructions above for how to obtain a token.
   Document Permission Level	No	The permission levels of Yuque documents to synchronize. Multiple values are supported. 0: Private, 1: Public, 2: Public within the organization. Example: to sync private and organization-public documents, enter 0,2. Default: 0,1,2 (this setting does not apply when the URL list specifies a specific document).
   Sync Period	Yes	Select the frequency for automatic data synchronization. Options: 1 minute, 1 hour (default), and 1 day.
   Connection Test	Yes	Click Connection Test to verify that the Yuque Token and URLs are accessible. You can submit the rule only after the test passes.
   Data Tags	No	Configure tags for the synchronized data for easier categorization. Each tag can be up to 32 characters and supports Chinese, English letters (upper and lower case), numbers, underscores (_), and hyphens (-). Press Enter to confirm after typing.

   After filling in the form, click Connection Test to verify connectivity. Once the test passes, configure data tags.

3. Configure the data tags, and then click Confirm to complete the creation.

After creation, the rule is displayed on the synchronization rule list page, where you can Enable or Disable it in the Actions column.

File dataset tools

After a file dataset is created, the system automatically generates the following two tools:

Manage datasets

• Edit: Click a dataset's card to open its details page. Click the Edit button in the upper-right corner to modify the dataset's name and description. You cannot change the dataset type or storage method after creation. For database datasets, the connection information (database address, port, username, and password) also cannot be changed after creation. If the database password is changed, delete the original dataset and create a new one.

• Copy: On the dataset list page or its details page, click the Copy button to create a new dataset with the same configuration.

• Delete: On the dataset list page, click the More icon (···) on the target dataset card and select Delete, or click the Delete button on the dataset details page.

  ImportantDeletion requires confirmation. You cannot undo this action.

Use a data connector in an agent

After you create a connector, add its tools to your agent. The agent can then automatically call these tools in conversations to query and reference external data.

1. On the agent configuration page, click Skills in the left pane and find the Data Connector section.

2. Click the + button in the Data Connector section. In the Select Data Connector dialog box:

   1. Browse or search for the target connector. You can filter by connector type.
   2. Click Add next to the target connector.

• Once added, the connector's tools automatically appear in the configuration list. You can click the settings button next to a tool to adjust its parameters.
• Publish the agent so it can automatically call these tools in conversations.

Use a data connector in a workflow

After you create a connector, you can add a data connector node to your workflow. This allows the workflow to call connector tools to query external data and pass the results to downstream nodes.

1. On the workflow canvas configuration page, expand the node library on the left. Under the tools category, find the data connector node and drag it onto the canvas.

2. In the Select data connector dialog box:

   • Browse or search for the target connector. You can filter by connector type, such as File, Table, MySQL, PostgreSQL, Yuque, or OSS.
   • Expand the connector, select the tool you want to use (such as searchFile or getFile), and then click OK.

3. Configure node input: In the Input section of the node configuration panel, configure the tool's input parameters. Map outputs from upstream nodes or built-in variables to the required parameters, such as fileId.

4. Connect nodes: Connect the data connector node to upstream and downstream nodes to ensure the correct data flow.

5. The node's output is a result object with two fields: content (Array<Object>, the returned content) and isError (Boolean, indicates whether an error occurred). Downstream nodes can reference this output.

References

To create a knowledge base and import content from a data source for retrieval, see Create and use a knowledge base.

For guidance on configuring and using applications, see Application types.

FAQ

Permissions and security

File parsing

Import files from OSS

{
    "Action": [
        "ram:CreateServiceLinkedRole"
    ],
    "Resource": "*",
    "Effect": "Allow",
    "Condition": {
        "StringEquals": {
            "ram:ServiceName": "datahub.sfm.aliyuncs.com"
        }
    }
}

Import MySQL data

{
    "Version": "1",
    "Statement": [
        {
            "Action": "ram:CreateServiceLinkedRole",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

SET GLOBAL wait_timeout = 1159200;

SHOW VARIABLES LIKE 'wait_timeout';

[mysqld]
wait_timeout = 1159200

SHOW VARIABLES LIKE 'wait_timeout';

[mysqld]
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON

sudo systemctl restart mysqld

SHOW VARIABLES WHERE Variable_name IN ('binlog_format', 'gtid_mode', 'enforce_gtid_consistency');