CreateFormationCrawler

Operation description

• This operation creates a Formation Crawler metadata discovery task in an AnalyticDB for MySQL instance.

• The created task configuration is not executed immediately. Call StartFormationCrawler to start the task.

• The CrawlerInfo field is a JSON string that contains the core configuration of the task, such as the target database name and data source type.

• Some parameters, such as classifiers and frequency, require double JSON encoding.

• The database name specified in dbName is automatically converted to lowercase by the server.

• schemaChangePolicy is required. You must specify both updateRule and deleteRule.

• Use the RUN_ON_DEMAND scheduling mode to avoid unnecessary repeated scans.

• Make sure the product name is adb and the endpoint format is adb.{regionId}.aliyuncs.com.

• After the task is created, manually call StartFormationCrawler to trigger the first metadata discovery.

CrawlerInfo field details:

sourceType (String, required) The source type identifier. Set this to OSSWAREHOUSE.

dbName (String, required) The name of the target database. The crawled table metadata is written to this database.

sourceInfo (Object, required) The source information container object. For the OSSWAREHOUSE type, this object must contain the ossSourceInfo sub-object. Otherwise, validation fails.

• sourceInfo.ossSourceInfo (Object, required) The specific configuration for the OSS source. This object contains the following fields:

  • sourceMode (String, required): The source mode. Set this to WAREHOUSE for the OSSWAREHOUSE type.

  • ossLocations (List, required): The list of OSS paths. Each path must start with oss:// and end with /, such as oss://my-bucket/my-db/.

  • exclusions (List, optional): The list of path prefixes to exclude. Matched paths are not crawled. Set this to [] if no exclusions are needed.

  • inclusions (List, optional): The list of path prefixes to include. This limits the crawling scope. Set this to [] if no inclusions are needed.

classifiers (List, required) The data format classifiers that declare the file formats in the OSS location. Valid values: JSON, PARQUET, CSV, ORC, AVRO, and ICEBERG (case-insensitive). If iceberg is included, the task is identified as an Iceberg subtype. Set this to [] for automatic format detection.

schemaChangePolicy (Object, required) The schema change policy object that controls how table schema changes are handled during crawling.

• updateRule (String, required): The schema update rule. Valid values:

  • UPDATE_SCHEMA_DEFINE: Fully updates the schema. This supports adding columns, modifying column names and types, and dropping and recreating tables when necessary.

  • ONLY_ADD_COLUMN: Default value. Only appends new columns without modifying or deleting existing columns.

  • IGNORE: Completely ignores schema changes and does not generate any DDL. For the OSSDataWarehouse type, partition changes are still processed.

• deleteRule (String, optional): The schema deletion rule. Typically set to IGNORE.

frequency (Object, required) The scheduling frequency configuration. If the scheduling type is not on-demand, the cron expression is validated for legality and frequency limits. This object contains the following fields:

• type (String): The scheduling type, such as monthly, daily, or run_on_demand (on-demand execution, set cron to "").

• cron (String): The cron expression.

configuration (Map, optional) The extended configuration parameters for crawling behavior.

• adb.crawler.csv.columns.specify.delimiter.char Specifies the field delimiter for CSV files. Default value: auto (automatic delimiter detection). Valid values: comma ,, tab \t, pipe |, semicolon ;, space, or auto. When the value is auto or not set, automatic detection is used. Otherwise, the first character of the configured string is used as the delimiter.

• adb.crawler.csv.columns.specify.quote.char Specifies the quote character for field values (used to wrap values that contain special characters). Default value: auto (automatic quote character detection). Valid values: double quote ", single quote ', or auto. The behavior is the same as the delimiter: automatic detection is used when the value is auto or not set. Otherwise, the first character is used.

• adb.crawler.csv.columns.specify.header.mode Specifies the header identification mode. Default value: HASHEAD. Valid values: DETECTHEAD (detection mode, automatically determines whether the first row is a header), HASHEAD (the first row contains column names), and NOHEAD (no header, column names are automatically generated by the system).

• adb.crawler.csv.columns.specify.header.columns Specifies the header column names manually. Separate multiple column names with commas, such as id,name,age. Default value: empty string (not specified, the system processes the header based on header.mode).

• adb.crawler.csv.columns.allow.single.column Specifies whether to allow the identification of CSV files with only a single field. Default value: false (single-column files are not allowed). Valid values: true and false.