CreateCrawler-DataWorks(DataWorks)-阿里云帮助中心

Creates a metadata crawler and configures the data source, collection scope, resource group, and scheduling method.

Operation description

Scenarios

Creates a metadata crawler for a specified data source and configures the collection scope, resource group, scheduling method, and extended configurations.

Recommended workflow

Call GetCrawlerTypeCapabilities to query the crawler types and their configuration capabilities supported in the current region.
Create a crawler by using a data source that matches the Type value. Before creating a crawler, ensure that the data source and the selected resource group pass the connectivity test by calling the TestDataSourceConnectivity API to avoid creating an invalid crawler.
After the crawler is created, call RunCrawler to manually run it, or configure periodic scheduling for automatic execution.

Edition requirements

DataWorks Basic Edition or a higher edition is required.

Precautions

A successful creation only indicates that the crawler configuration has been generated. Metadata collection is not immediately executed.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

dataworks:CreateCrawler

create

*All Resource

*

None

Request syntax

POST  HTTP/1.1

Request parameters

Parameter	Type	Required	Description	Example
Name	string	Yes	The name of the metadata crawler. The name can be up to 128 characters in length.	example_crawler
DataSourceId	integer	Yes	The ID of the data source associated with the crawler. The data source must be bound to a DataWorks workspace, and the data source type must match the Type value.	12345
Type	string	Yes	The crawler type. Call GetCrawlerTypeCapabilities to query the valid values supported in the current region.	starrocks
ResourceGroupId	string	No	The ID of the Serverless 2.0 resource group used to run the collection task. Whether this parameter is required depends on the RequireResourceGroup value returned by GetCrawlerTypeCapabilities.	Serverless_res_group_1234567890123456_1234567890
Scope	object	No	The collection scope configuration. If this parameter is not specified, the DefaultScopeUnit value returned by GetCrawlerTypeCapabilities is used.
Unit	string	Yes	The collection scope granularity. Valid values are determined by the SupportedScopeUnits value returned by GetCrawlerTypeCapabilities.	DATABASE
Items	array	No	The list of database names. This parameter is supported only when Unit is set to DATABASE. A maximum of 1000 entries are allowed. Names cannot be empty or duplicated.
	string	No	A single database name. The name can be up to 256 characters in length.	example_database
ExcludeRegex	string	No	The regular expression used to exclude objects from the collection scope. This parameter is supported only when the SupportExcludeRegex value returned by GetCrawlerTypeCapabilities is true.	^tmp_.*
ScheduleConfig	object	No	The scheduling configuration. If this parameter is not specified, manual scheduling is used.
Type	string	Yes	The scheduling type. MANUAL indicates manual execution, and NORMAL indicates periodic scheduling. Data sources in the development environment support only MANUAL. Whether NORMAL is available depends on the SupportSchedule value returned by GetCrawlerTypeCapabilities.	NORMAL
CronExpress	string	No	The six-field cron expression for periodic scheduling. This parameter is required when Type is set to NORMAL. The seconds field must be 0, and the scheduling frequency cannot exceed once per hour.	0 0 2 ? * *
EnableAiComment	boolean	No	Specifies whether to enable AI metadata descriptions. This parameter is supported only when the SupportAiComment value returned by GetCrawlerTypeCapabilities is true.
Options	object	No	The extended configuration for the crawler type. The key names, value types, required fields, default values, and valid values are determined by the SupportedOptionKeys value returned by GetCrawlerTypeCapabilities.
	string	No	The value of a single extended configuration item for the crawler. The supported configuration items, value types, and value ranges are determined by the SupportedOptionKeys value returned by GetCrawlerTypeCapabilities.	v1

DataSourceId must match the Type value. Whether ResourceGroupId, Scope, ScheduleConfig, EnableAiComment, and Options are supported depends on GetCrawlerTypeCapabilities.

Response elements

Element	Type	Description	Example
	object	The response parameters.
RequestId	string	The request ID. Used to locate logs and troubleshoot issues.	9252F32F-D855-549E-8898-61CF5A733050
Success	boolean	Indicates whether the request was successful.
Id	integer	The ID of the newly created metadata crawler.	1234

Id is the ID of the newly created metadata crawler. A successful response does not indicate that metadata collection has been completed.

Examples

Success response

JSON format

{
  "RequestId": "9252F32F-D855-549E-8898-61CF5A733050",
  "Success": false,
  "Id": 1234
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.