Create a data pipeline-阿里云帮助中心

Sends the putDataPipelineConfiguration operation through the do-data-pipe-line-action command to create a data pipeline (DataPipeline) rule. After the rule is created, OSS automatically calls Alibaba Cloud Model Studio to vectorize the raw data in the source bucket and write the vectors to the vector bucket.

Command syntax

ossutil api do-data-pipe-line-action --action putDataPipelineConfiguration \
  --parameters dataPipelineName=my-data-pipeline \
  --parameters role=acs:ram::<AccountId>:role/my-data-pipeline-role \
  --body file://config.xml \
  --region cn-hangzhou

Parameters

Parameter	Type	Description
--action	string	The name of the data pipeline operation to send.
--parameters	stringArray	The query parameters of the request, in k=v format. You can specify this option multiple times. For example, dataPipelineName=my-data-pipeline and role=acs:ram::<AccountId>:role/my-data-pipeline-role.
--body	string	The request body. file:// indicates that the content is read from a file. This option is required only for putDataPipelineConfiguration, and its content is the DataPipelineConfiguration XML.
--region	string	The region in which the bucket is located. For example, cn-hangzhou.
-i, --access-key-id / -k, --access-key-secret	string	The AccessKey ID and AccessKey secret used to access OSS. You can also configure them in advance by running ossutil config.

For information about the supported global command-line options, see the global command-line options supported by ossutil.

Examples

First, write the data pipeline configuration to config.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<DataPipelineConfiguration>
  <Sources>
      <InputBucket>my-bucket</InputBucket>
      <InputDataScope>All</InputDataScope>
  </Sources>
  <DataPipelineEmbeddingConfiguration>
      <EmbeddingProvider>bailian</EmbeddingProvider>
      <ApiKey>sk-xxxx</ApiKey>
      <Model>qwen2.5-vl-embedding</Model>
  </DataPipelineEmbeddingConfiguration>
  <Destination>
      <VectorBucketName>my-vector-bucket</VectorBucketName>
      <VectorIndexNames>my-index</VectorIndexNames>
  </Destination>
  <DataPipelineError>
      <ErrorMode>ignore</ErrorMode>
  </DataPipelineError>
</DataPipelineConfiguration>

Then, run the command to create the data pipeline:

ossutil api do-data-pipe-line-action --action putDataPipelineConfiguration \
  --parameters dataPipelineName=my-data-pipeline \
  --parameters role=acs:ram::<AccountId>:role/my-data-pipeline-role \
  --body file://config.xml \
  --region cn-hangzhou