COS data source-DataWorks(DataWorks)-阿里云帮助中心

Data types

COS data sources support the following data types.

Type	Description
STRING	Text.
LONG	Integer.
BYTES	Byte array. Text content is converted to a UTF-8 encoded byte array.
BOOL	Boolean.
DOUBLE	Floating-point number.
DATE	Date and time type. Supported formats include `YYYY-MM-dd HH:mm:ss` and `yyyy-MM-ddHH:mm:ss`.

Create a data source

Before you develop a data sync task, create a data source in DataWorks. For the procedure, see Data Source Management. Tooltips in the configuration UI provide detailed descriptions for each parameter.

Key parameters:

Parameter	Description
Data Source Name	A custom name for the data source, unique within the workspace. The name can contain letters, numbers, and underscores (_) but cannot start with a number or underscore.
Region	The region where the bucket is located. Specify the region ID. For more information, see Regions and access domains.
Bucket	The name of the COS bucket.
Endpoint	The COS endpoint. For more information, see Regions and access domains.
AccessKey ID	This is the `SecretId` on the Tencent Cloud platform. You can obtain it from the API Key Management page in the console.
AccessKey Secret	This is the `SecretKey` on the Tencent Cloud platform. You can obtain it from the API Key Management page in the console.

Develop a data sync task

For the general procedure and configuration details, see the following sections.

Single-table offline sync task

For the procedure, see Configure in codeless UI and Configure in code editor. For a full list of parameters and a script example for the code editor, see the appendix.

Appendix: Script demo and parameters

Offline sync task script

When using the code editor to configure an offline sync task, format the script parameters as required. For more information, see Configure in code editor. The following sections describe the reader parameters for this data source.

Reader script demo

{
  "type": "job",
  "version": "2.0",
  "steps": [
    {
      "stepType": "cos",
      "parameter": {
        "datasource": "",
        "object": ["f/z/1.csv"],
        "fileFormat": "csv",
        "encoding": "utf8/gbk/...",
        "fieldDelimiter": ",",
        "useMultiCharDelimiter": true,
        "lineDelimiter": "\n",
        "skipHeader": true,
        "compress": "zip/gzip",
        "column": [
          {
            "index": 0,
            "type": "long"
          },
          {
            "index": 1,
            "type": "boolean"
          },
          {
            "index": 2,
            "type": "double"
          },
          {
            "index": 3,
            "type": "string"
          },
          {
            "index": 4,
            "type": "date"
          }
        ]
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "stream",
      "parameter": {},
      "name": "Writer",
      "category": "writer"
    }
  ],
  "setting": {
    "errorLimit": {
      "record": "0"
    },
    "speed": {
      "concurrent": 1
    }
  },
  "order": {
    "hops": [
      {
        "from": "Reader",
        "to": "Writer"
      }
    ]
  }
}

Reader script parameters

Parameter	Description	Required	Default
datasource	The data source name. This value must match the name of the data source you created in DataWorks.	Yes	None
fileFormat	The source file format. Supported formats: `csv`, `text`, `parquet`, and `orc`.	Yes	None
object	The file path. Supports the asterisk (``) wildcard and arrays of paths. For example, to sync `a/b/1.csv` and `a/b/2.csv`, you can set this parameter to `a/b/.csv`.	Yes	None
column	The list of fields to read. `type` specifies the source data type, `index` specifies the zero-based column number in the source file, and `value` specifies a constant. Using `value` generates a column with the specified constant instead of reading from the source file. To read all columns as STRING, use the following configuration. `"column": [""]` To specify individual columns: `"column": { "type": "long", "index": 0 // Get a LONG field from the first column of the COS text file. }, { "type": "string", "value": "alibaba" // Generate a STRING field with the constant value "alibaba". }` Note* Each column definition must include the `type` parameter and either the `index` or `value` parameter.	Yes	All columns are read as the STRING type.
fieldDelimiter	The field separator used when reading data. You must specify a separator; if omitted, a comma (,) is used by default and is pre-filled in the UI. For invisible characters, enter the Unicode encoding, for example: `\u001b` or `\u007c`.	Yes	,
lineDelimiter	The line delimiter. This parameter is valid only when `fileFormat` is set to `text`.	No	None
compress	The compression type. By default, no compression is applied. Supported types: `gzip`, `bzip2`, and `zip`.	No	Uncompressed
encoding	The character encoding of the source files.	No	utf-8
nullFormat	A string in the source file to interpret as a null value. For example: If you set `nullFormat` to "null", the string "null" in source data is synced to the target table as a NULL value. Set `nullFormat` to `"\u0001"`, which is an invisible character. If a source field is the string `"\u0001"`, it is treated as a null field and written to the target table. If you do not specify `nullFormat`, source data is written to the destination as-is without conversion.	No	None
skipHeader	For CSV files, specifies whether to skip the first line (the header row). `true`: The first line of the file is skipped. `false`: The first line of the file is read as a data record. Note The skipHeader parameter is not supported for compressed files.	No	false
parquetSchema	The schema for reading Parquet files. Required only when fileFormat is set to parquet. The general format is: `message MessageTypeName { required\|optional DataType ColumnName; required\|optional DataType ColumnName; ...; }` `parquetSchema` format details: `MessageTypeName`: The message type name. `required` or `optional`: `required` indicates a non-null field, while `optional` allows null values. We recommend setting all fields to `optional`. `DataType`: Supported Parquet data types include `BOOLEAN`, `Int32`, `Int64`, `Int96`, `FLOAT`, `DOUBLE`, `BINARY` (use for string types), and `fixed_len_byte_array`. Each field definition must end with a semicolon (;), including the last one. Example: `{"parquetSchema": "message UserProfile { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }" }`	No	None
csvReaderConfig	Advanced parameters for the CSV reader. If not specified, default values are used.	No	None