DocsICP FilingConsole
Documentation
Home Page DataWorks User Guide Data Integration Data source list TOS data source

TOS data source

Updated at:
Copy as MD
Product
My Favorites

The TOS data source provides the capability to read TOS files. You can use this data source to retrieve files stored in TOS, parse them, and synchronize the data to any destination data source. This topic describes the data synchronization capabilities supported by DataWorks for TOS.

Limitations

The TOS data source supports the following column types in DataWorks.

Data type

Description

STRING

Text type.

LONG

Integer type.

BYTES

Byte array. Converts the read text content into a UTF-8-encoded byte array.

BOOL

Boolean type.

DOUBLE

Floating-point type.

DATE

Date and time type. The following date and time formats are supported:

  • YYYY-MM-dd HH:mm:ss

  • yyyy-MM-dd

  • HH:mm:ss

Create a TOS data source

Before you develop a synchronization task in DataWorks, you must add the required data source to DataWorks by following the instructions in Data source configuration. You can view parameter descriptions in the DataWorks console to understand the meanings of the parameters when you add a data source.

Develop a data synchronization task

The TOS data source can only be used as the source in single-table batch synchronization tasks. The following describes the entry point and the general configuration process for data synchronization tasks.

Appendix: Script demo and parameters

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Script mode configuration. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Reader script demo

{
  "type": "job",
  "version": "2.0",
  "steps": [
    {
      "stepType": "tos",
      "parameter": {
        "datasource": "",
        "object": ["f/z/1.csv"],
        "fileFormat": "csv",
        "encoding": "utf8/gbk/...",
        "fieldDelimiter": ",",
        "useMultiCharDelimiter": true,
        "skipHeader": true,
        "compress": "zip/gzip",
        "column": [
          {
            "index": 0,
            "type": "long"
          },
          {
            "index": 1,
            "type": "boolean"
          },
          {
            "index": 2,
            "type": "double"
          },
          {
            "index": 3,
            "type": "string"
          },
          {
            "index": 4,
            "type": "date"
          }
        ]
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "stream",
      "parameter": {},
      "name": "Writer",
      "category": "writer"
    }
  ],
  "setting": {
    "errorLimit": {
      "record": "0"
    },
    "speed": {
      "concurrent": 1
    }
  },
  "order": {
    "hops": [
      {
        "from": "Reader",
        "to": "Writer"
      }
    ]
  }
}

Reader script parameters

Parameter

Description

Required

Default value

datasource

The data source name. Script mode supports adding data sources. The value of this parameter must match the name of the added data source.

Yes

None

fileFormat

The source file type. Supported types: csv, text, parquet, orc, json, and jsonl.

Yes

None

object

The file path. This parameter is used when fileFormat is set to csv, text, json, or jsonl. This parameter supports the * wildcard and can be configured as an array.

For example, to synchronize the two files a/b/1.csv and a/b/2.csv, you can configure a/b/*.csv.

Yes

Required when fileFormat is set to csv, text, json, or jsonl.

None

column

The list of columns to read. type specifies the source data type:

  • CSV/TEXT format: index specifies which column the current column comes from in the text (starting from 0), and value specifies a constant value.

  • JSON/JSONL format: Use jsonPath to specify a JSONPath expression to extract fields, and value specifies a constant value.

  • By default, you can read all data as the String type. The configuration is as follows.

    column": ["*"]

  • You can specify column information as follows.

    // CSV/TEXT format
"column":    
    {       
        "type": "long",       
        "index": 0 //Get the int field from the first column of the TOS text.
    },    
    {       
        "type": "string",       
        "value": "alibaba" //Generate the string field 'alibaba' internally from TOS as the current field.    
}
    // JSON/JSONL format
"column":    
    {       
        "name": "id",       
        "jsonPath": "$.id",       
        "type": "LONG"
    },    
    {       
        "name": "name",       
        "jsonPath": "$.user.name",       
        "type": "STRING"
    },    
    {       
        "name": "source",       
        "value": "tos",       
        "type": "STRING"    
}
Note

For the column information you specify, type is required. For the CSV/TEXT format, choose one of index or value. For the JSON/JSONL format, choose one of jsonPath or value.

Yes

All read as STRING.

fieldDelimiter

The field delimiter for reading.

Note

  • When TOS Reader reads data, you must specify a field delimiter. If not specified, the default is comma (,), which is also pre-filled in the UI configuration.

  • If the delimiter is not visible, enter the Unicode code. For example: \u001b or \u007c.

Yes

,

lineDelimiter

The row delimiter for reading.

Note

This parameter takes effect only when fileFormat is set to text.

No

None

compress

The text compression type. Empty by default (no compression). Supported compression types: gzip, bzip2, and zip.

No

No compression

encoding

The encoding used to read the file.

No

utf-8

nullFormat

Text files cannot define null (null pointer) using a standard string. The data synchronization system provides nullFormat to define which strings can be treated as null. For example:

  • If you set nullFormat:"null", this is equivalent to a "visible character". If the source data is null, data synchronization treats it as a null field.

  • If you set nullFormat:"\u0001", this is equivalent to an "invisible character". If the source data is the string "\u0001", data synchronization treats it as a null field.

  • If "nullFormat" is not specified, this is equivalent to "not configured", which means the source data is written to the destination as is, with no conversion.

No

None

skipHeader

For CSV files, use skipHeader to control whether the header is read.

  • True: Read the header when synchronizing the data source.

  • False: Do not read the header when synchronizing the data source.

Note

skipHeader is not supported in compressed file mode.

No

false

parquetSchema

Configure this parameter when reading TOS in Parquet file format. It takes effect only when fileFormat is parquet, and describes the storage type of the Parquet data. Make sure that the entire configuration conforms to JSON syntax after parquetSchema is filled in.

message MessageTypeName {
required or optional, data type, column name;
......................;
}

  • The format of parquetSchema is described as follows:

    • MessageType name: enter a name.

    • Required or not: required indicates non-null, optional indicates nullable. We recommend setting all to optional.

    • Data type: Parquet files support BOOLEAN, Int32, Int64, Int96, FLOAT, DOUBLE, BINARY (use BINARY for string type), and fixed_len_byte_array.

    • Each column line must end with a semicolon, including the last line.

  • Configuration example:

    "parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"

No

None

csvReaderConfig

The configuration parameters for reading CSV files, as a Map type. Reading CSV files uses csvReader. If not configured, default values are used.

No

None
Previous：TiDB data sourceNext：TSDB