OSS

Supported

Not supported

• Supports TXT files, which must have a two-dimensional table schema.

• Supports CSV-like files with a custom delimiter.

  Note

  Text formats (TXT and CSV) support gzip, bzip2, and zip compression.

  You cannot package multiple files into a single compressed archive.

• Supports files in ORC and Parquet formats.

• Supports reading various data types (represented as STRING), column pruning, and column constants.

• Supports recursive reads and object name filtering.

• Supports concurrent reads from multiple objects.

• The reader cannot read a single object (file) concurrently using multiple threads.

• A single compressed object does not support concurrent multi-threaded reads.

Supported

Not supported

• Supports writing data to text files that have a two-dimensional table schema. Writing files that contain binary large objects (BLOBs), such as videos and images, is not supported.

• Supports CSV-like files with a custom delimiter.

• Supports files in ORC and Parquet formats.

  Note

  In the code editor, SNAPPY compression is supported.

• Supports multi-threaded writes, where each thread writes to a separate part-file.

• Supports file rotation, where a new file is created for subsequent writes after the current file exceeds a specified size.

• Concurrent writes to a single file are not supported.

• Because OSS does not provide native data types, the OSS Writer writes all data as the STRING type to OSS objects.

• You cannot write data to an OSS bucket whose storage class is Cold Archive.

• A single object (file) cannot exceed 100 GB.

Category

Column type

Integer

LONG

String

STRING

Floating-point

DOUBLE

Boolean

BOOLEAN

Date and time

DATE

Parameter

Description

Required

Default

datasource

Specifies the name of the data source. In script mode, this value must match the name of a configured data source.

Yes

None

object

Specifies one or more objects to synchronize from OSS. You can specify objects by using explicit paths, wildcards, or dynamic parameters.

1. Configuration methods

• Explicit path

  • Basic rule: Paths must start from the root directory of the bucket and must not include the bucket name.

  • Specify a single file: Enter the full path of the file. For example: my_folder/my_file.txt.

  • Specify multiple objects: Separate the paths of multiple files or folders with a comma (,). For example: folder_a/file1.txt, folder_a/file2.txt.

• Wildcard path

  • You can use wildcards to match multiple files with a specific pattern.

  • *: Matches zero or more characters.

  • ?: Matches exactly one character.

  • Examples:

    • abc*[0-9].txt matches files such as abc0.txt, abc10.txt, and abc_test_9.txt.

    • abc?.txt matches files such as abc1.txt and abcX.txt.

• Dynamic parameter path

  • Embed scheduling parameters in the path to automate data synchronization. When the job runs, the system replaces these parameters with their actual values.

  • For example, if you configure the path as raw_data/${bizdate}/abc.txt, the job can dynamically synchronize the folder for the corresponding business date each day.

  • For more information about how to use scheduling parameters, see Scheduling Parameter Sources and Their Expressions.

2. Concurrency and performance

The configuration method directly determines data extraction concurrency:

• Single-threaded mode: When you specify a single, uncompressed file with an explicit path, the job extracts data in single-threaded mode.

• Multi-threaded mode: When you specify multiple explicit files or match multiple files with a wildcard, the job uses concurrent reads to significantly improve extraction throughput. You can configure the concurrency in the Channel settings.

Yes

None

parquetSchema

Use this parameter to define the schema for files in the Parquet file format. It applies only when fileFormat is set to parquet. After you specify the parquetSchema, ensure that the overall configuration is valid JSON.

message MessageTypeName {
required/optional, data_type, column_name;
......................;
}

"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"

No

None

column

Specifies the list of columns to read. 'type' specifies the data type in the source. 'index' specifies the zero-based index of the column in the text file. 'value' specifies a constant value for the column, which instructs the system to generate the value instead of reading it from the source file.

"column": ["*"]

"column":
    {
       "type": "long",
       "index": 0    // Reads an integer from the first column of the OSS text file.
    },
    {
       "type": "string",
       "value": "alibaba"  // Generates a constant string 'alibaba' for the current column.
    }

Yes

All columns are read as the STRING data type.

fileFormat

The file format of the source objects in OSS. Valid values are 'csv' and 'text'. Both formats support custom delimiters.

Yes

csv

fieldDelimiter

The field delimiter.

Yes

,

lineDelimiter

The line delimiter.

No

None

compress

The compression type of the source files. If this parameter is not set, files are assumed to be uncompressed. Supported types: gzip, bzip2, and zip.

No

No compression

encoding

The character encoding of the source files.

No

utf-8

nullFormat

Specifies a string in the source data that represents a null value. This is necessary because text files lack a standard representation for null values. For example:

• If you set nullFormat:"null", the literal string "null" in the source data is treated as a null value.

• If you set nullFormat:"\u0001", the string "\u0001" (an invisible character) in the source data is treated as a null value.

• If you omit the "nullFormat" parameter, the system writes the source data to the destination as-is, without any value conversion.

No

None

skipHeader

CSV-like files may have a header row that serves as a title, which needs to be skipped. By default, this row is not skipped. The skipHeader parameter is not supported in compressed file mode.

No

false

csvReaderConfig

A map of additional settings for the CsvReader used to parse CSV files. If this parameter is omitted, the CsvReader uses its default settings.

No

None

Parameter

Description

Required

Default

datasource

The name of the data source. In the code editor, this value must match the name of the configured data source.

Yes

None

object

The name of the output object in OSS. OSS uses object names to simulate a directory structure. The object name has the following conventions:

• If you set "object": "datax", the written object name starts with datax, and a random string is appended as a suffix.

• If you set "object": "cdo/datax", the written object name starts with /cdo/datax, and a random string is appended as a suffix. OSS uses the forward slash (/) as a delimiter to simulate directories.

If you do not want a random UUID suffix, configure "writeSingleObject": "true". For more information, see the description of the writeSingleObject parameter.

Yes

None

ossBlockSize

The size of an OSS part. The default value is 16 MB. When the file output format is parquet or ORC, you can configure this parameter at the same level as the object parameter.

Because an OSS multipart upload is limited to 10,000 parts, the default part size restricts the maximum file size to 160 GB. To upload larger files, you must increase the part size.

No

16

writeMode

Specifies how OSS Writer handles existing data before writing:

• truncate: Deletes all objects that match the prefix specified in the object parameter before writing new data. For example, if you set "object":"abc", all objects with the abc prefix are deleted.

• append: Does not perform any cleanup. OSS Writer writes new data, using the specified object name as a prefix and appending a random UUID to ensure a unique name. For example, if you specify the object name as DataIntegration, the actual object name is similar to DataIntegration_****_****_****.

• nonConflict: If an object whose name matches the specified object prefix already exists, the synchronization job fails. For example, if you set "object":"abc" and an object named abc123 exists, an error is reported.

Yes

None

writeSingleObject

Determines whether to write all data into a single object.

• true: Writes all data to a single object. If no data is read from the source, an empty file is not created.

• false: Writes data to multiple objects. If no data is read from the source and a header is configured, an empty file that contains only the header is created. Otherwise, a zero-byte empty file is created.

No

false

fileFormat

The file format of the output objects. The following formats are supported:

• csv: Writes data in strict CSV format. If data includes the field delimiter, it is escaped by being enclosed in double quotes (") according to standard CSV escaping rules.

• text: Writes data as plain text, using the specified field delimiter. Unlike csv, this format does not escape delimiters that appear within the data.

• parquet: If you use this file format, you must specify the parquetSchema parameter to define the data schema.

  Important

  • To write data in the Parquet format, you must switch to the code editor and configure the parquetSchema parameter. For a configuration example, see Appendix: Script Demo and Parameter Descriptions.

  • If you do not configure the parquetSchema parameter, DataWorks converts the data types from the source based on a specific policy. For more information about the conversion policy, see Appendix: Parquet data type conversion policy.

• ORC: To use this format, you must switch to the code editor.

No

text

compress

Specifies the compression format for the data files written to OSS. This parameter must be configured in the code editor.

No

None

fieldDelimiter

The field delimiter for the output data.

No

,

encoding

Specifies the character encoding for the output files.

No

utf-8

parquetSchema

Required when fileFormat is set to parquet. This parameter defines the schema of the output Parquet file. Use the following format:

message MessageTypeName {
modifier, data_type, column_name;
......................;
}

The configuration items are described as follows:

• MessageTypeName: The name of the message type.

• modifier: required indicates that the field cannot contain a null value. optional indicates that the field can contain a null value. We recommend setting all fields to optional.

• data_type: Supported Parquet data types include BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BINARY (for string types), and FIXED_LEN_BYTE_ARRAY.

Example:

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}

No

None

nullFormat

In text files, you cannot use a standard string to define a null value. The data synchronization system provides the nullFormat parameter to specify a string that represents a null value. For example, if you configure nullFormat="null" and the source data is null, the data synchronization system treats it as a null field.

No

None

header

Specifies the header to write to the output files. The value must be a JSON array of strings, such as ["id", "name", "age"].

No

None

maxFileSize (Advanced configuration, not supported in the codeless UI)

Controls the maximum size of a single output object before the file is rotated, similar to log file rotation. For multipart uploads, the part size is 10 MB, which also serves as the minimum rotation granularity. A maxFileSize value smaller than 10 MB is treated as 10 MB. An OSS InitiateMultipartUploadRequest supports a maximum of 10,000 parts.

When a file is rotated, the new object name is created by appending a sequence number (such as _1, _2, or _3) to the base object name, which already includes a prefix and a random UUID.

No

100,000

suffix (Advanced configuration, not supported in the codeless UI)

Specifies a suffix to append to the output object names. For example, if you set suffix to .csv, the final object names are similar to object-prefix_random-uuid.csv.

No

None