Overview of Data Transmission Service

Usage classification

• Commands

  You can run commands only using the MaxCompute client (odpscmd).

• Batch operations

  Tunnel lets you perform offline batch operations to upload and download data. These operations include uploading and downloading data for a single table and downloading data from a query instance.

• Streaming operations

  Stream Tunnel lets you write streaming data to tables in micro-batches.

Service architecture

For more information about shared resource groups and subscription dedicated resource groups for Data Transmission Service, see Purchase and use dedicated Data Transmission Service resource groups.

Data Transmission Service Limits

• Limits on batch data channels

  • Batch data upload

    Limit	Limitations
    UploadSession lifecycle	24 hours
    Number of blocks written per UploadSession	20,000
    Write speed per block	10 MB/s
    Data volume per block	100 GB
    Number of UploadSessions created per table	500 every 5 minutes
    Number of blocks written per table	500 every 5 minutes
    Number of concurrent UploadSession commits per table	32
    Number of commits per table	75 every 15 seconds
    Number of concurrent block writes	Limited by the number of concurrent slots. A single block write occupies one slot.
    Concurrent writes	MaxCompute ensures concurrent writes based on atomicity, consistency, isolation, and durability (ACID). For more information about ACID semantics, see ACID semantics.

  • Batch data download

    Limitations	Limitations
    DownloadSession lifecycle	24 hours
    InstanceDownloadSession lifecycle	24 hours, limited by the instance lifetime.
    Number of InstanceDownloadSessions created per project	200 every 5 minutes
    Number of DownloadSessions created per table	200 every 5 minutes
    Speed per download request	10 MB/s
    Number of concurrent DownloadSession creations	Limited by the number of concurrent slots. A single DownloadSession creation occupies one slot.
    Number of concurrent InstanceDownloadSession creations	Limited by the number of concurrent slots. A single InstanceDownloadSession creation occupies one slot.
    Number of concurrent download requests	Limited by the number of concurrent slots. A single data download request occupies one slot.

  Note

  Batch data supports the upsert feature for Delta tables. The limits are as follows:

  • UpsertSession lifecycle: 24 hours (h).

  • Maximum write speed for an UpsertSession: Number of buckets in the table or partition × 10 MB/s.

  • Maximum slot quota usage for an UpsertSession: Number of buckets in the table or partition.

  • UpsertSession commit frequency: You can commit data to each partition of a Delta table only once per minute (min). If the commit interval for a partition is less than 1 minute, the system returns the following error message: ErrorCode=FlowExceeded, ErrorMessage=CommitUpsert QPS Quota exceeded.

• Limits on streaming data channels

  Limits	Limitations
  Write speed per slot	10 MB/s
  Number of concurrent partitions written to per table	64
  Maximum number of available slots per partition	32
  Number of concurrent flushes	Limited by the number of concurrent slots. A single flush occupies one slot.

• Data upload limits

  • The size of each field cannot exceed its limit. For more information, see Data type versions.

    Note

    The size of a STRING field cannot exceed 8 MB.

  • During an upload, multiple data records are packaged for transmission.

• Network limits for Data Transmission Service (dedicated resource groups)

  • Only VPC access is supported. Public network access is not supported.

  • Only data transmission within the same region is supported. Cross-region data transmission is not supported.

Information on shared Data Transmission Service resource groups

Valid status codes for Data Transmission Service

• Retry policy for failed requests

  • After a request fails, the client must wait for a period of time before retrying.

  • The wait time for consecutive failed requests must increase exponentially, starting from a minimum of 1 second. For example: 1 s, 2 s, 4 s, 8 s, 16 s, 32 s, and so on.

• Duplicate requests

  • The URL is the same (URI and URI parameters).

  • Consecutive requests are sent from the same client IP address.

• Valid requests

  A request that returns a valid status code and follows the retry policy.

• Invalid requests

  A request that returns a valid status code but does not follow the retry policy.

  Note

  The Service-level agreement (SLA) does not cover invalid requests.

• Attack requests

  • Requests that do not follow the retry policy for throttle status codes 429 and 503.

  • For attack requests, the service isolates the client IP address, UID, and project that initiated the attack. The isolated object can no longer access the service.

  Note

  The SLA does not cover attack requests.

FAQ