SDK interfaces-MaxCompute(MaxCompute)-阿里云帮助中心

The Streaming Tunnel Java SDK exposes three interfaces for writing data into MaxCompute tables: TableTunnel, StreamUploadSession, and StreamRecordPack. For full API details, see ODPS SDK Core 0.36.4-public API.

SDK behavior varies by version. The interfaces described here apply to version 0.36.4-public.

Prerequisites

To use SDK for Java to access MaxCompute Streaming Tunnel on MaxCompute Studio, add the following Maven dependency to your pom.xml:

<dependency>
  <groupId>com.aliyun.odps</groupId>
  <artifactId>odps-sdk-core</artifactId>
  <version>0.36.4-public</version>
</dependency>

Interface overview

Interface	Description
`TableTunnel`	Entry-class interface for accessing MaxCompute Tunnel. Supports connections over the Internet and virtual private cloud (VPC) of Alibaba Cloud.
`TableTunnel.StreamUploadSession`	Manages a streaming upload session to a MaxCompute table.
`TableTunnel.StreamRecordPack`	Buffers records in memory and flushes them to the server.

How it works

A typical data write follows this sequence:

Create a StreamUploadSession using TableTunnel.
Call newRecordPack() to get a StreamRecordPack.
Call append() to add records to the pack.
Call flush() to send the buffered data to the server.

After a successful flush(), the StreamRecordPack object can be reused.

StreamUploadSession

StreamUploadSession manages the lifecycle of a streaming upload session, including session metadata, record pack creation, and point-to-point mode configuration.

Method summary

Return type	Method	Description
`void`	`setP2pMode(boolean mode)`	Enables or disables point-to-point mode.
`String`	`getId()`	Returns the session ID.
`TableSchema`	`getSchema()`	Returns the table schema.
`StreamRecordPack`	`newRecordPack()`	Creates an uncompressed `StreamRecordPack`. Throws `IOException`.
`StreamRecordPack`	`newRecordPack(CompressOption compressOption)`	Creates a compressed `StreamRecordPack` using the specified compression option. Throws `IOException`, `TunnelException`.
`Record`	`newRecord()`	Creates a `Record` object.

Interface definition

public interface StreamUploadSession {

    /**
     * Enables or disables point-to-point mode.
     * @param mode
     */
    public void setP2pMode(boolean mode);

    /**
     * Returns the session ID.
     * @return Session ID
     */
    public String getId();

    /**
     * Returns the table schema.
     */
    public TableSchema getSchema();

    /**
     * Creates an uncompressed {@link StreamRecordPack}.
     * @return StreamRecordPack object
     */
    public StreamRecordPack newRecordPack() throws IOException;

    /**
     * Creates a compressed {@link StreamRecordPack}.
     * @param compressOption Compression option for data transfer
     * @return StreamRecordPack object
     */
    public StreamRecordPack newRecordPack(CompressOption compressOption) throws IOException, TunnelException;

    /**
     * Creates a {@link Record} object.
     * @return Record object
     */
    public Record newRecord();

}

StreamRecordPack

StreamRecordPack accumulates records in a memory buffer before sending them to the server. Call flush() to trigger the transfer. After a successful flush, the pack can be reused.

Method summary

Return type	Method	Description
`void`	`append(Record record)`	Appends a record to the pack. Throws `IOException`.
`long`	`getRecordCount()`	Returns the number of records in the pack.
`long`	`getDataSize()`	Returns the current data size in bytes. See the note below.
`String`	`flush()`	Sends buffered data to the server and returns a `traceId`. Throws `IOException`.
`FlushResult`	`flush(FlushOption flushOption)`	Sends buffered data to the server with the specified options and returns a `FlushResult`. Throws `IOException`.

getDataSize() may not increase after every append() call. Data is staged across multiple internal buffers before being written to the memory buffer, so the reported size changes in discrete steps rather than continuously.

Interface definition

public interface StreamRecordPack {

    /**
     * Appends a record.
     * @param record
     */
    public void append(Record record) throws IOException;

    /**
     * Returns the number of records stored in this StreamRecordPack.
     */
    public long getRecordCount();

    /**
     * Returns the size of data stored in this StreamRecordPack.
     *
     * Note: Data is staged in multiple buffers before being written to the memory buffer.
     * The reported size changes in discrete steps and may remain unchanged
     * even after appending new records.
     */
    public long getDataSize();

    /**
     * Sends buffered data to the server. The StreamRecordPack can be reused after a
     * successful flush.
     * @return traceId
     * @throws IOException
     */
    public String flush() throws IOException;

    /**
     * Sends buffered data to the server with the specified options. The StreamRecordPack
     * can be reused after a successful flush.
     * @param flushOption {@link FlushOption}
     * @return FlushResult
     * @throws IOException
     */
    public FlushResult flush(FlushOption flushOption) throws IOException;

}

Performance considerations

Pack size directly affects throughput. Small packs increase overhead per flush; larger packs amortize that cost across more data.

Without compression: keep each StreamRecordPack above 4 MB before flushing.
With compression: keep each StreamRecordPack above 1 MB before flushing.