Tunnel SDK FAQ

When I upload data using the Tunnel SDK, can the data be automatically distributed to partitions?

The Tunnel SDK cannot automatically distribute uploaded data to different partitions. Each upload can target only one table or a single partition within a table. For partitioned tables, you must specify the target partition. For multi-level partitioned tables, you must specify the lowest-level partition.

When I upload data using the Tunnel SDK, is there a size limit for the JAR package created from a User-Defined Function (UDF)?

The JAR package cannot exceed 10 MB. If your JAR package is larger than 10 MB, you can use the MaxCompute Tunnel Upload command to upload data.

Is there a limit on the number of partitions when I upload data using the Tunnel SDK?

A maximum of 60,000 partitions are supported. An excessive number of partitions can complicate statistics and analysis. MaxCompute limits the number of instances in a single job, and the number of job instances is closely related to the input data volume and the number of partitions. Therefore, you should evaluate your business needs and choose an appropriate partitioning policy to prevent issues caused by too many partitions.

How do I resolve the "StatusConflict" error that occurs when I upload data using the Tunnel SDK?

• Symptom

  When you upload data using the Tunnel SDK, the following error is returned.

  RequestId=20170116xxxxxxx, ErrorCode=StatusConflict, ErrorMessage=You cannot complete the specified operation under the current upload or download status. java.io.IOException: RequestId=20170116xxxxxxx, ErrorCode=StatusConflict, ErrorMessage=You cannot complete the specified operation under the current upload or download status.at com.aliyun.odps.tunnel.io.TunnelRecordWriter.close(TunnelRecordWriter.java:93)

• Cause: This error occurs when you attempt to close a RecordWriter. Possible causes are as follows:

  • You attempted to close a RecordWriter that was already closed.

  • The session corresponding to the RecordWriter was already closed.

  • The session was already committed.

• Solution: Check for the preceding causes. For example, you can print logs to check the status of the RecordWriter and the session before you commit the session. After you confirm that none of these issues are present, try to upload the data again.

How do I resolve the "Blocks Not Match" error that occurs when I upload data using the Tunnel SDK?

• Symptom:

  When you upload data using the Tunnel SDK, the following error is returned.

  ErrorCode=Local Error, ErrorMessage=Blocks not match, server: 0, tunnelServiceClient: 1
  at com.aliyun.odps.tunnel.TableTunnel$UploadSession.commit(TableTunnel.java:814)      

• Cause

  The number of blocks that the server received does not match the number of blocks specified in the commit parameters.

• Solution

  • In your code, check the number of writers opened by uploadSession.openRecordWriter(i) and the block array used at commit time. Ensure that they correspond to each other.

  • After the write operation is complete, confirm that recordWriter.close(); is called. If you directly commit the session, the number of blocks on the server-side may not match the expected number.

How do I resolve the "StatusConflict" error that occurs when I run odps tunnel recordWriter.close() after uploading 80 million records at a time?

• Symptom

  When you upload 80 million records at once, the following error is returned when you run odps tunnel recordWriter.close().

  ErrorCode=StatusConflict, ErrorMessage=You cannot complete the specified operation under the current upload or download status.

• Causes

  The session is in an invalid state because a previous operation has already closed or committed it.

• Solution

  Create a new session and upload the data again. Each partition requires a separate session. If the error is caused by multiple commits, first check whether the data was uploaded successfully. If the upload failed, retry the operation. For more information, see Multi-threaded upload example.

How can I use TunnelBufferedWriter to avoid errors when I perform batch data uploads with the Tunnel SDK?

Starting from version 0.21.3-public, the MaxCompute Java SDK includes the BufferedWriter SDK, which simplifies data uploads and provides fault tolerance. From a user's perspective, you only need to open a writer on a session and then write records. In practice, BufferedWriter first caches records in a client-side buffer. When the buffer is full, it opens an HTTP connection to upload the data.

BufferedWriter is designed to tolerate faults and ensure successful data uploads. For more information, see TunnelBufferedWriter.

How do I resolve the "You need to specify a partitionspec along with the specified table" error that occurs when I download data using the Tunnel SDK?

• Symptom

  When you download data from a partitioned table using the Tunnel SDK, the following error is returned.

  ErrorCode=MissingPartitionSpec, ErrorMessage=You need to specify a partitionspec along with the specified table.       

• Cause

  When you download data from a partitioned table using the Tunnel SDK, you must specify the values for the partition key columns. Otherwise, an error occurs.

• Solution

  • If you use the Tunnel command line in the MaxCompute client tool to export data, the client supports exporting the entire partitioned table. The data is exported to a folder.

  • If you use the Tunnel SDK to export data, first use the SDK to retrieve all partitions of the table. The following is a sample command.

    odps.tables().get(tablename) t.getPartitions()      

What programming languages does the MaxCompute Tunnel SDK support?

MaxCompute Tunnel currently supports the Java SDK.

Can a BlockId be reused?

BlockIds cannot be reused within the same UploadSession. For a single UploadSession, after you open a RecordWriter with a BlockId, write a batch of data, and then call close(), you cannot use the same BlockId to open another RecordWriter. By default, there is a maximum of 20,000 blocks, with BlockIds ranging from 0 to 19,999.

Is there a size limit for a block?

The default size of a data block uploaded to Tunnel is 100 MiB. The maximum size for a single block is 100 GB. For data larger than 64 MB, ensure that each block corresponds to one file. Files smaller than 64 MB are considered small files. An excessive number of small files can degrade performance. If you are continuously uploading a large amount of data, set the block size to between 64 MB and 256 MB. If you are performing a daily batch upload, you can set the block size to about 1 GB.

Using the new BufferedWriter simplifies uploads and helps avoid issues such as small files. For more information, see TunnelBufferedWriter.

How do I resolve a read/write timeout or an IOException?

When you upload data, a network action is triggered for every 8 KB of data written by the Writer. If no network activity occurs for 120 seconds, the server-side actively closes the connection. The Writer then becomes unavailable. To continue writing, you must open a new Writer.

We recommend that you use TunnelBufferedWriter.

When you download data, the Reader has a similar mechanism. The connection is dropped if there is no network activity for an extended period. To prevent this, run the Reader process continuously without interspersing calls to other system interfaces.