Scan time series data in parallel

Use Tablestore SDK for Java to split a time series data scan into multiple tasks and run the tasks in parallel to quickly export large amounts of data from a time series table.

Prerequisites

Install the Tablestore SDK for Java and initialize a time series client.

Description

To scan time series data in parallel, call splitTimeseriesScanTask to split the scan range into independent tasks. Then, call scanTimeseriesData for each task and run the tasks in parallel. splitCountHint specifies only the expected number of tasks. The actual number is determined by the server.

public SplitTimeseriesScanTaskResponse splitTimeseriesScanTask(SplitTimeseriesScanTaskRequest request) throws TableStoreException, ClientException
public ScanTimeseriesDataResponse scanTimeseriesData(ScanTimeseriesDataRequest request) throws TableStoreException, ClientException

The following example splits the data whose measurement name is cpu in example_timeseries_table into an expected number of four tasks. The example scans the tasks by using a parallel stream and uses nextToken to retrieve all data from each task.

TimeseriesClient timeseriesClient = client.asTimeseriesClient();
String tableName = "example_timeseries_table";

SplitTimeseriesScanTaskRequest splitRequest =
        new SplitTimeseriesScanTaskRequest(tableName, "cpu", 4);
SplitTimeseriesScanTaskResponse splitResponse =
        timeseriesClient.splitTimeseriesScanTask(splitRequest);

List<TimeseriesRow> rows = splitResponse.getSplitInfos().parallelStream()
        .flatMap(splitInfo -> {
            ScanTimeseriesDataRequest scanRequest =
                    new ScanTimeseriesDataRequest(tableName);
            scanRequest.setSplitInfo(splitInfo);
            scanRequest.setLimit(5000);

            List<TimeseriesRow> splitRows = new ArrayList<>();
            do {
                ScanTimeseriesDataResponse scanResponse =
                        timeseriesClient.scanTimeseriesData(scanRequest);
                splitRows.addAll(scanResponse.getRows());
                scanRequest.setNextToken(scanResponse.getNextToken());
            } while (scanRequest.getNextToken() != null);
            return splitRows.stream();
        })
        .collect(Collectors.toList());

System.out.println(rows.size());

Important

A scan may return an empty page with a non-null nextToken. Always check whether nextToken is null to determine whether the current task is complete.

Parameters

Split a scan

SplitTimeseriesScanTaskRequest contains the following parameters.

Name	Type	Description
timeseriesTableName (required)	`String`	The time series table name.
splitCountHint (required)	`int`	The expected number of tasks. The value must be greater than `0`. The actual number of tasks is determined by the server.
measurementName (optional)	`String`	The measurement name to scan.

Scan a task

ScanTimeseriesDataRequest contains the following parameters.

Name	Type	Description
timeseriesTableName (required)	`String`	The time series table name. This value must be the same as the value used to split the scan.
splitInfo (required)	`TimeseriesScanSplitInfo`	The task information obtained by calling `SplitTimeseriesScanTaskResponse.getSplitInfos()`. Use a different task in each concurrent operation.
beginTimeInUs (optional)	`long`	The start time of the scan range, in microseconds since 1970-01-01 00:00:00 UTC. The value must be greater than or equal to `0`. When you call `setTimeRange`, specify this parameter together with `endTimeInUs`.
endTimeInUs (optional)	`long`	The end time of the scan range, in microseconds. The value must be greater than `beginTimeInUs`. When you call `setTimeRange`, specify this parameter together with `beginTimeInUs`.
fieldsToGet (optional)	`List<Pair<String, ColumnType>>`	The names and types of data fields to return. If this parameter is not specified, all data fields are returned.
limit (optional)	`int`	The maximum number of rows to return in a request. Default and maximum value: `5000`.
nextToken (optional)	`byte[]`	The pagination token for the current task. Do not specify this parameter when you first scan the task. If `nextToken` in the response is not empty, pass it to the next request for the same task.

Response

Split result

SplitTimeseriesScanTaskResponse contains the following operation-specific field.

Field	Type	Description
`splitInfos`	`List<TimeseriesScanSplitInfo>`	Call `getSplitInfos()` to obtain the task information generated by the server. Each element is used for an independent scan task.

Scan result

ScanTimeseriesDataResponse contains the following operation-specific fields.

Field	Type	Description
`rows`	`List<TimeseriesRow>`	Call `getRows()` to obtain the time series data rows returned by the current request for the task.
`nextToken`	`byte[]`	Call `getNextToken()` to obtain the token for the next page of the current task. A value of `null` indicates that the task is complete.

Time series data rows

Each element in rows[] is of the TimeseriesRow type and contains the following fields.

Field	Type	Description
`timeseriesKey`	`TimeseriesKey`	Call `getTimeseriesKey()` to obtain the time series identifiers.
`timeInUs`	`long`	Call `getTimeInUs()` to obtain the data point timestamp in microseconds.
`fields`	`SortedMap<String, ColumnValue>`	Call `getFields()` to obtain the data fields.