Tablestore Tunnel Service performance benchmark

Test environment

• Tablestore instance

  • Instance Type: High-performance instance

  • Region: China (Hangzhou)

  • Endpoint: Using a private IP address eliminates performance fluctuations caused by public network instability.

• Test machine configuration

  • Type: Alibaba Cloud ECS

  • Region: China (Hangzhou)

  • Instance Type: ecs.mn4.4xlarge general-purpose instance

  • Configuration:

    • CPU: 16 cores

    • Memory: 64 GB

    • NIC: Red Hat, Inc. Virtio network device

    • Operating system: CentOS 7u2

Test tools

• Load generator

  An internal Tablestore load testing tool performs concurrent batch writes. This tool leverages the BatchWrite operation from the Tablestore Java SDK.

• Pre-partitioning tool

  This tool also creates tables and performs pre-partitioning based on a specified table name and partition count.

• Rate monitoring tool

  We measured the real-time consumption rate using the Tunnel Service Java SDK. A custom callback, similar to the example below, is implemented to track the consumption rate and the total number of rows processed.

  Example

  private static final Gson GSON = new Gson();
  private static final int CAL_INTERVAL_MILLIS = 5000;
  static class PerfProcessor implements IChannelProcessor {
      private static final AtomicLong counter = new AtomicLong(0);
      private static final AtomicLong latestTs = new AtomicLong(0);
      private static final AtomicLong allCount = new AtomicLong(0);
      @Override
      public void process(ProcessRecordsInput input) {
          counter.addAndGet(input.getRecords().size());
          allCount.addAndGet(input.getRecords().size());
          if (System.currentTimeMillis() - latestTs.get() > CAL_INTERVAL_MILLIS) {
              synchronized (PerfProcessor.class) {
                  if (System.currentTimeMillis() - latestTs.get() > CAL_INTERVAL_MILLIS) {
                      long seconds = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - latestTs.get());
                      PerfElement element = new PerfElement(System.currentTimeMillis(), counter.get() / seconds, allCount.get());
                      System.out.println(GSON.toJson(element));
                      counter.set(0);
                      latestTs.set(System.currentTimeMillis());
                  }
              }
          }
      }
      @Override
      public void shutdown() {
          System.out.println("Mock shutdown");
      }
  }

Test plan

Tunnel Service processes data sequentially within a single channel to guarantee data order. However, it processes data from different channels in parallel. In an incremental data scenario, the number of channels equals the number of table partitions. Because the number of partitions is a key factor for overall throughput, this benchmark focuses on how performance scales as the number of partitions (and therefore channels) increases.

• Test scenarios

  The tests cover the following scenarios.

  • Single machine, single partition

  • Single machine, 4 partitions

  • Single machine, 8 partitions

  • Single machine, 32 partitions

  • Single machine, 64 partitions

  • Two machines, 64 partitions

  • Two machines, 128 partitions

  Note

  These tests are not designed to find the absolute performance limits of the service. They represent realistic use cases and place moderate load on the Tablestore instance.

• Test procedure

  1. Create and pre-partition a data table for each test scenario.

  2. Create a tunnel for incremental data.

  3. Use the load generator to write incremental data to the table.

  4. Use the rate monitoring tool to record the real-time QPS and observe system resource usage, including CPU and memory.

  5. Monitor the total network bandwidth consumed by data synchronization.

• Test data description

  The sample data consists of four primary key columns and one or two attribute columns. Each row is approximately 220 bytes. A 4-byte hash generates the values for the partition key (the first primary key column) to ensure an even distribution of the write workload across all partitions.

  The four primary key columns in the sample data are uid, name, class, and time. The attribute columns are col0 and col1. The name and class columns contain random strings. The time, col0, and col1 columns contain 13-digit timestamps.

Test metrics

This benchmark measures the following key performance metrics.

• QPS (rows/sec): The number of rows synchronized per second.

• Avg Latency (ms/1,000 rows): The time in milliseconds to synchronize 1,000 rows.

• CPU (cores): The total number of client-side CPU cores consumed for data synchronization.

• Mem (GB): The total physical memory on the client consumed for data synchronization.

• Bandwidth (MBps): The total network bandwidth consumed for data synchronization.

Test results

This section provides an overview of the test results for each scenario. For more information, see the Test details section.

• QPS and latency

  The following chart shows the QPS and the average time required to synchronize 1,000 rows in each scenario. The results show that QPS scales linearly with the number of partitions.

  In the single-machine, 64-partition scenario, the client's 1-gigabit NIC was saturated, limiting the QPS to 570,000. For more information, see the Test details section. After scaling out to two machines for the 64-partition scenario, the average QPS reached approximately 780,000 rows/sec. This is nearly double the rate of the single-machine, 32-partition scenario (420,000 QPS). In the final scenario with two machines and 128 partitions, the incremental synchronization QPS reached 1 million rows/sec.

  

• System resource consumption

  The following chart shows the CPU and memory consumption for each scenario. CPU usage scales linearly with the number of partitions.

  In the single-machine, single-partition scenario, CPU consumption was 0.25 cores. In the two-machine, 128-partition scenario, reaching a QPS of 1 million rows/sec consumed only 10.2 cores. For memory consumption, usage scales linearly with a small number of partitions. As the number of partitions increases to 32 and 64 on a single machine, memory consumption stabilizes at approximately 5.3 GB.

  

• Total network bandwidth consumption

  The following chart shows the total bandwidth consumed during incremental synchronization. Bandwidth consumption shows a linear relationship with the number of channels.

  In the single-machine, 64-partition scenario, the total consumed bandwidth was 125 MBps, which saturated the 1-gigabit NIC. After switching to two machines for the 64-partition data consumption, the true throughput reached 169 MBps. This is nearly double the 86 MBps consumed in the single-machine, 32-partition scenario. In the two-machine, 128-partition scenario that achieved 1 million QPS, the total throughput reached 220 MBps.

  

Test details

• Single machine, single channel: 19,000 QPS

  • Test time: 2019-01-30 17:40

  • QPS: Stable rate of approximately 19,000 rows/sec; peak rate of 21,800 rows/sec.

  • Latency: Approximately 50 ms/1,000 rows.

    {"timestamp":1548841516239,"speed":19000,"totalCount":3094000}
    {"timestamp":1548841521290,"speed":19200,"totalCount":3190000}
    {"timestamp":1548841526318,"speed":20400,"totalCount":3292000}
    {"timestamp":1548841531357,"speed":19600,"totalCount":3390000}
    {"timestamp":1548841536396,"speed":19400,"totalCount":3487000}
    {"timestamp":1548841541418,"speed":17800,"totalCount":3576000}
    {"timestamp":1548841546472,"speed":17600,"totalCount":3664000}
    {"timestamp":1548841551532,"speed":17200,"totalCount":3750000}
    {"timestamp":1548841556572,"speed":17400,"totalCount":3837000}
    {"timestamp":1548841561631,"speed":17400,"totalCount":3924000}
    {"timestamp":1548841566664,"speed":20000,"totalCount":4024000}
    {"timestamp":1548841571693,"speed":21600,"totalCount":4132000}
    {"timestamp":1548841576721,"speed":21200,"totalCount":4238000}
    {"timestamp":1548841581765,"speed":21800,"totalCount":4347000}
    {"timestamp":1548841586787,"speed":21400,"totalCount":4454000}
    {"timestamp":1548841591798,"speed":17800,"totalCount":4543000}
    {"timestamp":1548841596812,"speed":17800,"totalCount":4632000}
    {"timestamp":1548841601825,"speed":17800,"totalCount":4721000}
    {"timestamp":1548841606861,"speed":16200,"totalCount":4802000}
    {"timestamp":1548841611884,"speed":17400,"totalCount":4889000}
    {"timestamp":1548841616912,"speed":17200,"totalCount":4975000}
    {"timestamp":1548841621966,"speed":18000,"totalCount":5065000}
    {"timestamp":1548841626988,"speed":17600,"totalCount":5153000}
    {"timestamp":1548841632035,"speed":18200,"totalCount":5244000}

  • CPU utilization: Approximately 25% of a single core.

  • Memory usage: Approximately 0.4% of total physical memory, or about 0.256 GB (on a test machine with 64 GB of memory).

  • Bandwidth consumption: Approximately 4 MBps.

• Single machine, 4 partitions: 70,000 QPS

  In the Tunnel Service console, the teststream tunnel is of the incremental type and has 4 partitions. The consumption counts for each partition are 249,000, 259,000, 267,000, and 266,000, respectively. All partitions are in the Open state.

  • Test time: 2019-01-30 20:00

  • QPS: Stable rate of approximately 70,000 rows/sec; peak rate of 72,400 rows/sec.

  • Latency: Approximately 14.28 ms/1,000 rows.

    {"timestamp":1548849903425,"speed":68200,"totalCount":345000}
    {"timestamp":1548849908451,"speed":69400,"totalCount":692000}
    {"timestamp":1548849913454,"speed":71800,"totalCount":1051000}
    {"timestamp":1548849918470,"speed":70600,"totalCount":1404000}
    {"timestamp":1548849923479,"speed":69400,"totalCount":1751000}
    {"timestamp":1548849928501,"speed":71000,"totalCount":2106000}
    {"timestamp":1548849933544,"speed":70200,"totalCount":2457000}
    {"timestamp":1548849938558,"speed":71400,"totalCount":2814000}
    {"timestamp":1548849943585,"speed":71600,"totalCount":3172000}
    {"timestamp":1548849948600,"speed":70600,"totalCount":3525000}
    {"timestamp":1548849953609,"speed":71000,"totalCount":3880000}
    {"timestamp":1548849958624,"speed":68000,"totalCount":4220000}
    {"timestamp":1548849963645,"speed":69000,"totalCount":4565000}
    {"timestamp":1548849968651,"speed":70200,"totalCount":4916000}
    {"timestamp":1548849973661,"speed":70600,"totalCount":5269000}
    {"timestamp":1548849978664,"speed":72400,"totalCount":5631000}
    {"timestamp":1548849983676,"speed":68000,"totalCount":5971000}
    {"timestamp":1548849983699,"speed":68000,"totalCount":6311000}

  • CPU utilization: Approximately 70% of a single core.

  • Memory usage: Approximately 1.9% of physical memory, or about 1.1 GB (on a test machine with 64 GB of memory).

  • Bandwidth consumption: Approximately 13 MBps.

• Single machine, 8 partitions: 130,000 QPS

  The tunnel is named teststream2. All 8 partitions are of the incremental type and are in the Waiting state. The consumption count is 0, and the latest synchronization timestamp for the incremental tunnel partitions is at its initial value.

  • Test time: 2019-01-30 20:20

  • QPS: Stable rate of 130,000 rows/sec; peak rate of 141,644 rows/sec.

  • Latency: Approximately 7.69 ms/1,000 rows.

    {"timestamp":1548850971326,"speed":136000,"totalCount":688000}
    {"timestamp":1548850976329,"speed":137600,"totalCount":1376000}
    {"timestamp":1548850981335,"speed":137800,"totalCount":2065000}
    {"timestamp":1548850986351,"speed":139800,"totalCount":2764000}
    {"timestamp":1548850991360,"speed":139200,"totalCount":3460000}
    {"timestamp":1548850996362,"speed":134600,"totalCount":4133000}
    {"timestamp":1548851001377,"speed":133800,"totalCount":4802000}
    {"timestamp":1548851006389,"speed":137800,"totalCount":5491000}
    {"timestamp":1548851011390,"speed":138000,"totalCount":6181000}
    {"timestamp":1548851016412,"speed":137600,"totalCount":6869000}
    {"timestamp":1548851021417,"speed":135600,"totalCount":7547000}
    {"timestamp":1548851026418,"speed":134800,"totalCount":8221000}
    {"timestamp":1548851031420,"speed":134400,"totalCount":8893000}
    {"timestamp":1548851036430,"speed":136600,"totalCount":9576000}
    {"timestamp":1548851041443,"speed":141400,"totalCount":10283000}
    {"timestamp":1548851046452,"speed":141644,"totalCount":10991220}
    {"timestamp":1548851051455,"speed":124928,"totalCount":11615860}
    {"timestamp":1548851056456,"speed":122201,"totalCount":12226865}
    {"timestamp":1548851061466,"speed":121944,"totalCount":12836585}

  • CPU utilization: Approximately 120% of a single core.

  • Memory usage: Approximately 4.1% of total physical memory, or about 2.62 GB (on a test machine with 64 GB of memory).

  • Bandwidth consumption: Approximately 27 MBps.

• Single machine, 32 partitions: 420,000 QPS

  The tunnel is named teststream. It is an Incremental type tunnel with 32 partitions, all in the Open state. The consumption count for each partition is 0, and the latest synchronization timestamp is 1970-01-01 08:00:00.

  • Test time: 2019-01-31 15:50

  • QPS: Stable rate of 420,000 rows/sec; peak rate of 447,600 rows/sec.

  • Latency: 2.38 ms/1,000 rows.

    {"timestamp":1548921206560,"speed":401800,"totalCount":2016000}
    {"timestamp":1548921211565,"speed":435600,"totalCount":4194000}
    {"timestamp":1548921216569,"speed":440200,"totalCount":6397000}
    {"timestamp":1548921221571,"speed":439000,"totalCount":8592000}
    {"timestamp":1548921226573,"speed":440800,"totalCount":10796000}
    {"timestamp":1548921231577,"speed":437400,"totalCount":12983000}
    {"timestamp":1548921236579,"speed":421400,"totalCount":15090000}
    {"timestamp":1548921241580,"speed":434400,"totalCount":17262000}
    {"timestamp":1548921246581,"speed":445400,"totalCount":19489000}
    {"timestamp":1548921251583,"speed":447600,"totalCount":21727000}
    {"timestamp":1548921256591,"speed":447400,"totalCount":23964000}
    {"timestamp":1548921261594,"speed":440800,"totalCount":26169000}
    {"timestamp":1548921266595,"speed":425200,"totalCount":28295000}
    {"timestamp":1548921271599,"speed":408600,"totalCount":30339000}
    {"timestamp":1548921276603,"speed":403800,"totalCount":32358000}
    {"timestamp":1548921281608,"speed":405000,"totalCount":34383000}
    {"timestamp":1548921286610,"speed":403400,"totalCount":36400000}
    {"timestamp":1548921291612,"speed":409479,"totalCount":38447399}
    {"timestamp":1548921296617,"speed":400896,"totalCount":40452882}
    {"timestamp":1548921301618,"speed":391936,"totalCount":42412564}

  • CPU utilization: Approximately 450% of a single core.

  • Memory usage: Approximately 8.2%, or about 5.25 GB (on a machine with 64 GB of physical memory).

  • Bandwidth consumption: Approximately 86 MBps.

• Single machine, 64 partitions (1-gigabit NIC saturated): 570,000 QPS

  The teststream tunnel has 64 partitions. All partitions are of the Incremental type and are in the Open state. The consumption count for each partition is between approximately 1,228,941 and 1,231,572. The latest synchronization timestamps are around 2019-01-31 22:32.

  • Test time: 2019-01-31 22:10

  • QPS: Stable rate of approximately 570,000 rows/sec; peak rate of 581,400 rows/sec.

  • Latency: Approximately 1.75 ms/1,000 rows.

    {"timestamp":1548943781849,"speed":536200,"totalCount":2688000}
    {"timestamp":1548943786851,"speed":572000,"totalCount":5548000}
    {"timestamp":1548943791852,"speed":578800,"totalCount":8442000}
    {"timestamp":1548943796855,"speed":581800,"totalCount":11351000}
    {"timestamp":1548943801857,"speed":576200,"totalCount":14232000}
    {"timestamp":1548943806859,"speed":576200,"totalCount":17113000}
    {"timestamp":1548943811860,"speed":581400,"totalCount":20020000}
    {"timestamp":1548943816861,"speed":571600,"totalCount":22878000}
    {"timestamp":1548943821864,"speed":555800,"totalCount":25657000}
    {"timestamp":1548943826866,"speed":555000,"totalCount":28432000}
    {"timestamp":1548943831869,"speed":577000,"totalCount":31317000}
    {"timestamp":1548943836870,"speed":578800,"totalCount":34211000}
    {"timestamp":1548943841871,"speed":559600,"totalCount":37009000}
    {"timestamp":1548943846875,"speed":561400,"totalCount":39816000}
    {"timestamp":1548943851878,"speed":551600,"totalCount":42574000}
    {"timestamp":1548943856879,"speed":560600,"totalCount":45377000}

  • CPU utilization: Approximately 640% of a single core.

  • Memory usage: Approximately 8.4%, or about 5.376 GB.

  • Bandwidth consumption: Approximately 125 MBps (the maximum rate of the 1-gigabit NIC).

• Two machines, 64 partitions: 780,000 QPS

  The partition list for the teststream2 tunnel shows 64 partitions, which are consumed by two clients with ID prefixes Linux-6c74d- and Linux-37197-. The consumption count for each partition is approximately 1.22 to 1.23 million. All partitions are of the Incremental type and are in the Open state.

  • Test time: 2019-01-31 22:30

  • QPS: Stable rate of approximately 390,000 rows/sec on each machine, for a total stable rate of 780,000 rows/sec.

  • Latency: 1.28 ms/1,000 rows.

    {"timestamp":1548945217504,"speed":380200,"totalCount":1902000}
    {"timestamp":1548945222507,"speed":392400,"totalCount":3864000}
    {"timestamp":1548945227509,"speed":392800,"totalCount":5828000}
    {"timestamp":1548945232515,"speed":388200,"totalCount":7769000}
    {"timestamp":1548945237517,"speed":394200,"totalCount":9740000}
    {"timestamp":1548945242518,"speed":392800,"totalCount":11704000}
    {"timestamp":1548945247521,"speed":391200,"totalCount":13660000}
    {"timestamp":1548945252522,"speed":382200,"totalCount":15571000}
    {"timestamp":1548945257523,"speed":383400,"totalCount":17488000}
    {"timestamp":1548945262527,"speed":385600,"totalCount":19416000}
    {"timestamp":1548945267528,"speed":385000,"totalCount":21341000}
    {"timestamp":1548945272532,"speed":388600,"totalCount":23284000}
    {"timestamp":1548945277538,"speed":385800,"totalCount":25213000}
    {"timestamp":1548945282541,"speed":387400,"totalCount":27150000}
    {"timestamp":1548945287546,"speed":392200,"totalCount":29111000}

  • CPU utilization: Approximately 420% of a single core on each machine, for a total of 840%.

  • Memory usage: Approximately 8.2% on each machine, for a total of 16.4% (10.5 GB).

  • Total bandwidth consumption: Approximately 169 MBps. Compared with the single-machine, 64-partition scenario, this result shows that the network was the bottleneck for a single machine.

• Two machines, 128 partitions (1-gigabit NICs nearly saturated): 1 million QPS

  The teststream2 tunnel is configured for incremental data. All partitions in the partition list are in the Open state. The consumption count for each partition has reached hundreds of thousands of records, with synchronization timestamps between 2019-01-31 23:22 and 23:23.

  • Test time: 2019-01-31 23:20

  • QPS: Stable rate of approximately 500,000 rows/sec on each machine, for a total stable rate of 1 million rows/sec.

  • Latency: Approximately 1 ms/1,000 rows.

    $tail -f perf_128channel_2machine.txt
    {"timestamp":1548948013375,"speed":492400,"totalCount":27363000}
    {"timestamp":1548948018378,"speed":499800,"totalCount":29862000}
    {"timestamp":1548948023383,"speed":499800,"totalCount":32361000}
    {"timestamp":1548948028387,"speed":504400,"totalCount":34883000}
    {"timestamp":1548948033389,"speed":504200,"totalCount":37404000}
    {"timestamp":1548948038390,"speed":506800,"totalCount":39939000}
    {"timestamp":1548948043391,"speed":500800,"totalCount":42443000}
    {"timestamp":1548948048393,"speed":497400,"totalCount":44930000}
    {"timestamp":1548948053394,"speed":511800,"totalCount":47490000}
    {"timestamp":1548948058397,"speed":519600,"totalCount":50089000}
    {"timestamp":1548948063398,"speed":518800,"totalCount":52683000}
    {"timestamp":1548948068399,"speed":519600,"totalCount":55281000}
    {"timestamp":1548948073401,"speed":503800,"totalCount":57800000}

  • CPU utilization: Approximately 560% of a single core on each machine, for a total of 1020%.

  • Memory usage: Approximately 8.2% on each machine, for a total of 16.4% (10.5 GB).

  • Total bandwidth consumption: Approximately 220 MBps.

Summary

This performance benchmark reveals that for a single partition or a small number of partitions, the synchronization rate is primarily limited by server-side factors like disk read latency, with minimal resource consumption on the client. As the number of partitions increases, the overall throughput of incremental data synchronization scales linearly until a system bottleneck, such as client network bandwidth in this test, is reached. Furthermore, when a single client machine becomes a bottleneck, you can add more client machines to continue increasing the total throughput. This result demonstrates the horizontal scaling capability of Tunnel Service.