AI 助理
文档备案控制台
官方文档
首页 表格存储 Tablestore 用户指南 数据通道与投递 通道服务 增量同步性能白皮书

增量同步性能白皮书

更新时间：
复制 MD 格式
产品详情
我的收藏

通过本文您可以了解Tunnel增量性能测试的测试环境、测试工具、测试方案、测试指标、测试结果概述以及测试细则等。

测试环境

  • 表格存储实例

    • 实例类型：高性能实例

    • 实例地域：华东1（杭州）

    • 实例地址：私网地址，避免网络的不确定性因素对测试造成的干扰

  • 测试机器配置

    • 类型：阿里云ECS

    • 区域：华东1（杭州）

    • 型号：共享通用型(mn4) ecs.mn4.4xlarge

    • 配置：

      • CPU：16核

      • 内存：64 GB

      • 网卡：Red Hat、Inc Virtio network device

      • 操作系统：CentOS 7u2

测试工具

  • 压力器

    压力器使用表格存储内部使用的压力测试工具进行数据的批量并发写入，底层基于表格存储Java SDK的BatchWrite操作完成。

  • 预分区工具

    使用表格存储内部使用的压力测试工具，配置好表名和分区数等信息，进行表格的自动创建和预分区。

  • 速率统计器

    增量的实时消费速率统计基于Tunnel Java SDK完成，在传入的Callback中加入类似的速率统计逻辑，进行速率和消费总行数的实时统计。

    示例

    private static final Gson GSON = new Gson();
private static final int CAL_INTERVAL_MILLIS = 5000;
static class PerfProcessor implements IChannelProcessor {
    private static final AtomicLong counter = new AtomicLong(0);
    private static final AtomicLong latestTs = new AtomicLong(0);
    private static final AtomicLong allCount = new AtomicLong(0);
    @Override
    public void process(ProcessRecordsInput input) {
        counter.addAndGet(input.getRecords().size());
        allCount.addAndGet(input.getRecords().size());
        if (System.currentTimeMillis() - latestTs.get() > CAL_INTERVAL_MILLIS) {
            synchronized (PerfProcessor.class) {
                if (System.currentTimeMillis() - latestTs.get() > CAL_INTERVAL_MILLIS) {
                    long seconds = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - latestTs.get());
                    PerfElement element = new PerfElement(System.currentTimeMillis(), counter.get() / seconds, allCount.get());
                    System.out.println(GSON.toJson(element));
                    counter.set(0);
                    latestTs.set(System.currentTimeMillis());
                }
            }
        }
    }
    @Override
    public void shutdown() {
        System.out.println("Mock shutdown");
    }
}

测试方案

使用Tunnel进行数据同步时，在单Channel间是串行同步（串行是为了保障用户数据的有序性），不同Channel间是相互并行的。在增量场景下，Channel数和表的分区数是相等的。由于Tunnel的整体性能和表的分区数有很大的关联性，所以在本次的性能测试中，将主要考虑不同分区数（Channel数）对于Tunnel增量的同步速率的影响。

说明

  • 分区数会随着数据规模自动增长，如您需要预先创建分区，请联系表格存储技术支持。

  • 多台机器消费是自动进行的，每台机器用相同tunnelid启动Tunnel客户端即可，详情参见数据消费框架原理。

  • 测试场景

    我们将主要测试以下场景。

    • 单机同步单分区

    • 单机同步4个分区

    • 单机同步8个分区

    • 单机同步32分区

    • 单机同步64分区

    • 2台机器同步64分区

    • 2台机器同步128分区

    说明

    上述测试场景不是产品能力的极限测试，对表格存储服务端的整体压力较小。

  • 测试步骤

    1. 创建数据表并进行预分区（不同分区数的测试都会有单独的一张表）。

    2. 创建增量通道。

    3. 使用压力器进行增量数据的写入。

    4. 使用速率统计器进行QPS的实时统计，同时观察程序占用的系统资源（CPU、内存等）。

    5. 通过监控获得增量数据同步消耗的总网络带宽。

  • 测试数据说明

    样例数据由4个主键和1~2个属性列组成，单行的大小在220字节左右。第一主键（分区键）会使用4-Byte-Hash的方式产生，这样可以确保压测数据比较均匀的写入到每个分区上。

    样例数据的 4 个主键列分别为 uid、name、class、time，属性列为 col0 和 col1。其中 name 和 class 列为随机字符串，time、col0 和 col1 列为 13 位时间戳。

测试指标

本次测试主要包含以下几项指标。

  • QPS（row）：每秒同步的数据行数。

  • Avg Latency（ms/1000行）：同步1000行数据所需的时间，单位为毫秒。

  • CPU（核）：数据同步消耗的单核CPU总数。

  • Mem（GB）：数据同步消耗的物理总内存。

  • 带宽（MBps）：数据同步消耗的总网络带宽。

说明

本次性能测试不是产品能力的极限，是从实际使用角度出发进行的性能测试。

测试结果

该部分主要概述各个场景下的指标测试结果，测试的细节可以参见测试细则部分。

  • QPS和延迟

    下图展示的是各个场景下每秒同步的数据行数和同步1000行数据所需的时间。从图中我们看出QPS的增长和分区数的增加呈线性关系。

    在本次测试中，单机同步64分区场景下，将千兆的网卡成功打爆（参见测试细则部分），导致只有57万的QPS。两台机器对64分区进行同步后，平均QPS成功达到了78万行左右，约等于单机-32分区场景下（42万）的两倍速率。而在最后的两台机器-128分区场景下，Tunnel增量同步的QPS也成功达到了100万行。

  • 系统资源消耗

    下图展示的是各个场景下CPU和内存的消耗情况，CPU基本上和分区数呈线性关系。

    在单机-单分区场景下，消耗的CPU为0.25个单核CPU。2台机器-128个分区的场景下，当同步QPS达到100万行时，消耗的CPU也仅为10.2个单核CPU。从内存消耗方面看。在分区数较少时，CPU和分区数呈线性关系，而在分区数增加较多（32个和64个）时，单机内存消耗基本维持在5.3 GB左右。

  • 网络总带宽消耗

    下图展示的是增量同步消耗的总带宽，从图中我们可以看出带宽和Channel数的线性关系（略单机-16分区场景）。

    在单机-64分区场景下，我们可以看到带宽总消耗为125 MBps，已经成功把千兆网卡打爆，而在换成2台机器-64分区进行数据消费后，我们发现64分区真正的吞吐量为169 MBps，和单机-32分区的86 MBps的两倍近乎相等。而在两台机器-128分区的100万QPS场景中，总吞吐量也达到了220 MBps。

测试细则

  • 单机单Channel：1.9万QPS

    • 测试时间：2019/1/30 17:40

    • QPS：稳定速率19000行/秒左右；峰值速率：21800行

    • Latency：50ms/1000行左右

      {"timestamp":1548841516239,"speed":19000,"totalCount":3094000}
{"timestamp":1548841521290,"speed":19200,"totalCount":3190000}
{"timestamp":1548841526318,"speed":20400,"totalCount":3292000}
{"timestamp":1548841531357,"speed":19600,"totalCount":3390000}
{"timestamp":1548841536396,"speed":19400,"totalCount":3487000}
{"timestamp":1548841541418,"speed":17800,"totalCount":3576000}
{"timestamp":1548841546472,"speed":17600,"totalCount":3664000}
{"timestamp":1548841551532,"speed":17200,"totalCount":3750000}
{"timestamp":1548841556572,"speed":17400,"totalCount":3837000}
{"timestamp":1548841561631,"speed":17400,"totalCount":3924000}
{"timestamp":1548841566664,"speed":20000,"totalCount":4024000}
{"timestamp":1548841571693,"speed":21600,"totalCount":4132000}
{"timestamp":1548841576721,"speed":21200,"totalCount":4238000}
{"timestamp":1548841581765,"speed":21800,"totalCount":4347000}
{"timestamp":1548841586787,"speed":21400,"totalCount":4454000}
{"timestamp":1548841591798,"speed":17800,"totalCount":4543000}
{"timestamp":1548841596812,"speed":17800,"totalCount":4632000}
{"timestamp":1548841601825,"speed":17800,"totalCount":4721000}
{"timestamp":1548841606861,"speed":16200,"totalCount":4802000}
{"timestamp":1548841611884,"speed":17400,"totalCount":4889000}
{"timestamp":1548841616912,"speed":17200,"totalCount":4975000}
{"timestamp":1548841621966,"speed":18000,"totalCount":5065000}
{"timestamp":1548841626988,"speed":17600,"totalCount":5153000}
{"timestamp":1548841632035,"speed":18200,"totalCount":5244000}

    • CPU占用：单核25%左右

    • 内存占用：总物理内存0.4%左右，即0.256 GB左右（测试机器内存为64 GB）

    • 网络带宽消耗：4000 KB/s左右

  • 单机4分区：7万QPS

    在通道服务控制台中，通道 teststream 为增量类型，包含 4 个分区，各分区消费统计分别为 249000、259000、267000、266000，状态均为打开。

    • 测试时间：2019/1/30 20:00

    • QPS：稳定速率70000行/秒左右，峰值速度72400行/秒

    • Latency：14.28ms/1000行左右

      {"timestamp":1548849903425,"speed":68200,"totalCount":345000}
{"timestamp":1548849908451,"speed":69400,"totalCount":692000}
{"timestamp":1548849913454,"speed":71800,"totalCount":1051000}
{"timestamp":1548849918470,"speed":70600,"totalCount":1404000}
{"timestamp":1548849923479,"speed":69400,"totalCount":1751000}
{"timestamp":1548849928501,"speed":71000,"totalCount":2106000}
{"timestamp":1548849933544,"speed":70200,"totalCount":2457000}
{"timestamp":1548849938558,"speed":71400,"totalCount":2814000}
{"timestamp":1548849943585,"speed":71600,"totalCount":3172000}
{"timestamp":1548849948600,"speed":70600,"totalCount":3525000}
{"timestamp":1548849953609,"speed":71000,"totalCount":3880000}
{"timestamp":1548849958624,"speed":68000,"totalCount":4220000}
{"timestamp":1548849963645,"speed":69000,"totalCount":4565000}
{"timestamp":1548849968651,"speed":70200,"totalCount":4916000}
{"timestamp":1548849973661,"speed":70600,"totalCount":5269000}
{"timestamp":1548849978664,"speed":72400,"totalCount":5631000}
{"timestamp":1548849983676,"speed":68000,"totalCount":5971000}
{"timestamp":1548849983699,"speed":68000,"totalCount":6311000}

    • CPU占用：单核70%左右

    • 内存占用：物理内存1.9%左右，即1.1 GB左右。（测试机器内存为64 GB）

    • 网络带宽消耗：13 MBps左右

  • 单机8分区：13万QPS

    通道名为 teststream2，8 个分区均为增量类型，状态为等待，消费统计为 0，增量通道分区最新同步时间为初始值。

    • 测试时间：2019/1/30 20:20

    • QPS：稳定速率130000行/秒，峰值速率141644行/秒

    • Latency：7.69ms/1000行左右

      {"timestamp":1548850971326,"speed":136000,"totalCount":688000}
{"timestamp":1548850976329,"speed":137600,"totalCount":1376000}
{"timestamp":1548850981335,"speed":137800,"totalCount":2065000}
{"timestamp":1548850986351,"speed":139800,"totalCount":2764000}
{"timestamp":1548850991360,"speed":139200,"totalCount":3460000}
{"timestamp":1548850996362,"speed":134600,"totalCount":4133000}
{"timestamp":1548851001377,"speed":133800,"totalCount":4802000}
{"timestamp":1548851006389,"speed":137800,"totalCount":5491000}
{"timestamp":1548851011390,"speed":138000,"totalCount":6181000}
{"timestamp":1548851016412,"speed":137600,"totalCount":6869000}
{"timestamp":1548851021417,"speed":135600,"totalCount":7547000}
{"timestamp":1548851026418,"speed":134800,"totalCount":8221000}
{"timestamp":1548851031420,"speed":134400,"totalCount":8893000}
{"timestamp":1548851036430,"speed":136600,"totalCount":9576000}
{"timestamp":1548851041443,"speed":141400,"totalCount":10283000}
{"timestamp":1548851046452,"speed":141644,"totalCount":10991220}
{"timestamp":1548851051455,"speed":124928,"totalCount":11615860}
{"timestamp":1548851056456,"speed":122201,"totalCount":12226865}
{"timestamp":1548851061466,"speed":121944,"totalCount":12836585}

    • CPU占用：单核120%左右

    • 内存占用：物理总内存4.1%左右，即2.62 GB左右（测试机器内存为64 GB）

    • 消耗网络带宽：27 MBps左右

  • 单机32分区：42万QPS

    通道名为 teststream，通道类型为增量，32 个通道分区状态均为打开，消费统计均为 0，增量通道最新同步时间为 1970-01-01 08:00:00。

    • 测试时间：2019/1/31 15:50

    • QPS：稳定速率42万行/秒，峰值速率447600行/秒

    • Latency：2.38ms/1000行

      {"timestamp":1548921206560,"speed":401800,"totalCount":2016000}
{"timestamp":1548921211565,"speed":435600,"totalCount":4194000}
{"timestamp":1548921216569,"speed":440200,"totalCount":6397000}
{"timestamp":1548921221571,"speed":439000,"totalCount":8592000}
{"timestamp":1548921226573,"speed":440800,"totalCount":10796000}
{"timestamp":1548921231577,"speed":437400,"totalCount":12983000}
{"timestamp":1548921236579,"speed":421400,"totalCount":15090000}
{"timestamp":1548921241580,"speed":434400,"totalCount":17262000}
{"timestamp":1548921246581,"speed":445400,"totalCount":19489000}
{"timestamp":1548921251583,"speed":447600,"totalCount":21727000}
{"timestamp":1548921256591,"speed":447400,"totalCount":23964000}
{"timestamp":1548921261594,"speed":440800,"totalCount":26169000}
{"timestamp":1548921266595,"speed":425200,"totalCount":28295000}
{"timestamp":1548921271599,"speed":408600,"totalCount":30339000}
{"timestamp":1548921276603,"speed":403800,"totalCount":32358000}
{"timestamp":1548921281608,"speed":405000,"totalCount":34383000}
{"timestamp":1548921286610,"speed":403400,"totalCount":36400000}
{"timestamp":1548921291612,"speed":409479,"totalCount":38447399}
{"timestamp":1548921296617,"speed":400896,"totalCount":40452882}
{"timestamp":1548921301618,"speed":391936,"totalCount":42412564}

    • CPU占用：单核450%左右

    • 内存占用：8.2%左右，即5.25 GB左右（物理内存64 GB）

    • 增量数据消耗网络带宽：86 MBps左右

  • 单机64分区（千兆网卡被打满）：57万QPS

    通道teststream下共64个分区，所有分区类型为增量、状态为打开，各分区消费统计约1228941~1231572条，最新同步时间在2019-01-31 22:32附近。

    • 测试时间：2019/1/31 22:10

    • QPS：稳定速率57万行/秒左右，峰值速率581400行/秒

    • Latency：1.75ms/1000行左右

      {"timestamp":1548943781849,"speed":536200,"totalCount":2688000}
{"timestamp":1548943786851,"speed":572000,"totalCount":5548000}
{"timestamp":1548943791852,"speed":578800,"totalCount":8442000}
{"timestamp":1548943796855,"speed":581800,"totalCount":11351000}
{"timestamp":1548943801857,"speed":576200,"totalCount":14232000}
{"timestamp":1548943806859,"speed":576200,"totalCount":17113000}
{"timestamp":1548943811860,"speed":581400,"totalCount":20020000}
{"timestamp":1548943816861,"speed":571600,"totalCount":22878000}
{"timestamp":1548943821864,"speed":555800,"totalCount":25657000}
{"timestamp":1548943826866,"speed":555000,"totalCount":28432000}
{"timestamp":1548943831869,"speed":577000,"totalCount":31317000}
{"timestamp":1548943836870,"speed":578800,"totalCount":34211000}
{"timestamp":1548943841871,"speed":559600,"totalCount":37009000}
{"timestamp":1548943846875,"speed":561400,"totalCount":39816000}
{"timestamp":1548943851878,"speed":551600,"totalCount":42574000}
{"timestamp":1548943856879,"speed":560600,"totalCount":45377000}

    • CPU占用：单核640%左右

    • 内存占用：8.4%左右，即5.376 GB左右

    • 增量数据消耗网络带宽：125 MBps左右（达到千兆网卡的速率极限）

  • 2台机器共同消费64分区：78万QPS

    通道 teststream2 的分区列表显示共 64 个分区，由两个客户端（ID 前缀分别为 Linux-6c74d- 和 Linux-37197-）共同消费，每个分区消费统计约 122～123 万条，类型均为增量，状态均为打开。

    • 测试时间：2018/1/31 22:30

    • QPS：每台稳定速率在39万行/秒左右，总的稳定速率在78万行/秒左右

    • Latency：1.28ms/1000行

      {"timestamp":1548945217504,"speed":380200,"totalCount":1902000}
{"timestamp":1548945222507,"speed":392400,"totalCount":3864000}
{"timestamp":1548945227509,"speed":392800,"totalCount":5828000}
{"timestamp":1548945232515,"speed":388200,"totalCount":7769000}
{"timestamp":1548945237517,"speed":394200,"totalCount":9740000}
{"timestamp":1548945242518,"speed":392800,"totalCount":11704000}
{"timestamp":1548945247521,"speed":391200,"totalCount":13660000}
{"timestamp":1548945252522,"speed":382200,"totalCount":15571000}
{"timestamp":1548945257523,"speed":383400,"totalCount":17488000}
{"timestamp":1548945262527,"speed":385600,"totalCount":19416000}
{"timestamp":1548945267528,"speed":385000,"totalCount":21341000}
{"timestamp":1548945272532,"speed":388600,"totalCount":23284000}
{"timestamp":1548945277538,"speed":385800,"totalCount":25213000}
{"timestamp":1548945282541,"speed":387400,"totalCount":27150000}
{"timestamp":1548945287546,"speed":392200,"totalCount":29111000}

    • CPU占用：每台单核420%左右，共单核840%

    • 内存占用：每台8.2%左右，共16.4%（10.5 GB）

    • 增量数据消耗总网络带宽：169 MBps左右（和单机64分区对比，可以看出单机的网络已经成为瓶颈）

  • 2台机器共同消费128分区（两台千兆机器的网卡近乎被打满）：100万QPS

    通道teststream2为增量类型，通道分区列表中所有分区状态均为打开，每个分区的消费统计达数十万条记录，增量同步时间集中在2019-01-31 23:22至23:23之间。

    • 测试时间：2018/1/31 23:20

    • QPS：每台稳定速率在50万行/秒左右，总的稳定速率在100万行/秒左右

    • Latency：1ms/1000行左右

      $tail -f perf_128channel_2machine.txt
{"timestamp":1548948013375,"speed":492400,"totalCount":27363000}
{"timestamp":1548948018378,"speed":499800,"totalCount":29862000}
{"timestamp":1548948023383,"speed":499800,"totalCount":32361000}
{"timestamp":1548948028387,"speed":504400,"totalCount":34883000}
{"timestamp":1548948033389,"speed":504200,"totalCount":37404000}
{"timestamp":1548948038390,"speed":506800,"totalCount":39939000}
{"timestamp":1548948043391,"speed":500800,"totalCount":42443000}
{"timestamp":1548948048393,"speed":497400,"totalCount":44930000}
{"timestamp":1548948053394,"speed":511800,"totalCount":47490000}
{"timestamp":1548948058397,"speed":519600,"totalCount":50089000}
{"timestamp":1548948063398,"speed":518800,"totalCount":52683000}
{"timestamp":1548948068399,"speed":519600,"totalCount":55281000}
{"timestamp":1548948073401,"speed":503800,"totalCount":57800000}

    • CPU占用：每台单核560%左右，两台共计单核1020%

    • 内存占用：每台8.2%左右，共16.4%（10.5 GB）

    • 增量数据消耗总网络带宽：220 MBps左右

总结

通过这次对于增量性能的实际测试，我们发现了单分区（或分区数较少）的速率主要取决于服务器端的读盘等延迟，本身机器资源的消耗很小。而随着分区数增长，Tunnel增量的整体吞吐也进行了线性的增长直至达到系统的瓶颈（本文中是网络带宽）。最后，在单机资源被打满的情况下，我们也可以通过添加新的机器资源进一步的提升系统整体的吞吐量，有效的验证了Tunnel具备良好的水平扩展性。

上一篇：错误处理下一篇：数据湖投递