AI 助理
文档备案控制台
官方文档
首页

数据同步问题

更新时间：
复制 MD 格式
产品详情
我的收藏

本文为您介绍数据同步方面的常见问题。

同步Kafka JSON数据到Hologres，作业运行过程中JSON Schema发生改变，如何解决？

常规做法是停止Flink作业，修改代码并调整Hologres表结构后重新上线，这样可能会导致作业数据延时以及出现不必要的错误。

针对以上的问题，实时计算Flink版做了以下优化来规避该问题：

  • 支持Kafka JSON format Schema自适应变更。即使JSON Schema发生改变，也无需停止作业或修改SQL语句，系统会自动将变更的Schema同步到Hologres。

  • 支持Kafka JSON format的类型推导，不需要在DDL语句声明JSON里面的字段类型。

  • 支持递归式展开JSON的嵌套列，例如对于如下JSON数据：{"nested": {"col": true}}列col会被展开为nested.col。

报错：Encountered change event for table xxx.xxx whose schema isn't known to this connector

  • 问题详情

    202x-xx-xx xx:xx:xx,xxx ERROR io.debezium.connector.mysql.BinlogReader    [] - Encountered change event 'Event{header=EventHeaderV4{timestamp=xxx, eventType=TABLE_MAP, serverId=xxx, headerLength=xxx, dataLength=xxx, nextPosition=xxx, flags=xxx}, data=TableMapEventData{tableId=xxx, database='xxx', table='xxx',
columnTypes=xxx, xxx..., columnMetadata=xxx,xxx,..., columnNullability={xxx,xxx,...}, eventMetadata=null}}' at offset {ts_sec=xxx, file=mysql-bin.xxx, pos=xxx,
gtids=xxx, ...server_id=xxx, event=xxx} for table xxx.xxx whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take
a new snapshot in this case.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=30946 --stop-position=31028 --verbose mysql-bin.004419
202x-xx-xx xx:xx:xx,xxx ERROR io.debezium.connector.mysql.BinlogReader    [] - Error during binlog processing. Last offset stored = null, binlog
reader near position = mysql-bin.xxx/xxx
202x-xx-xx xx:xx:xx,xxx ERROR io.debezium.connector.mysql.BinlogReader    [] - Failed due to error: Error processing binlog event
org.apache.kafka.connect.errors.ConnectException: Encountered change event for table statistic.apk info whose schema isn't known to this connector
	at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241)
	~[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:210)
	~[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:607)
	~[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1104)
	[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:955)
	[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:595)
	[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at com.github.shyiko.mysql.binlog.BinaryLogClientIMT.run(BinaryLogClient.java:839)
	[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:834) [?:1.8.0_102]
Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table xxx.xxx whose schema isn't known to this connector
	at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:875)
	~[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:849)
	~[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:590)
	~[veverica-connector-mysql-1.12-vvr-3.0.0-SNAPSHOT-jar-with-dependencies.jar:1.12-vvr-3.0.0-SNAPSHOT]
	... 5 more

  • 问题原因

    在开启CDAS或CTAS新增表读取功能时，出现该报错的原因如下：

    • 当数据库用户缺少作业中使用的某些数据库所需的权限时，可能会出现该错误。

    • 作业中配置了'debezium.snapshot.mode'='never'。因为这会导致从Binlog开头读取数据，但是Binlog开头的变更事件对应的Table Schema和当前表的Schema可能不匹配，所以会报该错。

    • 遇到Debezium解析不了的变更，例如Debezium无法解析`DEFAULT (now())`，可能出现该错误。

  • 解决方案

    • 确保数据库用户拥有作业所需的所有数据库权限（通常需要读取、写入、修改数据、创建和修改表结构、以及创建表的权限，具体所需权限详情请参见对应连接器文档）。

    • 通过'debezium.inconsistent.schema.handling.mode' = 'warn'参数避免报错，不建议使用'debezium.snapshot.mode'='never'，

    • 继续观察日志，查询io.debezium.connector.mysql.MySqlSchema WARN的log，会打印出具体无法解析的变更详情，例如Debezium解析不了`DEFAULT (now())`。

报错：Currently does not support merge StreamExecMiniBatchAssigner type ExecNode in CTAS/CDAS syntax

  • 问题详情

    在部署作业或者启动作业时报错：

    USE CATALOG `test-holo`;
CREATE TABLE IF NOT EXISTS test_tongbu
AS TABLE `test-mysql`.`flink_test`.`test_tongbu`
/*+ OPTIONS('server-id'='9021-9030') */;
问题: test-ctas
org.apache.flink.table.api.TableException: Currently does not support merge StreamExecMiniBatchAssigner type ExecNode in CTAS/CDAS syntax.
    at org.apache.flink.table.planner.plan.nodes.exec.processor.EvolvingTableSourceMergeProcessor$EvolvingNodeMergeVisitor.visit
(EvolvingTableSourceMergeProcessor.java:285)
    at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.accept(ExecNodeBase.java:249)
    at org.apache.flink.table.planner.plan.nodes.exec.visitor.ExecNodeVisitorImpl.lambda$visitInputs$0(ExecNodeVisitorImpl.java:34)
    at java.util.ArrayList.forEach(ArrayList.java:1249)
    at org.apache.flink.table.planner.plan.nodes.exec.visitor.ExecNodeVisitorImpl.visitInputs(ExecNodeVisitorImpl.java:34)
    at org.apache.flink.table.planner.plan.nodes.exec.visitor.ExecNodeVisitorImpl.visit(ExecNodeVisitorImpl.java:30)
    at org.apache.flink.table.planner.plan.nodes.exec.processor.EvolvingTableSourceMergeProcessor$EvolvingNodeMergeVisitor.visit
(EvolvingTableSourceMergeProcessor.java:263)
    at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.accept(ExecNodeBase.java:249)
    at org.apache.flink.table.planner.plan.nodes.exec.visitor.ExecNodeVisitorImpl.lambda$visitInputs$0(ExecNodeVisitorImpl.java:34)
    at java.util.ArrayList.forEach(ArrayList.java:1249)
    at org.apache.flink.table.planner.plan.nodes.exec.visitor.ExecNodeVisitorImpl.visitInputs(ExecNodeVisitorImpl.java:34)

  • 问题原因

    CTAS和CDAS作业不支持微批（MiniBatch）操作，但您的SQL流作业中却启用了table.exec.mini-batch.enabled: 'true'配置。

  • 解决方案

    在SQL流作业中去掉MiniBatch相关配置。

    • 未部署成功的SQL流作业

      在配置管理页面的作业默认配置页签的其他配置中，删除MiniBatch相关配置信息或将table.exec.mini-batch.enabled取值设为false，保存修改后重新创建并部署SQL流作业。

    • 已部署成功的SQL流作业

      如果作业已跳过部署前的深度检查并完成部署，请前往作业运维页面的部署详情页签，在运行参数配置的其他配置中，删除MiniBatch配置信息或将table.exec.mini-batch.enabled取值设为false，并重启作业。

上一篇：作业性能问题下一篇：数据正确性