Dataphin管道任务将FTP数据源STRING类型数据同步到Hive数据源中DATE类型字段，当FTP中改STRING类型数据为空时，管道任务会提示脏数据

问题描述

管道任务将FTP数据源STRING类型数据同步到Hive数据源中DATE类型字段，当FTP中改STRING类型数据为空时，管道任务会提示脏数据。但是FTP中该字段为空数据正常业务场景。

2021-11-08 18:42:43.305 [0-0-0-reader] INFO  UnstructuredStorageReaderUtil - Header line RETURN_TIME^]CUST_ID^]UNIONID^]AUTH_FLAG^]AUTH_VERSION^]AUTH_TIME^]RETURN_ORDER^]REFUND_ORDER_NO^]REFUND_FEE^]STATUS^]REQUEST_TIME^]CONFIRM_TIME^]CONFIRM_BY^]CONFIRM_NAME^]AGREE_REQUEST_TIME^]AGREE_REQUEST_BY^]AGREE_REQUEST_NAME^]RETURN_TYPE^]RETURN_REASON has been skiped.
2021-11-08 18:42:43.359 [0-0-0-writer] ERROR DlinkTaskPluginCollector - 脏数据: 
{"message":"字段类型转换错误：你目标字段为[DATE]类型，实际字段值为[].","record":[{"byteSize":0,"index":0,"rawData":"","type":"STRING"},{"byteSize":32,"index":1,"rawData":"5652f243-0edb-48a8-b784-2c4a4568","type":"STRING"},{"byteSize":0,"index":2,"rawData":"","type":"STRING"},{"byteSize":1,"index":3,"rawData":"Y","type":"STRING"},{"byteSize":11,"index":4,"rawData":"1-1.0,2-1.5","type":"STRING"},{"byteSize":19,"index":5,"rawData":"2021-09-19 10:59:21","type":"STRING"},{"byteSize":19,"index":6,"rawData":"6323748241156875005","type":"STRING"},{"byteSize":19,"index":7,"rawData":6331641037041986007,"type":"LONG"},{"byteSize":1,"index":8,"rawData":0,"type":"LONG"},{"byteSize":3,"index":9,"rawData":-50,"type":"LONG"},{"byteSize":19,"index":10,"rawData":"2021-10-02 16:41:44","type":"STRING"},{"byteSize":19,"index":11,"rawData":"2021-09-23 13:27:06","type":"STRING"},{"byteSize":0,"index":12,"type":"LONG"},{"byteSize":0,"index":13,"rawData":"","type":"STRING"},{"byteSize":0,"index":14,"rawData":"","type":"STRING"},{"byteSize":0,"index":15,"type":"LONG"},{"byteSize":0,"index":16,"rawData":"","type":"STRING"},{"byteSize":1,"index":17,"rawData":"2","type":"STRING"},{"byteSize":10,"index":18,"rawData":"快递/物流一直未送到","type":"STRING"}],"type":"writer"}

FTP数据样例如下：

RETURN_TIME^]CUST_ID^]UNIONID^]AUTH_FLAG^]AUTH_VERSION^]AUTH_TIME^]RETURN_ORDER^]REFUND_ORDER_NO^]REFUND_FEE^]STATUS^]REQUEST_TIME^]CONFIRM_TIME^]CONFIRM_BY^]CONFIRM_NAME^]AGREE_REQUEST_TIME^]AGREE_REQUEST_BY^]AGREE_REQUEST_NAME^]RETURN_TYPE^]RETURN_REASON
^]5652f243-0edb-48a8-b784-2c4a4568^]^]Y^]1-1.0,2-1.5^]2021-09-19 10:59:21^]6323748241156875005^]6331641037041986007^]0^]-50^]2021-10-02 16:41:44^]2021-09-23 13:27:06^]^]^]^]^]^]2^]快递/物流一直未送到

问题原因

远程FTP文件本身不提供数据类型，该类型是DataX的定义。

从日志分析，FTP数据源中日期类型数据不能选择STRING数据类型。

解决方案

需要将FTP数据源中日期类型数据选择DATE数据类型。

适用于

• Dataphin