Dataphin管道任务将Oracle数据集成到Maxcompute,运行报“java.lang.NullPointerException: null、[DlinkTrans - 字段计算_1] ERROR DlinkTaskPluginCollector - 脏数据"错误

问题描述

配置管道任务将oracle数据库集成到Maxcompute数据库,使用“字段计算”组件中md5函数将字段CUST_NO进行加密之后,任务运行报空指针和脏数据的错误,具体异常堆栈信息如下:

2021-09-07 17:40:10.446 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select BIZ_DT,CUST_NO,CUST_NAME,TOT_AST,CRD_CL_AST,NET_AST,STK_MTK_VAL,FND_MTK_VAL,BOND_MTK_VAL,MNY_PROD,CASH,OTH,AST_BEGIN,AST_END,CPTL_FLW_IN,CPTL_FLW_OUT,SEC_FLW_IN,SEC_FLW_OUT,AST_INC,TRD_TOT,PRFT_BAL,STK_PRFT_BAL,FND_PRFT_BAL,BOND_PRFT_BAL,CASH_PRFT_BAL,PROD_PRFT_BAL,OTH_PRFT_BAL,YLD_RATE,STK_YLD_RATE,FND_YLD_RATE,BOND_YLD_RATE,CASH_YLD_RATE,PROD_YLD_RATE,OTH_YLD_RATE,AVG_HOLD_RATE,MAX_DRAWDOWN,SUCS_RATE,WAVE_RATE,TURN_RATE,YLD_RATE_RANK,
SEC_CNT,SEC_PAYOF_CNT,SEC_LOSS_CNT,SEC_NB_CNT,SEC_SELL_CNT,TRD_TMS,SEC_BUY_TMS,SEC_SELL_TMS,STK_TRD_CNT,CHA_HOLD_DAYS,OTH_AST,STK_SHOW,BON_SHOW,CASH_SHOW,PROD_SHOW,OTH_SHOW,BEGIN_NET_AST from "ASS_PROD".T_FIN_C_D_AST_ANLS_CRD 
] jdbcUrl:[jdbc:oracle:thin:@//IP:port/schema].
2021-09-07 17:40:10.477 [0-1-1-writer] INFO  OdpsWriter$Task - task uploadId:[202109071740108ec66f0a017a200a].
2021-09-07 17:40:10.521 [0-0-0-writer] INFO  OdpsWriter$Task - task uploadId:[20210907174010cec66f0a00cc3112].
2021-09-07 17:40:10.521 [0-2-2-writer] INFO  OdpsWriter$Task - task uploadId:[20210907174010cec66f0a00cc3113].
2021-09-07 17:40:10.579 [DlinkTrans - 字段计算_1] ERROR DlinkTaskPluginCollector - 
java.lang.NullPointerException: null
 at com.alibaba.datax.plugin.trans.calculation.functions.strings.Md5Function.call(Md5Function.java:30) ~[na:na]
 at Script_1631007610397_0/353509774.execute0(Unknown Source) ~[na:na]
 at com.googlecode.aviator.ClassExpression.execute(ClassExpression.java:73) ~[aviator-4.2.7.jar:na]
 at com.alibaba.datax.plugin.trans.calculation.CalculationTrans.calculateColumn(CalculationTrans.java:194) ~[na:na]
 at com.alibaba.datax.plugin.trans.calculation.CalculationTrans.evaluate(CalculationTrans.java:163) ~[na:na]
 at com.alibaba.dt.dlink.core.trans.adaptor.TransStepAdaptor.processRow(TransStepAdaptor.java:159) ~[dlink-engine-0.0.1-SNAPSHOT.jar:na]
 at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62) [kettle-engine-8.0.0.0-28.jar:8.0.0.0-28]
 at java.lang.Thread.run(Thread.java:882) [na:1.8.0_152]
2021-09-07 17:40:10.604 [DlinkTrans - 字段计算_1] ERROR DlinkTaskPluginCollector - 脏数据: 
{"record":[{"byteSize":8,"index":0,"rawData":"20180102","type":"STRING"},{"byteSize":8,"
index":1,"rawData":"XXXXXX","type":"STRING"},{"byteSize":3,"index":2,"rawData":"XXX","type":"STRING"},{"byteSize":7,"index":3,"rawData":"XXXXXX","type":"DOUBLE"},{"byteSize":9,"index":4,"rawData":"-XXXXXX","type":"DOUBLE"},{"byteSize":8,"index":5,"rawData":"XXXXX","type":"DOUBLE"},{"byteSize":5,"index":6,"rawData":"XXXXX","type":"DOUBLE"},{"byteSize":1,"index":7,"rawData":"XXXXXX","type":"DOUBLE"},{"byteSize":1,"index":8,"rawData":"XXXX","type":"DOUBLE"},{"byteSize":1,"index":9,"rawData":"XXXX","type":"DOUBLE"},{"byteSize":7,"index":10,"rawData":"XXXXX","type":"DOUBLE"},{"byteSize":5,"index":11,"rawData":"XXXX","type":"DOUBLE"},{"byteSize":7,"index":12,"rawData":"XXXX","type":"DOUBLE"},{"byteSize":7,"index":13,"rawData":"XXXX","type":"DOUBLE"},{"byteSize":1,"index":14,"rawData":"XXXX","type":"DOUBLE"},{"byteSize":1,"index":15,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":16,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":17,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":18,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":19,"rawData":"0","type":"DOUBLE"},{"byteSize":6,"index":20,"rawData":"-76.59","type":"DOUBLE"},{"byteSize":3,"index":21,"rawData":"-66","type":"DOUBLE"},{"byteSize":1,"index":22,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":23,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":24,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":25,"rawData":"0","type":"DOUBLE"},{"byteSize":6,"index":26,"rawData":"-10.59","type":"DOUBLE"},{"byteSize":6,"index":27,"rawData":"-.1907","type":"DOUBLE"},{"byteSize":6,"index":28,"rawData":"-.1819","type":"DOUBLE"},{"byteSize":1,"index":29,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":30,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":31,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":32,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":33,"rawData":"0","type":"DOUBLE"},{"byteSize":7,"index":34,"rawData":"70.7568","type":"DOUBLE"},{"byteSize":1,"index":35,"rawData":"0","type":"DOUBLE"},{"byteSize":7,"index":36,"rawData":"33.3333","type":"DOUBLE"},{"byteSize":7,"index":37,"rawData":"-2.0358","type":"DOUBLE"},{"byteSize":1,"index":38,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":39,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":40,"rawData":"3","type":"DOUBLE"},{"byteSize":1,"index":41,"rawData":"1","type":"DOUBLE"},{"byteSize":1,"index":42,"rawData":"2","type":"DOUBLE"},{"byteSize":1,"index":43,"rawData":"3","type":"DOUBLE"},{"byteSize":1,"index":44,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":45,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":46,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":47,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":48,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":49,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":50,"rawData":"0","type":"DOUBLE"},{"byteSize":1,"index":51,"rawData":"1","type":"STRING"},{"byteSize":1,"index":52,"rawData":"0","type":"STRING"},{"byteSize":1,"index":53,"rawData":"0","type":"STRING"},{"byteSize":1,"index":54,"rawData":"0","type":"STRING"},{"byteSize":1,"index":55,"rawData":"1","type":"STRING"},{"byteSize":8,"index":56,"rawData":"40160.85","type":"DOUBLE"}],"type":"map"}

字段计算配置情况如下:

问题原因

Oracle数据库字段名区分大小写,使用md5()函数加密Oracle字段时需要注意大小写。原始字段CUST_NO为大写,md5()函数中也要用大写。

解决方案

将“字段计算”配置中md5()函数中的cust_no改为大写CUST_NO问题解决。

更多信息

“字段计算”组件中的函数语法和对应的输入数据源的语法兼容,Oracle字段名区分大小写,这里也要区分;mysql字段名不区分大小写,可以不用区分。

适用于

  • Dataphin-数据集成