Dataphin中使用Shell任务调用DataX脚本,将MaxCompute数据同步至OSS,补数据运行报错“Shell command exit with code: 247”
更新时间:
问题描述
Dataphin中使用Shell任务调用Datax脚本,将MaxCompute数据同步至OSS,补数据运行报错“Shell command exit with code: 247”。完整的报错日志如下展示:
2022-07-08 13:45:10.962 [0-0-0-writer] INFO OssWriter$Task - upload part [191] size [10485812] Byte has been completed.
2022-07-08 13:45:11.389 [0-0-0-writer] INFO OssWriter$Task - upload part [192] size [10485814] Byte has been completed.
2022-07-08 13:45:11.878 [0-0-0-writer] INFO OssWriter$Task - upload part [193] size [10485786] Byte has been completed.
2022-07-08 13:45:18 No outputData produced.
2022-07-08 13:45:18 Shell command exit with code: 247
2022-07-08 13:45:18 =================================================================
2022-07-08 13:45:18 Current task status: FAILED
2022-07-08 13:45:18 Elapsed time: 1.606 min
2022-07-08 13:45:18 ---------------- voldemort task ends ----------------
完整的代码如下:
cat /mnt/mesos/sandbox/resources/0000004850313/resource.aksk_resource)
cat > datax-export_nba_product_recommend.json << EOF
{
"job": {
"setting": {
"speed": {
"channel": 1
}
},
"content": [{
"reader": {
"name": "odpsreader",
"parameter": {
"accessId": "${accessId}",
"accessKey": "${accessKey}",
"project": "XXXXX",
"table": "ads_nba_product_di",
"partition": ["ds='20220707'"],
"column": [
"id_type",
"id",
"item",
"ds"
],
"splitMode": "record",
"odpsServer": "http://XXXXX/api"
}
},
"writer": {
"name": "osswriter",
"parameter": {
"endpoint": "XXXX",
"accessId": "${OSS_NBM_AccessID}",
"accessKey": "${OSS_NBM_AccessKey}",
"bucket": "XXXX",
"object": "XXXX/nba_product_20220707",
"encoding": "UTF-8",
"fileFormat": "csv",
"suffix": ".csv",
"fieldDelimiter": ",",
"writeMode": "append",
"writeSingleObject": "true",
"header": [
"id_type",
"id",
"item",
"ds"
]
}
}
}]
}
}
EOF
python $DATAX_HOME/bin/datax.py datax-export_nba_product_recommend.json
问题原因
默认的内存不够,导致任务运行一半挂掉导致。
解决方案
在创建数据同步的Shell脚本的JSON中添加如下参数问题解决:
@required_resource{required_memory=2GB;required_cpus=1}
适用于
- Dataphin
文档内容是否对您有帮助?