Dataphin中使用Shell任务调用DataX脚本，将MaxCompute数据同步至OSS，补数据运行报错“Shell command exit with code: 247”

问题描述

Dataphin中使用Shell任务调用Datax脚本，将MaxCompute数据同步至OSS，补数据运行报错“Shell command exit with code: 247”。完整的报错日志如下展示：

2022-07-08 13:45:10.962 [0-0-0-writer] INFO  OssWriter$Task - upload part [191] size [10485812] Byte has been completed.
2022-07-08 13:45:11.389 [0-0-0-writer] INFO  OssWriter$Task - upload part [192] size [10485814] Byte has been completed.
2022-07-08 13:45:11.878 [0-0-0-writer] INFO  OssWriter$Task - upload part [193] size [10485786] Byte has been completed.

2022-07-08 13:45:18 No outputData produced.
2022-07-08 13:45:18 Shell command exit with code: 247

2022-07-08 13:45:18 =================================================================
2022-07-08 13:45:18 Current task status: FAILED
2022-07-08 13:45:18 Elapsed time: 1.606 min
2022-07-08 13:45:18 ---------------- voldemort task ends ----------------

完整的代码如下：

cat /mnt/mesos/sandbox/resources/0000004850313/resource.aksk_resource)
cat > datax-export_nba_product_recommend.json << EOF
{
 "job": {
  "setting": {
   "speed": {
    "channel": 1
   }
  },
  "content": [{
   "reader": {
    "name": "odpsreader",
    "parameter": {
     "accessId": "${accessId}",
     "accessKey": "${accessKey}",
     "project": "XXXXX",
     "table": "ads_nba_product_di",
                     "partition": ["ds='20220707'"],
     "column": [
                        "id_type",
                        "id",
                        "item",
                        "ds"
     ],
     "splitMode": "record",
     "odpsServer": "http://XXXXX/api"
    }
   },
   "writer": {
                "name": "osswriter",
    "parameter": {
     "endpoint": "XXXX",
     "accessId": "${OSS_NBM_AccessID}",
     "accessKey": "${OSS_NBM_AccessKey}",
     "bucket": "XXXX",
     "object": "XXXX/nba_product_20220707",
     "encoding": "UTF-8",
                    "fileFormat": "csv",
                    "suffix": ".csv",
                    "fieldDelimiter": ",",
     "writeMode": "append",
                    "writeSingleObject": "true",
     "header": [
      "id_type",
                        "id",
                        "item",
                        "ds"
      ]
    }
   }
  }]
 }
}
EOF
python $DATAX_HOME/bin/datax.py datax-export_nba_product_recommend.json

问题原因

默认的内存不够，导致任务运行一半挂掉导致。

解决方案

在创建数据同步的Shell脚本的JSON中添加如下参数问题解决：

@required_resource{required_memory=2GB;required_cpus=1}

适用于

• Dataphin