Dataphin管道任务将专有云环境OSS数据同步到MaxCompute,运行报错“com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method”

问题描述

Dataphin管道任务将专有云环境OSS数据源数据同步到MaxCompute数据源,运行报错“com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method”。完整的异常如下。

2021-11-04 15:15:30.286 [0-0-0-reader] ERROR ReaderRunner - Reader runner Received Exceptions:
com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method.
[ErrorCode]: SignatureDoesNotMatch
[RequestId]: 61838892DB1A73BEC31497B1
[HostId]: oss-cn-XXX-XXX-XXX-XXX.com.cn
[ResponseError]:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <RequestId>61838892DB1A73BEC31497B1</RequestId>
  <HostId>oss-cn-XXX-XXX-XXX-XXX.com.cn</HostId>
  <OSSAccessKeyId>XXXXXX</OSSAccessKeyId>
  <SignatureProvided>XXXXXXXXXXXXX</SignatureProvided>
  <StringToSign>GET

问题原因

访问专有云环境OSS数据源时,需要配置CNAME参数项。Dataphin平台接入OSS数据源时未提供配置CNAME的选项导致。

解决方案

解决方案一:创建个Shell任务,通过脚本方式进行数据同步,脚本中可以自定义配置项。

    cat >dlink.json <<EOF
{
    "order":{
      "hops":[
        {
          "from":"OSS_1",
          "to":"MaxCompute_1"
        }
      ]
    },
    "setting":{
      "engine":{
        "name":"dlink"
      },
      "errorLimit":{
        "record":0
      },
      "jvmOption":"",
      "speed":{
        "concurrent":3
      }
    },
    "stepErrorHandlers":[],
    "steps":[
      {
        "category":"reader",
        "distribute":true,
        "name":"OSS_1",
        "parameter":{
          "accessId":"hlocUfPUzlk4thAN",
          "accessKey":"******************************",
          "bucket":"************",
          "cname":"****************",
          "column":[
            {
              "index":0,
              "name":"id",
              "type":"String"
            },
            {
              "index":1,
              "name":"login_name",
              "type":"String"
            }
          ],
          "dsId":"6855333828453944256",
          "dsName":"oss_test",
          "encoding":"utf-8",
          "endpoint":"http://************.***.******.com.cn/",
          "fieldDelimiter":",",
          "fileFormat":"csv",
          "object":[
            "****/********"
          ],
          "skipHeader":"true"
        },
        "stepType":"oss"
      },
      {
        "category":"writer",
        "columnMapping":[
          {
            "dstColName":"id",
            "sourceColName":"id"
          },
          {
            "dstColName":"login_name",
            "sourceColName":"login_name"
          }
        ],
        "name":"MaxCompute_1",
        "parameter":{
          "accessId":"***************",
          "accessKey":"******************************",
          "column":[
            "id",
            "login_name"
          ],
          "dsId":"6821364645370481600",
          "dsName":"GH_ODS",
          "odpsServer":"http://*****************.*************.********.************.com.cn:80/api",
          "partition":"ds='20211103'",
          "project":"GH_ODS_DEV",
          "table":"user_info_test",
          "truncate":true
        },
        "stepType":"odps"
      }
    ]
  }
EOF
python $DLINK_HOME/bin/dlink.py dlink.json

解决方案二:用管道任务中的脚本模式组件,这样配置成本比较低。

完整的代码块如下:

{
    "parameter": {
        "object": [
            "*********/************.csv"
        ],
        "cname": "**********.com.cn",
        "skipHeader": true,
        "fieldDelimiter": ",",
        "encoding": "utf-8",
        "fileFormat": "csv"
    },
    "column": [
        {
            "index": 0,
            "name": "id",
            "type": "String"
        },
        {
            "index": 1,
            "name": "login_name",
            "type": "String"
        }
    ]
}

备注:Dataphin中配置专有云环境OSS数据源未提供CNAME配置项的问题,产品后续版本会进一步完善。

更多信息

OSS中bucket-cname(查看Bucket CNAME配置)

适用于

  • Dataphin
阿里云首页 相关技术圈