Dataphin管道任务将专有云环境OSS数据同步到MaxCompute，运行报错“com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method”
问题描述
Dataphin管道任务将专有云环境OSS数据源数据同步到MaxCompute数据源，运行报错“com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method”。完整的异常如下。
2021-11-04 15:15:30.286 [0-0-0-reader] ERROR ReaderRunner - Reader runner Received Exceptions:
com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method.
[ErrorCode]: SignatureDoesNotMatch
[RequestId]: 61838892DB1A73BEC31497B1
[HostId]: oss-cn-XXX-XXX-XXX-XXX.com.cn
[ResponseError]:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<RequestId>61838892DB1A73BEC31497B1</RequestId>
<HostId>oss-cn-XXX-XXX-XXX-XXX.com.cn</HostId>
<OSSAccessKeyId>XXXXXX</OSSAccessKeyId>
<SignatureProvided>XXXXXXXXXXXXX</SignatureProvided>
<StringToSign>GET
问题原因
访问专有云环境OSS数据源时，需要配置CNAME参数项。Dataphin平台接入OSS数据源时未提供配置CNAME的选项导致。
解决方案
解决方案一：创建个Shell任务，通过脚本方式进行数据同步，脚本中可以自定义配置项。
cat >dlink.json <<EOF
{
"order":{
"hops":[
{
"from":"OSS_1",
"to":"MaxCompute_1"
}
]
},
"setting":{
"engine":{
"name":"dlink"
},
"errorLimit":{
"record":0
},
"jvmOption":"",
"speed":{
"concurrent":3
}
},
"stepErrorHandlers":[],
"steps":[
{
"category":"reader",
"distribute":true,
"name":"OSS_1",
"parameter":{
"accessId":"hlocUfPUzlk4thAN",
"accessKey":"******************************",
"bucket":"************",
"cname":"****************",
"column":[
{
"index":0,
"name":"id",
"type":"String"
},
{
"index":1,
"name":"login_name",
"type":"String"
}
],
"dsId":"6855333828453944256",
"dsName":"oss_test",
"encoding":"utf-8",
"endpoint":"http://************.***.******.com.cn/",
"fieldDelimiter":",",
"fileFormat":"csv",
"object":[
"****/********"
],
"skipHeader":"true"
},
"stepType":"oss"
},
{
"category":"writer",
"columnMapping":[
{
"dstColName":"id",
"sourceColName":"id"
},
{
"dstColName":"login_name",
"sourceColName":"login_name"
}
],
"name":"MaxCompute_1",
"parameter":{
"accessId":"***************",
"accessKey":"******************************",
"column":[
"id",
"login_name"
],
"dsId":"6821364645370481600",
"dsName":"GH_ODS",
"odpsServer":"http://*****************.*************.********.************.com.cn:80/api",
"partition":"ds='20211103'",
"project":"GH_ODS_DEV",
"table":"user_info_test",
"truncate":true
},
"stepType":"odps"
}
]
}
EOF
python $DLINK_HOME/bin/dlink.py dlink.json
解决方案二：用管道任务中的脚本模式组件，这样配置成本比较低。
完整的代码块如下：
{
"parameter": {
"object": [
"*********/************.csv"
],
"cname": "**********.com.cn",
"skipHeader": true,
"fieldDelimiter": ",",
"encoding": "utf-8",
"fileFormat": "csv"
},
"column": [
{
"index": 0,
"name": "id",
"type": "String"
},
{
"index": 1,
"name": "login_name",
"type": "String"
}
]
}
备注：Dataphin中配置专有云环境OSS数据源未提供CNAME配置项的问题，产品后续版本会进一步完善。
更多信息
OSS中bucket-cname（查看Bucket CNAME配置）
适用于
- Dataphin
