首页 运维编排服务 实践教程 实例运维 从实例中拷贝文件到OSS

从实例中拷贝文件到OSS

更新时间: 2023-06-30 11:49:44

我们经常遇到从某台线上的机器把日志文件拷贝出来或者是进程卡了,通过jstack输出到某个文件再进行分析。我们使用osscmd把数据导到OSS里,但是手动配置过程实在是过于繁琐。通过如下模板您可很容易地处理此需求。

解决方案

  1. 从实例中拷贝文件可以利用云助手的RunCommand去到机器上去调用ossutil做数据拷贝,关键问题是AK怎么传,当然我们可以把AK当参数以命令的方式传入,但是这种太不安全了,相当于AK明文传输。那么有没有别的办法解决AK明文传输的问题呢?此时就想到了InstanceRole的功能,可以通过给instance attach一个ram role,那么在实例内部curl某个固定地址就能拿到以这个role身份产生的sts-token。详细参见概述

    curl http://100.100.100.200/latest/meta-data/Ram/security-credentials/{{InstanceAssumeRole}}	
  2. 以上AK的问题解决了,另外的问题ossutil在实例中访问不了公网的情况怎么下载呢?咨询了OSS的支持同事,他们并不提供内网下载的地址。只能自己搭建了,我们自己创建了个bucket为oos-public的bucket提供全局只读的权限可以方便内网下载。解析JSON我们需要个方便的工具jq,我们同时把这个文件传到公共bucket上经过调试最终的脚本如下。

    test -e ossutil64 || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/ossutil64 && chmod 755 ossutil64
    test -e jq || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/jq && chmod 755 jq
    stsToken=`curl http://100.100.100.200/latest/meta-data/Ram/security-credentials/{{InstanceAssumeRole}}`
    accessKeyId=`echo $stsToken |./jq .AccessKeyId | awk -F'\"' '{print $2}'` 
    accessKeySecret=`echo $stsToken | ./jq .AccessKeySecret | awk -F'\"' '{print $2}' 
    securityToken=`echo $stsToken | ./jq .SecurityToken | awk -F'\"' '{print $2}' 
    endpoint=https://oss-{{ACS::RegionId}}.aliyuncs.com;
    ./ossutil64 -i $accessKeyId -k $accessKeySecret -t $securityToken -e $endpoint cp {{SrcUrl}} {{DestUrl}}		
  3. 再转换成OOS模板。

    FormatVersion: OOS-2019-06-01
    Description: Copy ECS instance file to OSS
    Parameters:
      InstanceId:
        Type: String
        Description: The Ecs instance
      SrcUrl:
        Type: String
        Description: Src url
      DestUrl:
        Type: String
        Description: Dest url
      InstanceAssumeRole:
        Type: String
        Description: ''
      OOSAssumeRole:
        Type: String
        Description: oos assume this role to execution task.
        Default: OOSServiceRole
    RamRole: '{{OOSAssumeRole}}'
    Tasks:
      - Name: checkInstanceReady
        Action: ACS::CheckFor
        Description: describe instances with specified parameters.
        Properties:
          API: DescribeInstances
          Service: ECS
          PropertySelector: Instances.Instance[].Status
          DesiredValues:
            - Running
          Parameters:
            InstanceIds:
              - '{{ InstanceId }}'
      - Name: runCommand
        Action: ACS::ECS::RunCommand
        Description: ''
        Properties:
          commandContent:
            Fn::Join:
              - |+
    
              - - test -e oos || mkdir oos;
                - cd oos;
                - test -e ossutil64 || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/ossutil64 && chmod 755 ossutil64
                - test -e jq || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/jq && chmod 755 jq
                - stsToken=`curl http://100.100.100.200/latest/meta-data/Ram/security-credentials/{{InstanceAssumeRole}}`
                - accessKeyId=`echo $stsToken |./jq .AccessKeyId | awk -F'"' '{print $2}'` ;
                - accessKeySecret=`echo $stsToken | ./jq .AccessKeySecret | awk -F'"' '{print $2}'` ;
                - securityToken=`echo $stsToken | ./jq .SecurityToken | awk -F'"' '{print $2}'` ;
                - endpoint=https://oss-{{ACS::RegionId}}-internal.aliyuncs.com;
                - ./ossutil64 -i $accessKeyId -k $accessKeySecret -t $securityToken -e $endpoint cp {{SrcUrl}} {{DestUrl}}
          commandType: RunShellScript
          instanceId: '{{InstanceId}}'
        Outputs:
          CommandOutput:
            Type: String
            ValueSelector: InvocationResult[].Output
    
    {
      "FormatVersion": "OOS-2019-06-01",
      "Description": "Copy ECS instance file to OSS",
      "Parameters": {
        "InstanceId": {
          "Type": "String",
          "Description": "The Ecs instance"
        },
        "SrcUrl": {
          "Type": "String",
          "Description": "Src url"
        },
        "DestUrl": {
          "Type": "String",
          "Description": "Dest url"
        },
        "InstanceAssumeRole": {
          "Type": "String",
          "Description": ""
        },
        "OOSAssumeRole": {
          "Type": "String",
          "Description": "oos assume this role to execution task.",
          "Default": "OOSServiceRole"
        }
      },
      "RamRole": "{{OOSAssumeRole}}",
      "Tasks": [
        {
          "Name": "checkInstanceReady",
          "Action": "ACS::CheckFor",
          "Description": "describe instances with specified parameters.",
          "Properties": {
            "API": "DescribeInstances",
            "Service": "ECS",
            "PropertySelector": "Instances.Instance[].Status",
            "DesiredValues": [
              "Running"
            ],
            "Parameters": {
              "InstanceIds": [
                "{{ InstanceId }}"
              ]
            }
          }
        },
        {
          "Name": "runCommand",
          "Action": "ACS::ECS::RunCommand",
          "Description": "",
          "Properties": {
            "commandContent": {
              "Fn::Join": [
                "\n",
                [
                  "test -e oos || mkdir oos;",
                  "cd oos;",
                  "test -e ossutil64 || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/ossutil64 && chmod 755 ossutil64",
                  "test -e jq || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/jq && chmod 755 jq",
                  "stsToken=`curl http://100.100.100.200/latest/meta-data/Ram/security-credentials/{{InstanceAssumeRole}}`",
                  "accessKeyId=`echo $stsToken |./jq .AccessKeyId | awk -F'\"' '{print $2}'` ;",
                  "accessKeySecret=`echo $stsToken | ./jq .AccessKeySecret | awk -F'\"' '{print $2}'` ;",
                  "securityToken=`echo $stsToken | ./jq .SecurityToken | awk -F'\"' '{print $2}'` ;",
                  "endpoint=https://oss-{{ACS::RegionId}}-internal.aliyuncs.com;",
                  "./ossutil64 -i $accessKeyId -k $accessKeySecret -t $securityToken -e $endpoint cp {{SrcUrl}} {{DestUrl}}"
                ]
              ]
            },
            "commandType": "RunShellScript",
            "instanceId": "{{InstanceId}}"
          },
          "Outputs": {
            "CommandOutput": {
              "Type": "String",
              "ValueSelector": "InvocationResult[].Output"
            }
          }
        }
      ]
    }
  4. 传入参数实例ID,拷贝的文件,拷贝路径,传好之后执行。

  5. 执行成功后的结果。

  6. 根据Execution的执行日志可以看到执行过程base64 decode出来后可以看到Shell执行的具体信息,从结果上看是成功了,再到OSS上看,文件在不在。

总结

以上我们介绍了如果使用运维编排方便的从实例中拷贝文件到OSS上,结合拷贝文件的场景与执行命令的组合,我们可以方便的做出到某台机器jstack输出文件并拷贝出来等相应场景。

阿里云首页 运维编排服务 相关技术圈