DocsICP FilingConsole
官方文档
首页

Run a script on an ECS instance from a node

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic describes how to use the third-party Python module Paramiko in a DataWorks node to remotely connect to an ECS instance and run a shell script. This topic provides code examples that show how to log on to an ECS instance from different types of nodes.

Notes

  • If you run a script on an ECS instance from a DataWorks node, the script continues to run on the ECS instance even if the DataWorks node stops.

  • This method of running scripts on an ECS instance from a DataWorks node is recommended only for data migration scenarios and not for daily production.

Overview

This topic provides examples of how to log on to an ECS instance using PyODPS and EMR Shell nodes:

Note

DataWorks provides Secure Shell (SSH) nodes. You can use these nodes to access an ECS instance and run a script at a specified path.

Prerequisites

  1. Create a Serverless resource group and configure its network connection.

    For more information, see Network connection solutions.

    Note

    For more information about how to retrieve the IP addresses of the Serverless resource group, see General configurations: Add IP addresses to a whitelist.

  2. Install third-party Python packages in the Serverless resource group.

    Install the required third-party package Paramiko using the custom image feature of the Serverless resource group. For more information, see Custom images.

Method 1: Use a PyODPS node to log on to an ECS instance with a username and password

You can log on to an ECS instance from a PyODPS node using a username and password. This example uses a PyODPS 3 node. The following code provides an example:


# For more information, see Call third-party packages in a PyODPS node: https://help.aliyun.com/document_detail/94159.html#section-f47-6lb-txv
# /home/tops/bin/pip3 install paramiko==2.11.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
# /home/tops/bin/pip3 install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple


from paramiko import SSHClient
import paramiko
import sys
client = SSHClient()

client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

client.connect('172.16.0.0', username='root', password='****')


stdin, stdout, stderr = client.exec_command('sh /root/upload_mc_tpcds_1000_warehouse.sh')
stdout.channel.set_combine_stderr(True)
# print(type(stdin))  # <class 'paramiko.channel.ChannelStdinFile'>
# print(type(stdout))  # <class 'paramiko.channel.ChannelFile'>
# print(type(stderr))  # <class 'paramiko.channel.ChannelStderrFile'>

# Print output of command. Will wait for command to finish.
print(f'STDOUT: {stdout.read().decode("utf8")}')
# print(f'STDERR: {stderr.read().decode("utf8")}')

# Get return code from command (0 is default for success)
print(f'Return code: {stdout.channel.recv_exit_status()}')
return_code = stdout.channel.recv_exit_status()

stdin.close()
stdout.close()
stderr.close()
client.close()

if return_code == 0:
    print(f'Over...')
    sys.exit(0)

print(f'Remote shell is failed...')

Method 2: Use a PyODPS node to log on to an ECS instance with a username and private key and use a utility class

For more information about how to use a private key with Paramiko, see python - How do use paramiko.RSAKey.from_private_key()? - Stack Overflow.

  1. Define a utility class.

    You can define a utility class as a DataWorks Python resource. For more information about how to create a Python resource, see Create and use MaxCompute resources. In this example, the utility class is named RemoteShell.py. The following code provides an example:

    # -*- coding: utf-8 -*-
import sys

from paramiko import SSHClient
import paramiko

class RemoteShell:

    def __init__(self, hostname, username, key_filename):
        self.hostname = hostname
        self.username = username
        self.key_filename = key_filename

    def run_remote_shell(self, cmd):
        client = SSHClient()

        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        client.connect(hostname=self.hostname, username=self.username, key_filename=self.key_filename)

        stdin, stdout, stderr = client.exec_command(cmd)
        stdout.channel.set_combine_stderr(True)
        # Print output of command. Will wait for command to finish.
        print(f'STDOUT: {stdout.read().decode("utf8")}')
        # print(f'STDERR: {stderr.read().decode("utf8")}')

        # Get return code from command (0 is default for success)
        print(f'Return code: {stdout.channel.recv_exit_status()}')
        return_code = stdout.channel.recv_exit_status()

        stdin.close()
        stdout.close()
        stderr.close()
        client.close()

        if return_code == 0:
            print(f'Over...')
            sys.exit(0)

        print(f'Remote shell is failed...')

  2. Create the key file run_remote_shell_user.pem.

    Upload the private key file as a File resource. For more information about how to upload a resource, see Create and use MaxCompute resources.

  3. Create a PyODPS node and use the utility class and key file.

    1. In this example, create a PyODPS 3 node named run_remote_shell_private_key_template. In the node, reference the RemoteShell.py utility class and the run_remote_shell_user.pem key file. After you reference the resources, a comment is added to the code of the run_remote_shell_private_key_template node.

    2. In the run_remote_shell_private_key_template node, define variables and assign scheduling parameters to them. For more information about how to use scheduling parameters in a PyODPS node, see Use scheduling parameters in PyODPS.

      ##@resource_reference{"run_remote_shell_user.pem"}
##@resource_reference{"RemoteShell.py"}
# For more information, see Call third-party packages in a PyODPS node: https://help.aliyun.com/document_detail/94159.html#section-f47-6lb-txv
# /home/tops/bin/pip3 install paramiko==2.11.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
# /home/tops/bin/pip3 install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple
# Input parameters for PyODPS: https://help.aliyun.com/document_detail/417492.htm#section-uv0-uvh-oau
# Dependencies on normal Python scripts: https://help.aliyun.com/document_detail/94159.html
import sys
import os
sys.path.append(os.path.dirname(os.path.abspath('RemoteShell.py'))) # Import the resource to the current environment.
from RemoteShell import RemoteShell # Reference the resource. You must remove the .py extension from the resource name.

remoteShell = RemoteShell(hostname='172.16.0.0', username='****', key_filename='run_remote_shell_user.pem')

dp = args['dp']
cmd = f'sh /root/dw-emr-shell.sh {dp}'
print(f'Remote shell is : {cmd}')
remoteShell.run_remote_shell(cmd)

print("Remote Shell is finished.")

Method 3: Use an EMR Shell node to log on to an ECS instance with a username and password and use a utility class

  1. Define a utility class.

    Upload a Python file as an EMR resource file. For more information about Python resources, see Create an EMR resource. In this example, the Python resource is named ecs.py.

    from paramiko import SSHClient
import paramiko
import sys, getopt

username = ''
password = ''
ip = ''
cmd = ''
try:
    opts, args = getopt.getopt(sys.argv[1:], "u:p:i:c:", ["user=", "password=", "ip=", "cmd="])
except getopt.GetoptError:
    print('error get inputs')
    sys.exit(2)
for opt, arg in opts:

    if opt in ("-u", "--user"):
        username = arg
        print('username: ' + username)
    elif opt in ("-p", "--password"):
        password = arg
        print('password: ' + password)
    elif opt in ("-i", "--ip"):
        ip = arg
        print('ip: ' + ip)
    elif opt in ("-c", "--cmd"):
        cmd = arg
        print('cmd: ' + cmd)

print(username, password, ip, cmd)

client = SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(ip, username=username, password=password)

stdin, stdout, stderr = client.exec_command(cmd)
stdout.channel.set_combine_stderr(True)
# Print output of command. Will wait for command to finish.
print(f'STDOUT: {stdout.read().decode("utf8")}')
# print(f'STDERR: {stderr.read().decode("utf8")}')

# Get return code from command (0 is default for success)
return_code = stdout.channel.recv_exit_status()
print(f'Return code: {return_code}')

stdin.close()
stdout.close()
stderr.close()
client.close()

if return_code == 0:
    print(f'Over...')
    sys.exit(0)

print(f'Remote shell is failed...')

  2. Create an EMR Shell node and use the utility class.

    In this example, create an EMR Shell node named run_remote_shell_EMR. In the run_remote_shell_EMR node, reference the ecs.py utility class. After you reference the resource, a comment is added.

    ##@resource_reference{"ecs.py"}
/home/tops/bin/python3 ecs.py  -u root -p 'password' -i '172.0.X.X' -c 'sh /abc.sh'
    Note

    The parameters are described as follows: -u specifies the username to log on to the ECS instance. -p specifies the password. -i specifies the private IP address of the ECS instance. -c specifies the command to run.

Method 4: Use an EMR Shell node to log on to an ECS instance with a username and private key and use a utility class

For more information about how to use a private key with Paramiko, see python - How do use paramiko.RSAKey.from_private_key()? - Stack Overflow.

  1. Upload the private key.

    Upload the private key for logging on to the ECS instance as an EMR File resource. For more information, see Create an EMR resource. In this example, the private key is ssh_pair_yunlin_beijing.pem.

  2. Define a utility class.

    You can define a utility class by uploading a Python file as an EMR resource file. For more information about how to upload a Python resource, see Create an EMR resource. In this example, the Python resource is named run_remote_shell.py. The following code provides an example.

    Note

    Before you upload the Python file, modify the parameters as needed.

    • username: The username.

    • ip: The IP address used to connect to the ECS instance.

    • private_key: The name of the EMR resource file for the private key that you uploaded in Step 1.

    from paramiko import SSHClient
import paramiko
import sys, getopt

username = 'emr-user'
# Replace the private_key name with the name of the uploaded private key resource.
private_key = 'ssh_pair_yunlin_beijing.pem'
password = ''
ip = '172.16.8.188'
cmd = sys.argv[1]

print(f'params - ip: {ip}, username: {username}, ip: {ip}, cmd: {cmd}')


client = SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(ip, username=username, key_filename=private_key)

stdin, stdout, stderr = client.exec_command(cmd)
stdout.channel.set_combine_stderr(True)
# Print output of command. Will wait for command to finish.
print(f'STDOUT: {stdout.read().decode("utf8")}')

# Get return code from command (0 is default for success)
return_code = stdout.channel.recv_exit_status()
print(f'Return code: {return_code}')

stdin.close()
stdout.close()
stderr.close()
client.close()

if return_code == 0:
    print(f'Over...')
    sys.exit(0)

print(f'Remote shell is failed...')

  3. Create an EMR Shell node and use the utility class.

    In this example, create an EMR Shell node named run_remote_shell. In the node, reference the run_remote_shell.py utility class and the ssh_pair_yunlin_beijing.pem private key. After you reference the resources, two comments are added. For more information about how to create an EMR Shell node and reference resources, see Create an EMR Shell node.

    ##@resource_reference{"run_remote_shell.py"}
##@resource_reference{"ssh_pair_yunlin_beijing.pem"}
/home/tops/bin/python3 run_remote_shell.py "ls /tmp/"
    Note

    • When you use the code, replace the resource_reference placeholder with the name of the resource that you uploaded.

    • The command that follows run_remote_shell.py is the command to run on the ECS instance.

Previous：Personalized video recommendation (collaborative filtering)Next：Develop tasks on a self-managed Hadoop cluster