DocsICP FilingConsole
官方文档
首页

Agent management

更新时间：
复制 MD 格式
我的收藏

Learn how to create, deploy, scale, and upgrade agents.

Create an agent

Agents support two network types: public network and VPC. A VPC connection can be established by using a leased line or a VPN. Select a network type based on your requirements. Each agent must be associated with an existing tunnel, so you must create one first. For instructions on how to create a tunnel, see Tunnel Management.

Important

  • For use cases such as migrating data to the cloud over a leased line or VPN, migrating data from self-managed storage, or using a local file system as the source, you must create a tunnel and an agent.

  • The agent must be deployed on a 64-bit Linux (x86_64) machine running kernel 2.6 or later. Ensure you have a machine that meets these requirements.

  • You can create a maximum of 100 agents in each region. Plan your resources accordingly.

  • We recommend using 7th-generation or later ECS instance types, such as ecs.c7.xlarge.

  1. Log on to the Data Online Migration console.

  2. In the left-side navigation pane, choose Agent Management.

  3. On the Agent Management page, click New Agent.

  1. In the New Agent dialog box, configure the following parameters and click OK.

    Parameter

    Required

    Description

    Name

    Yes

    The agent's name.

    • The name must be 3 to 63 characters long.

      • The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.

      • The name must be UTF-8 encoded and cannot start with a hyphen (-) or an underscore (_).

    Network Type

    Yes

    The network connection type for the agent. The following options are available:

    • VPC (Recommended): The agent connects to the Data Online Migration service over a VPC. This method requires the machine that hosts the agent to access the internal endpoint of Data Online Migration in the corresponding region. For example, if you use Data Online Migration in the China (Beijing) region, the agent machine must be able to access the internal endpoint {TunnelId}.cn-beijing.mgw-tc-internal.aliyuncs.com. We recommend using an ECS instance in the same region as the Data Online Migration console to deploy the agent.

    • Public network: The agent connects to the Data Online Migration service over the public network. This method requires the machine that hosts the agent to access the public endpoint of Data Online Migration in the corresponding region. For example, if you use Data Online Migration in the China (Beijing) region, the agent machine must be able to access the public endpoint {TunnelId}.cn-beijing.mgw-tc.aliyuncs.com.

    Note

    • {TunnelId} is a placeholder for the tunnel ID.

    • You can use the ping command to test the network connectivity between the agent and the Data Online Migration service.

    Deployment method

    Yes

    The agent's deployment method. Currently, only standalone process deployment is supported.

    Tunnel

    Yes

    The tunnel to associate with the agent. An agent can be associated with only one tunnel. The bandwidth of the agent is limited by the total bandwidth of the tunnel.

    For example, a tunnel named tunnel-1 is configured with a maximum bandwidth of 10 Gbit/s. tunnel-1 is associated with three agents: agent-1, agent-2, and agent-3. The combined bandwidth of these three agents cannot exceed 10 Gbit/s. If agent-1 is allocated 3 Gbit/s of bandwidth, only 7 Gbit/s of bandwidth remains available for agent-2 and agent-3. Plan and allocate your bandwidth carefully.

Generate an agent deployment script

Prerequisites

  • The machine requires at least 4 CPU cores, 8 GB of memory, and 4 GB of available space for logs.

  • Create a new RAM user to deploy the agent. Grant the AliyunOSSImportReadOnlyAccess permission to the RAM user and set the Resource Scope to Account.

    Note

    You can use the access key pair of a single RAM user to deploy multiple agents without conflicts.

Procedure

  1. On the Agent Management page, click an agent's name to open its details page.

  2. Click Deploy and Run. In the Deploy and Run dialog box, configure the following parameters in the AccessKey Pairs section.

    Parameter

    Required

    Description

    AccessKey ID

    Yes

    Authenticates the agent when it establishes a tunnel with the Data Online Migration service. The mgw:VerifyAgentTunnel permission is required.

    Note

    Follow the principle of least privilege by granting only a custom policy with the mgw:VerifyAgentTunnel permission. You can also grant one of the following system policies as needed:

    • AliyunOSSImportReadOnlyAccess (read-only permissions to manage Data Online Migration)

    • AliyunOSSImportFullAccess (full management permissions)

    AccessKey Secret

    Yes

  3. Click the image.png icon next to Advanced Settings to configure the following parameters.

    Parameter

    Required

    Description

    CPU Cores

    No

    The number of CPU cores available to the agent.

    • If you do not specify this parameter, all CPU cores of the machine are used by default.

    • If you specify this parameter, carefully evaluate the value based on the prompt in the console.

    Maximum Memory Size (GB)

    No

    The maximum memory available to the agent.

    • If you do not specify this parameter, all memory of the machine is used by default.

    • If you specify this parameter, carefully evaluate the value based on the prompt in the console.

    Space Used by Logs (GB)

    No

    The maximum disk space for agent logs. When this limit is reached, the oldest logs are discarded.

    • If you do not specify this parameter, 4 GB of space is used by default.

    • If you specify this parameter, evaluate the value based on the number of files to migrate and refer to the prompt in the console. Migrating one million files generates about 4 GB of disk logs.

    Maximum Bandwidth

    No

    The maximum bandwidth for the agent. This value cannot exceed the bandwidth limit of the associated tunnel. For example:

    • If the maximum bandwidth for the tunnel is 5 Gbit/s, the maximum available bandwidth for a single agent cannot exceed 5 Gbit/s.

    • If no maximum bandwidth is specified for the tunnel, the maximum available bandwidth for the agent cannot exceed 100 Gbit/s.

    Note

    Each agent can be associated with only one tunnel, and its bandwidth is limited by that tunnel's total bandwidth.

    For example, a tunnel named tunnel-1 is configured with a maximum bandwidth of 10 Gbit/s. tunnel-1 is associated with three agents: agent-1, agent-2, and agent-3. The combined bandwidth of these three agents cannot exceed 10 Gbit/s. If agent-1 is allocated 3 Gbit/s of bandwidth, only 7 Gbit/s of bandwidth remains available for agent-2 and agent-3. Plan and allocate your bandwidth carefully.

  4. Click Deploy with Command.

Deploy an agent

Important

  • Each deployment command is automatically generated with unique information, such as the agent ID, and can deploy only one agent. To deploy multiple agents, generate a separate command for each one and run it on a different machine. Otherwise, the deployment may fail or the agent's status might become abnormal.

  • When migrating LocalFS data (from LocalFS to OSS, OSS to LocalFS, or between LocalFS systems), you must run the agent as the root user. Otherwise, file migration may fail due to insufficient permissions for system calls such as chown and chmod.

Online deployment

If you set Network Type to Internet for the agent, deploy the agent online.

  1. In the Use commands to deploy and run the agent dialog box, review the command and log on to the deployment machine. You can use Alibaba Cloud tools such as Workbench or VNC, or third-party client tools.

  2. On the deployment machine, verify that curl is installed. Then, run the curl -I https://www.aliyun.com command to check for Internet connectivity.

  3. Copy the agent deployment command from the dialog box. Run the command in a local disk directory on the deployment machine, not in a remotely mounted directory such as a NAS directory. A successful execution indicates that the agent is deployed. The command provided on the console is subject to change, so refer to the actual output for the most current version.

    wget https://gosspublic.alicdn.com/data_online_migration/agent/aliyun_import_agent_deploy.sh -O aliyun_import_agent_deploy.sh;chmod 755 aliyun_import_agent_deploy.sh;./aliyun_import_agent_deploy.sh -t <TunnelID> -u <UID> -i <AK> -k <SK> -n <AgentName> -v <AgentID> -e <Endpoint> -f <AgentDeployMethod> -c <CpuUsage> -m <MemoryUsage> -d <LogMemoryUsage> -b <MaxBandwidth>

    Parameter

    Description

    TunnelID

    The tunnel ID.

    UID

    The Alibaba Cloud account ID.

    AK

    The AccessKey ID.

    SK

    The AccessKey Secret.

    AgentName

    The agent name.

    AgentID

    The agent ID.

    Endpoint

    The endpoint. public indicates an Internet connection.

    AgentDeployMethod

    The deployment method of the agent. default indicates that the agent runs as a standalone process.

    CpuUsage

    The number of CPU cores. This parameter is included in the command only if a value is set.

    MemoryUsage

    The amount of memory to use, in GB. This parameter is included in the command only if a value is set.

    LogMemoryUsage

    The amount of disk space for logs, in GB. This parameter is included in the command only if a value is set.

    MaxBandwidth

    The maximum bandwidth to use, in MB/s. This parameter is included in the command only if a value is set.

  4. After you successfully deploy the agent, create a data address and associate it with the agent. Migration tasks that use this data address will run on the agent's machine.

Offline deployment

If you set Network Type to Leased Line/VPN (VPC) for the agent, deploy the agent offline.

  1. In the Use commands to deploy and run the agent dialog box, review the command and log on to the deployment machine. You can use Alibaba Cloud tools such as Workbench or VNC, or third-party client tools.

  2. Manually download the agent installation package aliyun_import_agent_offline_bundle.tar.gz and upload it to the deployment machine.

  3. On the deployment machine, navigate to the directory where the installation package is saved and run the tar -zxvf aliyun_import_agent_offline_bundle.tar.gz command to extract the package.

  4. After the package is extracted, navigate to the aliyun_import_agent_offline_bundle directory. Copy the agent deployment command from the dialog box. Run the command in a local disk directory on the deployment machine, not in a remotely mounted directory such as a NAS directory. A successful execution indicates that the agent is deployed. The command provided on the console is subject to change, so refer to the actual output for the most current version.

    chmod 755 aliyun_import_agent_deploy.sh;./aliyun_import_agent_deploy.sh -t <TunnelID> -u <UID> -i <AK> -k <SK> -n <AgentName> -v <AgentID> -e <Endpoint> -f <AgentDeployMethod> -c <CpuUsage> -m <MemoryUsage> -d <LogMemoryUsage> -b <MaxBandwidth> -l <VPCType>

    Parameter

    Description

    TunnelID

    The tunnel ID.

    UID

    The Alibaba Cloud account ID.

    AK

    The AccessKey ID.

    SK

    The AccessKey Secret.

    AgentName

    The agent name.

    AgentID

    The agent ID.

    Endpoint

    The endpoint. vpc indicates a connection over a leased line or VPN.

    AgentDeployMethod

    The deployment method of the agent. default indicates that the agent runs as a standalone process.

    CpuUsage

    The number of CPU cores. This parameter is included in the command only if a value is set.

    MemoryUsage

    The amount of memory to use, in GB. This parameter is included in the command only if a value is set.

    LogMemoryUsage

    The amount of disk space for logs, in GB. This parameter is included in the command only if a value is set.

    MaxBandwidth

    The maximum bandwidth to use, in MB/s. This parameter is included in the command only if a value is set.

    VPCType

    Settings related to the VPC network type.

  5. After you successfully deploy the agent, create a data address and associate it with the agent. Migration tasks that use this data address will run on the agent's machine.

Agent scale-out

To scale out an agent, create a new agent and generate a deployment script. After deploying the agent, go to the Address Management page to modify the corresponding address and associate it with the new agent. This process does not affect ongoing migration tasks.

  1. Create a new agent. For more information, see Create an agent.

  2. Generate a deployment script for the agent. For more information, see Generate a deployment script for an agent.

  3. Deploy the agent. For more information, see Deploy an agent.

  4. Add the new agent to the data address:

    1. In the navigation pane, choose Address Management.

    2. On the Address Management page, click Modify in the Actions column to open the Modify Address dialog box.

    3. In the dialog box, click the Agent drop-down list and select the checkbox for the new agent.

    4. Click OK.

Agent scale-in

To scale in an agent, first remove the agent from its corresponding address on the Address Management page. Then, delete it from the Agent Management page. If the agent instance is used only for migration and its removal does not affect other services, you can release it. This process does not affect ongoing migration tasks.

  1. Remove the agent from the address.

    1. Log on to the Data Online Migration console. In the navigation pane, choose Address Management.

    2. On the Address Management page, find the address that you want to manage and click Modify in the Actions column. The Modify Address dialog box appears.

    3. In the pop-up box, click the Agent dropdown list, and remove the agent that you want to exclude.

    4. Click OK.

  2. Delete the agent.

    1. Delete the agent from the Data Online Migration console.

      1. In the navigation pane, choose Agent Management.

      2. On the Agent Management page, find the agent that you want to delete and click Delete in the Delete column. The Confirm Delete dialog box appears.

      3. Click OK.

  3. Release the agent instance (Optional)

    You can release the agent instance if it is used only for migration and its removal will not affect other services.

Upgrade an agent

  1. Pause the migration task:

    1. Log on to the Data Online Migration console. In the navigation pane, choose Migration Tasks.

    2. On the Migration Tasks page, click Manage in the Actions column to open the migration task details page.

    3. In the History section at the bottom of the migration task details page, click Suspended. The migration task details page displays the migration progress and an execution history table. In the Actions column of this table, click Pause.

  2. Stop the agent program on the agent machine:

    1. Log on to the agent machine and go to the root directory of the agent program, for example, /root/aliyun_import_agent_online_res.

    2. Run the bash bin/stop.sh command and wait for the agent program to stop.

  3. Check the old agent version number:

    1. Log on to the agent machine, run the ./bin/drs_import_agent -v command, and record the Version number.

      [r xxx : aliyun_import_agent_online_res]# ./bin/drs_import_agent -v
Version: 1.4.2.992
BuildTime: 2024-11-11_16:04:36_CST
Bu xxx p_ts-1.5.6

  4. Deploy the agent:

    1. Generate a deployment script for the agent. For more information, see Generate a deployment script for an agent.

    2. Deploy the agent. For more information, see Deploy an agent.

  5. Verify that the agent is upgraded:

    1. Log on to the agent machine and go to the root directory of the agent program, for example, /root/aliyun_import_agent_online_res.

    2. Run the ./bin/drs_import_agent -v command and compare the new version number with the old one. If the version number has been updated, the upgrade was successful.

Manage the agent process

Note

  • After a successful agent deployment, the agent process starts automatically. No further action is usually required.

  • The agent process is not configured to run on startup. Therefore, you must manually start the agent process if the machine restarts or if the process stops for any other reason.

  • If an agent is no longer in use, we recommend that you promptly stop the agent process and delete the corresponding agent instance in the console.

  1. To view the agent process, follow these steps:

    1. Log on to the agent machine and navigate to the agent's root directory, for example, /root/aliyun_import_agent_online_res.

    2. Run the ps -ef | grep drs_import_agent command to view process information, such as the process ID and the running directory. The following output indicates that the drs_import_agent process is running successfully.

      [root@xxx aliyun_import_agent_online_res]# ps -ef | grep drs_import_agent
root     19182     1  1 14:35 pts/0    00:00:05 /xxx/aliyun_import_agent_online_res/bin/drs_import_agent -c /xxx/aliyun_import_agent_online_res/config/agent-config.properties
root     20305 18122  0 14:40 pts/0    00:00:00 grep --color=auto drs_import_agent

    3. The agent generates log files during runtime. By default, the agent saves log files in the logs/ directory within the agent's root directory. The log files are automatically rotated based on the log space parameter specified during agent deployment.

  2. To start the agent process, follow these steps:

    1. Log on to the agent machine and navigate to the agent's root directory, for example, /root/aliyun_import_agent_online_res.

    2. Check if the agent process is running. If the process is not running, run the bash bin/start.sh command to start it.

  3. To stop the agent process, follow these steps:

    1. Log on to the agent machine and navigate to the agent's root directory, for example, /root/aliyun_import_agent_online_res.

    2. Run the bash bin/stop.sh command and wait for the agent to stop.

    3. After the agent stops, log on to the Data Online Migration console and check the status of this agent instance. A status of Connection Abnormal indicates that the agent has completely stopped.

Key agent parameters

When you deploy an agent, you configure its runtime parameters by using the deployment command generated in the console. These parameters are saved to the agent-config.properties file in the config/ directory under the agent's root directory, for example, /root/aliyun_import_agent_online_res/config/agent-config.properties.

Important

  • The default configuration parameters suit most migration scenarios and typically require no modification.

  • The following table lists only some key agent parameters. Do not modify parameters that are not listed in this table.

  • Before you modify the parameters in this table, make sure that you understand their functions, scopes, and potential risks.

Parameter

Value

Description

ssrfBlacklistOn

Default: true

Valid values: true, false
Important

Disabling this feature reduces control over the agent process and introduces security risks.

Specifies whether to enable Server-Side Request Forgery (SSRF) blacklist detection. If this feature is enabled, the agent checks the IP address of the remote host before establishing any network connection. The agent rejects the connection if the IP address is in the private address range.

Example:

ssrfBlacklistOn=false

The private address range includes:

  • 0.0.0.0/8

  • 127.0.0.0/8

  • 10.0.0.0/8

  • 11.0.0.0/8

  • 25.0.0.0/8

  • 26.0.0.0/8

  • 30.0.0.0/8

  • 33.0.0.0/8

  • 192.168.0.0/16

  • 100.64.0.0/10

  • 172.16.0.0/12

  • 169.254.0.0/16

  • ::1/128

  • fe80::/10

  • fc00::/7

allowRedirectProviders

Default: empty

Valid values: Any data address type. Separate multiple types with a ,.
Note

By default, Data Online Migration does not support redirection for any data address type. If a request to a source or destination address returns a redirection status code, the migration reports the UnsupportedRedirect error code.

Enables HTTP redirection when accessing source and destination addresses. This setting applies only to the specified data address types.

Example:

allowRedirectProviders=qiniu,s3

underscoreUserMetaDropOrSkipUids

Default: empty

Valid values: A multi-field value in the format <UID>,drop;, where <UID> is the UID of the Alibaba Cloud account used for the migration.

Determines whether the agent drops metadata items from a source file if their metadata keys contain invalid characters. For details, see the SourceMetaInvalid error code.

Example:

underscoreUserMetaDropOrSkipUids=1234567890123456,drop;

After you modify and save the configuration items, restart the agent process to apply the changes.

Previous：Manage tunnelsNext：Select an agent