DocsICP FilingConsole
官方文档
首页

Create a training job

更新时间：
复制 MD 格式
产品详情
我的收藏

PAI-DLC lets you create single-node or distributed training jobs. The service automatically provisions compute nodes through Kubernetes, so you don't need to purchase machines or configure runtime environments. It supports multiple deep learning frameworks, including TensorFlow and PyTorch, and offers flexible resource configuration.

Quick start

This topic demonstrates single-GPU and multi-node multi-GPU distributed training in DLC using the MNIST handwritten digit recognition example. For more information, see Get started with Deep Learning Containers (DLC).

Console parameters

Basic information

Configure the Job Name and Tag.

Environment

Parameter

Description

Image Configuration

In addition to Alibaba Cloud Image, the following image types are supported:

  • Custom Image: Use a custom image added to PAI. The image repository must be set to public, or the image must be stored in Container Registry (ACR). For more information, see Custom images.

    Note

    When you select Lingjun intelligent computing resources for the resource quota and use a custom image, you must manually install RDMA to take advantage of the high-performance RDMA network. For more information, see RDMA: High-performance networking for distributed training.

  • Image Address: Specify a custom or official image URL that is accessible over the public network.

    • For a private image URL, click Enter Account Password and configure the image repository username and password.

    • To improve image pull speed, see Image acceleration.

Mount dataset

Datasets provide data files for model training. Two dataset types are supported:

  • Custom Dataset: Create a custom dataset to store your training data files. You can set Read-only and select a dataset version from the version list.

  • Public Dataset: PAI provides public datasets that support read-only mount mode only.

Mount Path: the path where the dataset is mounted in the DLC container, for example, /mnt/data. You can access the dataset at this path in your code. For more mount configuration options, see Use cloud storage.

Important

If a CPFS dataset is configured, DLC must be configured with a VPC that matches the CPFS VPC. Otherwise, the job may get stuck in the environment preparation state for an extended period.

Mount storage

You can also mount a data source path directly to read data or store intermediate and output files.

  • Supported data source types: OSS, General-purpose NAS, Extreme NAS, CPFS, and BMCPFS (Lingjun intelligent computing resources only).

  • Advanced Settings: Different data source types support specific features through advanced settings:

    • OSS: Set {"mountType":"ossfs"} in advanced settings to mount OSS storage using ossfs.

    • General-purpose NAS and CPFS: Set the nconnect parameter in advanced settings to improve throughput when the DLC container accesses NAS. For more information, see How do I improve NAS performance on Linux?. Example: {"nconnect":"<example value>"}. Replace <example value> with a positive integer.

For more information, see Use cloud storage.

Startup Command

Specify the startup command for the job. Shell commands are supported. DLC automatically injects common environment variables for PyTorch and TensorFlow, such as MASTER_ADDR and WORLD_SIZE. You can reference these variables using $EnvironmentVariableName. Examples:

  • Run Python: python -c "print('Hello World')"

  • Multi-node multi-GPU distributed training with torch: python -m torch.distributed.launch \ --nproc_per_node=2 \ --master_addr=${MASTER_ADDR} \ --master_port=${MASTER_PORT} \ --nnodes=${WORLD_SIZE} \ --node_rank=${RANK} \ train.py --epochs=100

  • Set a shell file path as the startup command: /ml/input/config/launch.sh

Important

The mount path configured for code (such as /mnt/data) is the location of code files in the DLC container. Your startup command must use this path to reference code files.

For example, if the code mount path is /mnt/data and the code file is train.py, the startup command should be: python /mnt/data/train.py

More settings: environment variables, third-party libraries, and code

Environment Variable

In addition to the automatically injected common environment variables for PyTorch and TensorFlow, you can add up to 20 custom environment variables in Key:Value format.

Third-party Libraries

If your container image is missing third-party libraries, you can add them through Third-party Libraries in two ways:

  • Select from List: Enter library names in the text box, separated by spaces. Example: griffon numpy pandas.

  • Directory of requirements.txt: Write the libraries in a requirements.txt file, upload it to the DLC container through code, dataset, or direct mount, and then specify the file path in the container.

Code Builds

Upload code files to the DLC container in one of the following ways:

  • Online configuration: If you have a Git repository with access permissions, you can Create a code configuration to provide the job code to DLC.

  • Local Upload: Click the image.png button to upload local code files. After the upload, set the Mount path to a path inside the container, such as /mnt/data.

Resources

Parameter

Description

Resource Type

Default: General Computing. Lingjun Intelligence Resources Available regions: China (Ulanqab), Singapore, China (Shenzhen), China (Beijing), China (Shanghai), China (Hangzhou), China (Guangzhou), China (Hong Kong), Malaysia (Kuala Lumpur), Germany (Frankfurt), US (Atlanta).

Source

  • Public Resources:

    • Billing method: pay-as-you-go.

    • Use case: suitable for workloads with low volume or flexible timelines. Public resources may require queuing.

    • Limits: up to 2 GPUs and 8 CPU cores. To increase these limits, contact your account manager.

  • Resource Quota: includes general-purpose compute or Lingjun intelligent computing resources.

    • Billing method: subscription.

    • Use case: suitable for high-volume workloads that require guaranteed resources.

    • Special parameters:

      • Resource Quota: Set the number of GPUs, CPUs, and other resources. For more information about how to prepare resource quotas, see Create a resource quota.

      • Priority: the execution priority of concurrent jobs. Valid values: 1 to 9. A value of 1 indicates the lowest priority.

    • Pre-check: automatically verifies GPU model and image compatibility before job submission to detect configuration errors early.

  • Preemptible Resources:

    • Billing method: pay-as-you-go.

    • Use case: spot resources are priced lower than pay-as-you-go resources and are suitable for cost-sensitive workloads.

    • Limits: availability is not guaranteed. Resources may be unavailable or reclaimed at any time. For more information, see Use a preemptible job.

Framework

The following deep learning frameworks and training tools are supported: TensorFlow, PyTorch, ElasticBatch, XGBoost, OneFlow, MPIJob、Slurm, Ray, Custom, DataJuicer, and MPI.

Note

When Resource Quota is set to Lingjun intelligent computing resources, only TensorFlow, PyTorch, ElasticBatch, MPIJob、Slurm, and Ray jobs are supported.

Job Resource

Based on the selected Framework, configure resources for Worker, PS, Chief, Evaluator, and GraphLearn node types. When using the Ray framework, you can click Add Role to define custom Worker roles for heterogeneous resource mixing.

  • Use public resources: configure the following parameters:

    • Number of Nodes: the number of nodes that run the DLC job.

    • Resource Type: select a resource specification. The console displays the corresponding price. For more information, see DLC billing.

  • Use resource quotas: configure the number of nodes, CPUs (cores), GPUs, memory (GiB), and shared memory (GiB) for each node type. The following special parameters are also available:

    • Node-Specific Scheduling: run the job on specific compute nodes.

    • Idle Resources: use idle resources from other quotas to improve resource utilization. Preemptible jobs are terminated and resources are returned when the original quota needs them. For more information, see Use idle-time resources.

    • CPU Affinity: bind processes to specific CPU cores to reduce cache misses and context switches. Suitable for performance-sensitive and latency-critical workloads.

  • Use spot resources: in addition to the number of nodes and resource specifications, you can also configure Bid Price to set a maximum bid for spot resources. Click the image button to choose a bidding method:

    • By discount: the maximum price is based on the market price of the resource specification, with discrete options from 10% to 90% of the market price. The bid represents the upper limit for spot resource bidding. Spot resources are allocated when your maximum bid is at or above the market price and sufficient inventory is available.

    • By price: the maximum price falls within the market price range.

Scheduling Policy

You can customize the resource scheduling strategy for job containers based on your workload characteristics:

  • Intelligent: GPU jobs are packed onto the fewest GPU nodes possible, while CPU jobs are spread across CPU nodes. This is the default strategy.

  • Centralized: jobs are packed onto the fewest nodes possible. This strategy reduces resource fragmentation and is ideal for GPU model training jobs.

  • Spread: jobs are spread across as many nodes as possible. This strategy improves the job scheduling success rate and is ideal for distributed CPU jobs.

Note

DLC does not support modifying the instance specification of a submitted job. To switch to a different specification (for example, an instance that supports eRDMA high-performance networking such as ecs.ebmgn8v.48xlarge), use the Clone Job feature to create a new job with the target specification. The platform automatically enables high-performance networking and configures NCCL parameters.

More settings: maximum duration, retention period, and advanced framework configuration

Maximum Duration

Set the maximum duration for a job. Jobs that exceed this duration are automatically stopped. Default: 30 days.

Retention Period

Set the retention period for completed or failed jobs. Enabling job retention keeps resources occupied. Jobs that exceed this period are deleted.

Important

Deleted DLC jobs cannot be recovered. Proceed with caution.

Start Developer Machine

When using a resource quota, you can launch a development workstation (DSW) for online debugging. On the job overview page, click Development Workstation (DSW) in the Actions column of the instance list to access it.

Advanced Framework Configuration

For the full list of configurable parameters and their descriptions, see Advanced parameters.

  • The parameters ReleaseResourcePolicy, EnableNvidiaIBGDA, EnableNvidiaGDRCopy, EnablePaiNUMACoreBinding, and EnableResourcePreCheck are supported for all frameworks.

  • When Framework is set to PyTorch, the following parameters are available: createSvcForAllWorkers, customPortList, and customPortNumPerWorker.

    Important

    Lingjun intelligent computing resources do not support custom ports. The customPortNumPerWorker parameter is not available when submitting DLC jobs with Lingjun resources.

  • When Framework is set to Ray, the following parameters are available: RayRuntimeEnv, RayRedisAddress, RayRedisUsername, RayRedisPassword, RaySubmitterBackoffLimit, and RayObjectStoreMemoryBytes. Note that environment variables and third-party library settings are overridden by the RayRuntimeEnv configuration.

The following configuration formats are supported:

  • Plaintext: a comma-separated list of strings, where each string follows the key=value format. The key is a supported advanced parameter and the value is the corresponding setting.

  • JSON

Typical configuration scenarios:

  • Scenario 1: PyTorch advanced configuration

    Use advanced parameters to enable network connectivity between Workers for more flexible training approaches. For example, you can use the additional open ports to launch a framework such as Ray inside a DLC container and combine it with PyTorch for advanced distributed training. Example configuration:

    createSvcForAllWorkers=true,customPortNumPerWorker=100

    After this configuration, you can use the $JOB_NAME and $CUSTOM_PORTS environment variables in Startup Command to obtain the domain name and available port numbers, and then launch and connect to frameworks such as Ray.

  • Scenario 2: Manually configure RayRuntimeEnv for Ray (including dependencies and environment variables)

    Example configuration:

    {"RayRuntimeEnv": "{pip: requirements.txt, env_vars: {key: value}}"}

  • Scenario 3: Custom resource release policy

    Currently, only the pod-exit release policy is supported, which automatically releases resources when your Pod exits. Example configuration:

    {
      "ReleaseResourcePolicy": "pod-exit"
    }

VPC configuration

  • If no VPC is configured, the job uses the public network and the public gateway. The public gateway has limited bandwidth, which may cause job slowdowns or failures.

  • Configure a VPC and select the corresponding vSwitch and security group to improve bandwidth, stability, and security. Job clusters can also directly access services within the VPC.

    Important

    • When using a VPC, make sure the job resource group, dataset storage (OSS), and code repository are in the same region and have network connectivity.

    • When using CPFS datasets, a VPC must be configured and the selected VPC must match the CPFS network. Otherwise, the DLC job may get stuck in the environment preparation state for an extended period.

    • A VPC must be configured when submitting DLC jobs with Lingjun intelligent computing spot resources.

    You can also configure Internet Access Gateway. Two options are available:

    • Public Gateway: bandwidth is limited. Network speed may not meet requirements during high-concurrency workloads or large file downloads.

    • Private Gateway: to overcome public gateway bandwidth limits, create a public-facing NAT gateway in the DLC VPC, bind an EIP, and configure SNAT entries. For more information, see Improve internet access with a private gateway.

Fault tolerance and diagnostics

Parameter

Description

Automatic Fault Tolerance

Enable Automatic Fault Tolerance and configure the parameters to enable job detection and control. The system detects algorithm-level errors in real time and takes corrective actions to improve GPU utilization. For more information, see AIMaster: An elastic and automatic fault tolerance engine.

Note

When auto fault tolerance is enabled, an AIMaster instance runs alongside the job instance and consumes compute resources. AIMaster resource usage:

  • Resource quota: 1 CPU core and 1 GiB of memory.

  • Public resources: uses the ecs.c6.large instance specification.

Sanity Check

Enable Sanity Check to perform a comprehensive health check on all training resources, automatically isolate faulty nodes, and trigger backend automated operations. This reduces the likelihood of early-stage training failures and improves training success rates. For more information, see SanityCheck: Compute health checks.

Note

Health checks are available only for PyTorch jobs submitted with Lingjun intelligent computing resource quotas with at least one GPU.

Roles and permissions

Configure the RAM role for the instance. For more information, see Configure a RAM role for DLC.

Instance RAM role

Description

Default Role of PAI

Uses the AliyunPAIDLCDefaultRole service role, which has fine-grained permissions for accessing only MaxCompute and OSS. Temporary credentials issued under the default PAI role:

  • When accessing MaxCompute tables, the role has the same permissions as the DLC job owner.

  • When accessing OSS, the role can only access the default OSS bucket configured for the current workspace.

Custom Role

Select or enter a custom RAM role. When accessing cloud services from within the instance using STS temporary credentials, the permissions match those of the custom role.

Does Not Associate Role

Do not associate a RAM role with the DLC job. This is the default option.

References

Appendix

Create a job by using the SDK or CLI

Python SDK

Step 1: Install the Alibaba Cloud Credentials tool

To use Alibaba Cloud SDKs for resource management API calls, install the Credentials tool to configure your access credentials. Requirements:

  • Python version >= 3.7.

  • Alibaba Cloud SDK V2.0 is used.

pip install alibabacloud_credentials

Step 2: Obtain an AccessKey pair

This example uses the AccessKey pair to configure access credentials. To prevent credential leakage, we recommend that you configure the AccessKey pair as environment variables. The environment variable names are ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET.

Step 3: Install the Python SDK

  • Install the workspace SDK.

    pip install alibabacloud_aiworkspace20210204==3.0.1

  • Install the DLC SDK.

    pip install alibabacloud_pai_dlc20201203==1.4.17

Step 4: Submit a job

Use public resources to submit a job

The following sample code shows how to create and submit a job.

Sample code for creating and submitting a job

#!/usr/bin/env python3

from __future__ import print_function

import json
import time

from alibabacloud_tea_openapi.models import Config
from alibabacloud_credentials.client import Client as CredClient
from alibabacloud_pai_dlc20201203.client import Client as DLCClient
from alibabacloud_pai_dlc20201203.models import (
    ListJobsRequest,
    ListEcsSpecsRequest,
    CreateJobRequest,
    GetJobRequest,
)

from alibabacloud_aiworkspace20210204.client import Client as AIWorkspaceClient
from alibabacloud_aiworkspace20210204.models import (
    ListWorkspacesRequest,
    CreateDatasetRequest,
    ListDatasetsRequest,
    ListImagesRequest,
    ListCodeSourcesRequest
)

def create_nas_dataset(client, region, workspace_id, name,
                       nas_id, nas_path, mount_path):
    '''Create a NAS dataset.
    '''
    response = client.create_dataset(CreateDatasetRequest(
        workspace_id=workspace_id,
        name=name,
        data_type='COMMON',
        data_source_type='NAS',
        property='DIRECTORY',
        uri=f'nas://{nas_id}.{region}{nas_path}',
        accessibility='PRIVATE',
        source_type='USER',
        options=json.dumps({
            'mountPath': mount_path
        })
    ))
    return response.body.dataset_id

def create_oss_dataset(client, region, workspace_id, name,
                       oss_bucket, oss_endpoint, oss_path, mount_path):
    '''Create an OSS dataset.
    '''
    response = client.create_dataset(CreateDatasetRequest(
        workspace_id=workspace_id,
        name=name,
        data_type='COMMON',
        data_source_type='OSS',
        property='DIRECTORY',
        uri=f'oss://{oss_bucket}.{oss_endpoint}{oss_path}',
        accessibility='PRIVATE',
        source_type='USER',
        options=json.dumps({
            'mountPath': mount_path
        })
    ))
    return response.body.dataset_id

def wait_for_job_to_terminate(client, job_id):
    while True:
        job = client.get_job(job_id, GetJobRequest()).body
        print('job({}) is {}'.format(job_id, job.status))
        if job.status in ('Succeeded', 'Failed', 'Stopped'):
            return job.status
        time.sleep(5)
    return None

def main():

    # Make sure your Alibaba Cloud account has been authorized for DLC and has sufficient permissions.
    region_id = 'cn-hangzhou'
    # The AccessKey of an Alibaba Cloud account has access to all APIs. We recommend that you use a RAM user for API access and routine O&M.
    # We strongly recommend that you do not store the AccessKey ID and AccessKey Secret in your project code. Otherwise, your AccessKey may be leaked, threatening the security of all resources under your account.
    # In this example, the Credentials SDK is used to read the AccessKey from environment variables by default for identity verification.
    cred = CredClient()

    # 1. create client;
    workspace_client = AIWorkspaceClient(
        config=Config(
            credential=cred,
            region_id=region_id,
            endpoint="aiworkspace.{}.aliyuncs.com".format(region_id),
        )
    )

    dlc_client = DLCClient(
         config=Config(
            credential=cred,
            region_id=region_id,
            endpoint='pai-dlc.{}.aliyuncs.com'.format(region_id),
         )
    )

    print('------- Workspaces -----------')
    # Get the workspace list. You can also fill in the workspace name you created in the workspace_name parameter.
    workspaces = workspace_client.list_workspaces(ListWorkspacesRequest(
        page_number=1, page_size=1, workspace_name='',
        module_list='PAI'
    ))
    for workspace in workspaces.body.workspaces:
        print(workspace.workspace_name, workspace.workspace_id,
              workspace.status, workspace.creator)

    if len(workspaces.body.workspaces) == 0:
        raise RuntimeError('found no workspaces')

    workspace_id = workspaces.body.workspaces[0].workspace_id

    print('------- Images ------------')
    # Get the image list.
    images = workspace_client.list_images(ListImagesRequest(
        labels=','.join(['system.supported.dlc=true',
                         'system.framework=Tensorflow 1.15',
                         'system.pythonVersion=3.6',
                         'system.chipType=CPU'])))
    for image in images.body.images:
        print(json.dumps(image.to_map(), indent=2))

    image_uri = images.body.images[0].image_uri

    print('------- Datasets ----------')
    # Get datasets.
    datasets = workspace_client.list_datasets(ListDatasetsRequest(
        workspace_id=workspace_id,
        name='example-nas-data', properties='DIRECTORY'))
    for dataset in datasets.body.datasets:
        print(dataset.name, dataset.dataset_id, dataset.uri, dataset.options)

    if len(datasets.body.datasets) == 0:
        # Create the dataset when it does not exist.
        dataset_id = create_nas_dataset(
            client=workspace_client,
            region=region_id,
            workspace_id=workspace_id,
            name='example-nas-data',
            # NAS file system ID.
            # General-purpose NAS: 31a8e4****.
            # Extreme NAS: must start with extreme-, for example, extreme-0015****.
            # CPFS: must start with cpfs-, for example, cpfs-125487****.
            nas_id='***',
            nas_path='/',
            mount_path='/mnt/data/nas')
        print('create dataset with id: {}'.format(dataset_id))
    else:
        dataset_id = datasets.body.datasets[0].dataset_id

    print('------- Code Sources ----------')
    # Get the code source list.
    code_sources = workspace_client.list_code_sources(ListCodeSourcesRequest(
        workspace_id=workspace_id))
    for code_source in code_sources.body.code_sources:
        print(code_source.display_name, code_source.code_source_id, code_source.code_repo)

    print('-------- ECS SPECS ----------')
    # Get the list of DLC node specifications.
    ecs_specs = dlc_client.list_ecs_specs(ListEcsSpecsRequest(page_size=100, sort_by='Memory', order='asc'))
    for spec in ecs_specs.body.ecs_specs:
        print(spec.instance_type, spec.cpu, spec.memory, spec.memory, spec.gpu_type)

    print('-------- Create Job ----------')
    # Create a DLC job.
    create_job_resp = dlc_client.create_job(CreateJobRequest().from_map({
        'WorkspaceId': workspace_id,
        'DisplayName': 'sample-dlc-job',
        'JobType': 'TFJob',
        'JobSpecs': [
            {
                "Type": "Worker",
                "Image": image_uri,
                "PodCount": 1,
                "EcsSpec": ecs_specs.body.ecs_specs[0].instance_type,
            },
        ],
        "UserCommand": "echo 'Hello World' && ls -R /mnt/data/ && sleep 30 && echo 'DONE'",
        'DataSources': [
            {
                "DataSourceId": dataset_id,
            },
        ],
    }))
    job_id = create_job_resp.body.job_id

    wait_for_job_to_terminate(dlc_client, job_id)

    print('-------- List Jobs ----------')
    # Get the DLC job list.
    jobs = dlc_client.list_jobs(ListJobsRequest(
        workspace_id=workspace_id,
        page_number=1,
        page_size=10,
    ))
    for job in jobs.body.jobs:
        print(job.display_name, job.job_id, job.workspace_name,
              job.status, job.job_type)
    pass

if __name__ == '__main__':
    main()

Use a prepaid resource quota to submit a job

  1. Log on to the PAI console.

  2. View the workspace ID. In the left-side navigation pane, click Workspaces. Find the target workspace and click the ⓘ icon next to its name. In the information card that appears, view and copy the Workspace ID.

  3. View the resource quota ID. In the left-side navigation pane, choose AI Computing Resources > Quotas. Click the General Computing Resources tab and obtain the Quota ID from the Name/ID column in the resource quota list.

  4. Use the following code to create and submit a job. For the list of available public images, see Step 2: Prepare an image.

    from alibabacloud_pai_dlc20201203.client import Client
from alibabacloud_credentials.client import Client as CredClient
from alibabacloud_tea_openapi.models import Config
from alibabacloud_pai_dlc20201203.models import (
    CreateJobRequest,
    JobSpec,
    ResourceConfig, GetJobRequest
)

# Initialize a client to access DLC APIs.
region = 'cn-hangzhou'
# The AccessKey of an Alibaba Cloud account has access to all APIs. We recommend that you use a RAM user for API access and routine O&M.
# We strongly recommend that you do not store the AccessKey ID and AccessKey Secret in your project code. Otherwise, your AccessKey may be leaked, threatening the security of all resources under your account.
# In this example, the Credentials SDK is used to read the AccessKey from environment variables by default for identity verification.
cred = CredClient()
client = Client(
    config=Config(
        credential=cred,
        region_id=region,
        endpoint=f'pai-dlc.{region}.aliyuncs.com',
    )
)

# Declare the resource configuration for the job. For image selection, refer to the public image list in the documentation, or pass in your own image address.
spec = JobSpec(
    type='Worker',
    image=f'registry-vpc.cn-hangzhou.aliyuncs.com/pai-dlc/tensorflow-training:1.15-cpu-py36-ubuntu18.04',
    pod_count=1,
    resource_config=ResourceConfig(cpu='1', memory='2Gi')
)

# Declare the execution content for the job.
req = CreateJobRequest(
        resource_id='<Replace with your own resource quota ID>',
        workspace_id='<Replace with your own Workspace ID>',
        display_name='sample-dlc-job',
        job_type='TFJob',
        job_specs=[spec],
        user_command='echo "Hello World"',
)

# Submit the job.
response = client.create_job(req)
# Get the job ID.
job_id = response.body.job_id

# Query the job status.
job = client.get_job(job_id, GetJobRequest()).body
print('job status:', job.status)

# View the command executed by the job.
job.user_command

Use spot resources to submit a job

  • SpotDiscountLimit (spot discount)

    #!/usr/bin/env python3

from alibabacloud_tea_openapi.models import Config
from alibabacloud_credentials.client import Client as CredClient

from alibabacloud_pai_dlc20201203.client import Client as DLCClient
from alibabacloud_pai_dlc20201203.models import CreateJobRequest

region_id = '<region-id>'
cred = CredClient()
workspace_id = '12****' 

dlc_client = DLCClient(
    Config(credential=cred,
           region_id=region_id,
           endpoint='pai-dlc.{}.aliyuncs.com'.format(region_id),
           protocol='http'))

create_job_resp = dlc_client.create_job(CreateJobRequest().from_map({
    'WorkspaceId': workspace_id,
    'DisplayName': 'sample-spot-job',
    'JobType': 'PyTorchJob',
    'JobSpecs': [
        {
            "Type": "Worker",
            "Image": "dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/pytorch-training:1.12-cpu-py39-ubuntu20.04",
            "PodCount": 1,
            "EcsSpec": 'ecs.g7.xlarge',
            "SpotSpec": {
                "SpotStrategy": "SpotWithPriceLimit",
                "SpotDiscountLimit": 0.4,
            }
        },
    ],
    'UserVpc': {
        "VpcId": "vpc-0jlq8l7qech3m2ta2****",
        "SwitchId": "vsw-0jlc46eg4k3pivwpz8****",
        "SecurityGroupId": "sg-0jl4bd9wwh5auei9****",
    },
    "UserCommand": "echo 'Hello World' && ls -R /mnt/data/ && sleep 30 && echo 'DONE'",
}))
job_id = create_job_resp.body.job_id
print(f'jobId is {job_id}')

  • SpotPriceLimit (spot price)

    #!/usr/bin/env python3

from alibabacloud_tea_openapi.models import Config
from alibabacloud_credentials.client import Client as CredClient

from alibabacloud_pai_dlc20201203.client import Client as DLCClient
from alibabacloud_pai_dlc20201203.models import CreateJobRequest

region_id = '<region-id>'
cred = CredClient()
workspace_id = '12****'

dlc_client = DLCClient(
    Config(credential=cred,
           region_id=region_id,
           endpoint='pai-dlc.{}.aliyuncs.com'.format(region_id),
           protocol='http'))

create_job_resp = dlc_client.create_job(CreateJobRequest().from_map({
    'WorkspaceId': workspace_id,
    'DisplayName': 'sample-spot-job',
    'JobType': 'PyTorchJob',
    'JobSpecs': [
        {
            "Type": "Worker",
            "Image": "dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/pytorch-training:1.12-cpu-py39-ubuntu20.04",
            "PodCount": 1,
            "EcsSpec": 'ecs.g7.xlarge',
            "SpotSpec": {
                "SpotStrategy": "SpotWithPriceLimit",
                "SpotPriceLimit": 0.011,
            }
        },
    ],
    'UserVpc': {
        "VpcId": "vpc-0jlq8l7qech3m2ta2****",
        "SwitchId": "vsw-0jlc46eg4k3pivwpz8****",
        "SecurityGroupId": "sg-0jl4bd9wwh5auei9****",
    },
    "UserCommand": "echo 'Hello World' && ls -R /mnt/data/ && sleep 30 && echo 'DONE'",
}))
job_id = create_job_resp.body.job_id
print(f'jobId is {job_id}')

The following table describes the key parameters.

Parameter

Description

SpotStrategy

The preemption policy. This parameter takes effect only when this parameter is set toSpotWithPriceLimit.

SpotDiscountLimit

The bidding type isspotdiscount.

Note

  • SpotDiscountLimitandSpotPriceLimitcannot be set at the same time.

  • For Lingjun preemptible resources, onlySpotDiscountLimitis supported.

SpotPriceLimit

The bidding type isspotprice.

UserVpc

Required when you use Lingjun preemptible resources to submit a job. Specify the VPC ID, vSwitch ID, and security group ID in the region where the job runs.

Note

When you create a DLC job by calling the API or using the SDK (createJob), you cannot directly upload local code files. You must first upload your code to one of the following locations:

  • Git repository: Create a code configuration through AI asset management and associate it with a Git repository URL.

  • OSS storage: Upload the code to OSS and then provide it to the job through dataset mounting.

  • Custom image: Package the code into a Docker image.

CLI

Step 1: Download the CLI tool and complete user authentication

Download the CLI tool for Linux 64-bit or Mac based on your operating system and complete user authentication. For more information, see Before you begin.

Step 2: Submit a job

  1. Log on to the PAI console.

  2. View your workspace ID on the Workspaces page. In the left-side navigation pane, click Workspaces. Find the target workspace and click the ⓘ icon next to its name. In the information panel that appears, view the Workspace ID.

  3. View your resource quota ID. In the left-side navigation pane, choose AI Computing Resources > Quotas. Select the target resource type tab (such as General Computing Resources) and obtain the resource quota ID from the Name/ID column.

  4. Prepare a parameter file tfjob.params based on the following file content. For more information about how to configure a parameter file, see Commands used to submit jobs.

    name=test_cli_tfjob_001
workers=1
worker_cpu=4
worker_gpu=0
worker_memory=4Gi
worker_shared_memory=4Gi
worker_image=registry-vpc.cn-beijing.aliyuncs.com/pai-dlc/tensorflow-training:1.12.2PAI-cpu-py27-ubuntu16.04
command=echo good && sleep 120
resource_id=<Replace with your resource quota ID>
workspace_id=<Replace with your WorkspaceID>

  5. Use the following code sample to pass the params_file parameter and submit a job. You can submit the DLC job to the specified workspace and resource quota.

    ./dlc submit tfjob --job_file  ./tfjob.params

  6. Use the following code to view the DLC job that you submitted.

    ./dlc get job <jobID>

Advanced parameters

Parameter (key)

Supported framework

Description

Value

ReleaseResourcePolicy

ALL

Configure a custom resource release policy. If left unspecified, all pod resources are released when the job completes. If configured, the only supported value is pod-exit, which releases the resources of a pod when the pod exits.

pod-exit

EnableNvidiaIBGDA

ALL

Specifies whether to enable IBGDA when loading the GPU driver.

true or false

EnableNvidiaGDRCopy

ALL

Specifies whether to install the GDRCopy kernel module. Version 2.4.4 is installed.

true or false

EnablePaiNUMACoreBinding

ALL

Specifies whether to enable NUMA.

true or false

EnableResourcePreCheck

ALL

Specifies whether to verify that the total resources (node specifications) in the quota can meet the specifications of all roles in the job when a job is submitted.

true or false

createSvcForAllWorkers

PyTorch

Whether to allow network communication between workers.

  • If set to true, all PyTorch workers can communicate with each other.

  • If set to false or left unspecified, only the master is accessible by default.

After this parameter is enabled, the domain name of each worker is the worker name, such as dlcxxxxx-master-0. The job name (dlcxxxxx) is passed to the worker through the JOB_NAME environment variable. You can then derive the domain name of the specific worker that you want to access.

true or false

customPortList

PyTorch

Allows you to specify the network ports to open on each worker. This parameter can be used together with createSvcForAllWorkers to enable network communication between workers.

If left unspecified, only port 23456 is open on the master by default. Make sure to avoid port 23456 in the custom port list.

Important

This parameter is mutually exclusive with customPortNumPerWorker. Do not set both parameters at the same time.

A semicolon-separated list of strings, where each string is a port number or a port range connected by a hyphen, such as 10000;10001-10010 (which is expanded to 11 consecutive port numbers from 10000 to 10010)

customPortNumPerWorker

PyTorch

Allows you to request a number of network ports to open on each worker. This parameter can be used together with createSvcForAllWorkers to enable network communication between workers.

If left unspecified, only port 23456 is open on the master by default. DLC randomly assigns ports to workers based on the number of ports specified in this parameter. The assigned port numbers are passed to the workers through the CUSTOM_PORTS environment variable in the format of a semicolon-separated list of port numbers.

Important

  • This parameter is mutually exclusive with customPortList. Do not set both parameters at the same time.

  • The customPortNumPerWorker parameter is not supported when you use Lingjun intelligent computing resources to submit DLC jobs, because Lingjun intelligent computing resources do not provide custom port capabilities.

Integer (maximum: 65536)

RayRuntimeEnv

Ray

When the framework is Ray, you can manually configure RayRuntimeEnv to define the runtime environment.

Important

Environment variable and third-party library configurations will be overridden by this setting.

Configure environment variables and third-party libraries ({pip: requirements.txt, env_vars: {key: value}})

RayRedisAddress

Ray

The address of the external GCS Redis.

String

RayRedisUsername

Ray

The username of the external GCS Redis.

String

RayRedisPassword

Ray

The password of the external GCS Redis.

String

RaySubmitterBackoffLimit

Ray

The number of submitter retries.

Positive integer

RayObjectStoreMemoryBytes

Ray

Configure shared memory for nodes. For example, to allocate 1 GiB of shared memory per node, use the following configuration:

{
  "RayObjectStoreMemoryBytes": "1073741824"
}

Positive integer

Previous：PreparationsNext：Use cloud storage