AI distributed training job creation-Platform For AI(PAI)-阿里云帮助中心

PAI-DLC lets you quickly create single-node or distributed training jobs. Kubernetes automatically provisions compute nodes under the hood, eliminating the need to manually purchase machines or configure runtime environments. It supports TensorFlow, PyTorch, and other popular deep learning frameworks, with flexible resource configuration options.

Quick start

Using MNIST handwritten digit recognition as an example, this guide walks you through single-node, single-GPU training and multi-node, multi-GPU distributed training with DLC. For details, see Get started with Deep Learning Containers (DLC).

Console parameter reference

Basic information

Parameter	Description
Job Name	Set an easily identifiable name for the job.
Tag	Apply standardized tags to your jobs. Tags provide the foundation for archiving jobs by label, analyzing job distribution, and tracking resource consumption. The following constraints apply: Key constraints: Must not start with `aliyun` or `acs:`. Must not contain `http://` or `https://`. Maximum length: 128 characters. Keys can contain only letters, numbers, hyphens (-), underscores (_), and periods (.). Value constraints: Must not contain `http://` or `https://`. Maximum length: 256 characters. Separate multiple values with commas. For guidance on standardized tagging, see Standardize tagging for DLC, DSW, and EAS.
Task Description	Use this field to describe the purpose, background, or other context of the job.

Environment information

Parameter	Description
Image Configuration	In addition to the Alibaba Cloud Image, the following image types are supported: Custom Image: Use custom images added to PAI. The image repository must be set to public, or the image must be stored in Alibaba Cloud Container Registry (ACR). For details, see Custom images. Note When you select Lingjun resources and use a custom image, you must manually install RDMA drivers to take full advantage of the high-performance RDMA network. For details, see RDMA: High-performance networking for distributed training. Image Address: Supports configuring publicly accessible custom or official image addresses. For private image addresses, click Enter credentials and configure the image repository username and password. To improve image pull speed, see Image acceleration.
Mount dataset	Datasets provide the data files needed for model training. The following two dataset types are supported: Custom Dataset: You can create a custom dataset to store the data files needed for training. You can enable Read-only mode and select a dataset version from the version list. Public Dataset: PAI provides built-in public datasets, which support only read-only mounting. Mount Path: The path where the dataset is mounted in the DLC container, for example `/mnt/data`. You can access the dataset through this path in your code. For more mount configuration details, see Use cloud storage. Important If you use a CPFS dataset, you must configure a VPC for DLC that matches the CPFS VPC. Otherwise, the submitted job may remain in the Preparing environment state for an extended period.
Mount storage	You can also mount data source paths directly to read required data or store intermediate and result files. Supported data source types: OSS, General-purpose NAS, Extreme NAS, CPFS, and BMCPFS (available only with Lingjun resources). Advanced Settings: Different data source types support feature-specific capabilities through advanced settings. For example: OSS: Set `{"mountType":"ossfs"}` in advanced settings to mount OSS storage using ossfs. General-purpose NAS and CPFS: Set the nconnect parameter in advanced settings to improve NAS throughput from the DLC container. For details, see How do I improve NAS performance on Linux?. For example, `{"nconnect":"<sample value>"}`, where `<sample value>` is replaced with a specific positive integer. For more information, see Use cloud storage.
Startup Command	Set the job startup command. Shell commands are supported. DLC automatically injects common PyTorch and TensorFlow environment variables such as `MASTER_ADDR` and `WORLD_SIZE`. You can reference them using `$VARIABLE_NAME`. Example startup commands: Run Python: `python -c "print('Hello World')"` PyTorch multi-node, multi-GPU distributed training: `python -m torch.distributed.launch \ --nproc_per_node=2 \ --master_addr=${MASTER_ADDR} \ --master_port=${MASTER_PORT} \ --nnodes=${WORLD_SIZE} \ --node_rank=${RANK} \ train.py --epochs=100` Specify a shell script path as the startup command: `/ml/input/config/launch.sh` Important The mount path configured for code (such as /mnt/data) is the location of your code files within the DLC container. Your startup command must reference code files using this path. For example, if the code mount path is /mnt/data and your code file is train.py, the startup command should be: python /mnt/data/train.py

Expand for more settings: environment variables, third-party libraries, code configuration

Environment Variable

In addition to the automatically injected common PyTorch and TensorFlow environment variables, you can provide custom environment variables in Key:Value format. Up to 20 environment variables are supported.

Third-party Libraries

If the container image is missing some third-party libraries, you can add them through Third-party Libraries. Two methods are supported:

Select from List: Enter the library names directly in the text box below. Separate multiple library names with spaces, for example griffon numpy pandas.
Directory of requirements.txt: Add the libraries to a requirements.txt file, upload it to the DLC container through code configuration, dataset, or direct mount, and then specify the file path in the container in the text box.

Code Builds

Upload the code files needed for training to the DLC container. Two configuration methods are supported:

Online configuration: If you have an existing Git repository and have access permissions, you can link it using Create a code configuration so that DLC can retrieve the job code.
Local Upload: Click the button to upload local code files. After a successful upload, configure Mount path to a specific path within the container, for example /mnt/data.

Resource information

Parameter	Description
Resource Type	Defaults to General Computing. Lingjun Intelligence Resources is available in the following regions: China (Ulanqab), Singapore, China (Shenzhen), China (Beijing), China (Shanghai), China (Hangzhou), China (Guangzhou), China (Hong Kong), Malaysia, Germany, and Atlanta.
Source	Public Resources: Billing model: Pay-as-you-go. Suitable for workloads with infrequent jobs and no strict timing requirements. Public resources may require queuing. Resource limits: Up to 2 GPUs and 8 CPU cores. To exceed these limits, contact your account manager to raise the quota. Resource Quota: Includes general-purpose computing resources or Lingjun resources. Billing model: Subscription. Suitable for scenarios with high job volume that require guaranteed stable resources. Special parameters: Resource Quota: Set the amount of GPU, CPU, and other resources. For how to prepare resource quotas, see Create a resource quota. Priority: Indicates the execution priority of concurrently running jobs. Value range: [1, 9], where 1 is the lowest priority. Pre-check: Automatically verifies GPU model and image compatibility before job submission, catching configuration errors early and preventing job failures. Preemptible Resources: Billing model: Pay-as-you-go. Suitable for cost-sensitive scenarios. Spot resource pricing is lower than pay-as-you-go. No availability guarantee. Resources may not be immediately available or may be reclaimed. For more information, see Use a preemptible job.
Framework	The following deep learning training frameworks and tools are supported: TensorFlow, PyTorch, ElasticBatch, XGBoost, OneFlow, MPIJob, Slurm, Ray, Custom, DataJuicer, and MPI. Note When Resource Quota is set to Lingjun resources, only TensorFlow, PyTorch, ElasticBatch, MPIJob, Slurm, and Ray jobs are supported.
Job Resource	Based on the selected Framework, you can configure resources for Worker, PS, Chief, Evaluator, and GraphLearn node types. When using the Ray framework, you can click Add Role to customize Worker roles, enabling mixed execution of heterogeneous resources. Using public resources: The following parameters can be configured: Number of Nodes: The number of nodes running the DLC job. Resource Type: Select the instance type. The console displays the corresponding price. For more billing information, see DLC billing. Using resource quotas: You can configure the number of nodes, CPU cores, GPUs, memory (GiB), and shared memory (GiB) for each node type. The following special parameters are also available: Node-Specific Scheduling: Run jobs on specific compute nodes. Idle Resources: When using idle-time resources, jobs can run on idle capacity from other quotas, improving resource utilization. However, when the original quota needs these resources, idle-time jobs are automatically terminated and resources are returned. For more information, see Use idle-time resources. CPU Affinity: Enable CPU affinity to bind processes to specific CPU cores, reducing cache misses and context switches to improve performance. Suitable for scenarios with high performance and real-time requirements. Using spot resources: In addition to node count and instance type, you can configure the Bid Price parameter, which sets the maximum bid for spot resources. Click the button to select a bidding method: By discount: The maximum price is based on the market price of the instance type, with discrete options from 10% to 90% of the market price, representing the bidding cap. Spot resources are allocated when the maximum bid is greater than or equal to the market price and sufficient capacity is available. By price: The maximum bid range is within the market price range.
Scheduling Policy	You can customize the container resource scheduling strategy based on your business scenarios and job characteristics: Intelligent: GPU jobs are consolidated onto the fewest GPU nodes, while CPU jobs are distributed across CPU nodes. This is the default strategy. Centralized: Jobs are consolidated onto the fewest nodes. This strategy helps reduce resource fragmentation and is ideal for GPU model training. Spread: Jobs are spread across nodes. This strategy improves the job dequeue success rate and is ideal for distributed CPU jobs.

Note

DLC does not support directly modifying the instance type of a submitted job. To change the instance type (for example, to switch to an instance that supports eRDMA high-performance networking, such as ecs.ebmgn8v.48xlarge), use the Clone Job feature to create a new job, select the target instance type, and submit it. The platform automatically enables high-performance networking and configures NCCL-related parameters. To change the NVIDIA driver and CUDA version of an instance, use the instance configuration feature to change the driver version (for example, to version 580). The platform automatically pulls the corresponding DLC image based on the new configuration to update the runtime environment, completing the driver change, CUDA upgrade, or GPU driver replacement.

Expand for more settings: maximum running duration, retention period, framework advanced settings

Maximum Duration	Set the maximum duration a job can run. Jobs that exceed this duration are automatically stopped. Default: 30 days.
Retention Period	Configure how long completed or failed jobs are retained. Note that retention continues to consume resources. Jobs that exceed this duration are deleted. Important Deleted DLC jobs cannot be restored. Proceed with caution.
Start Developer Machine	When the resource source is a resource quota, you can launch a DSW instance for online debugging. In the instance list area of the job overview page, click DSW in the Actions column to access the instance.
Advanced Framework Configuration	For the list of configurable parameters and their values, see Advanced parameter list. The parameters `ReleaseResourcePolicy`, `EnableNvidiaIBGDA`, `EnableNvidiaGDRCopy`, `EnablePaiNUMACoreBinding`, and `EnableResourcePreCheck` are supported by all frameworks. When the framework is PyTorch, the supported parameters are: `createSvcForAllWorkers`, `customPortList`, and `customPortNumPerWorker`. Important Because Lingjun resources do not provide custom port capability, the customPortNumPerWorker parameter is not supported when submitting DLC jobs with Lingjun resources. When the framework is Ray, the supported parameters are: `RayRuntimeEnv`, `RayRedisAddress`, `RayRedisUsername`, `RayRedisPassword`, `RaySubmitterBackoffLimit`, and `RayObjectStoreMemoryBytes`. Note that environment variables and third-party library configurations will be overridden by the RayRuntimeEnv configuration. The following configuration formats are supported: Plaintext: Configure as a comma-separated (,) string where each entry follows the `key=value` format. The key is a supported advanced parameter, and the value is the corresponding parameter value. JSON Typical configuration scenarios: Scenario 1: PyTorch advanced configuration Advanced configuration parameters enable network connectivity between Workers, allowing for more flexible training approaches. For example, you can use the additional exposed ports to start a framework like Ray inside the DLC container and combine it with PyTorch for advanced distributed training. Example configuration: `createSvcForAllWorkers=true,customPortNumPerWorker=100` You can then use the Startup Command to configure the `$JOB_NAME` and `$CUSTOM_PORTS` environment variables to obtain the domain name and available port numbers, and start and connect to frameworks like Ray. Scenario 2: Manually configure RayRuntimeEnv for the Ray framework (including dependencies and environment variables) Example configuration: `{"RayRuntimeEnv": "{pip: requirements.txt, env_vars: {key: value}}"}` Scenario 3: Custom resource release policy Currently, the only supported release policy is pod-exit, which automatically releases resources when your Pod exits. Example configuration: `{ "ReleaseResourcePolicy": "pod-exit" }`

VPC configuration

If no VPC is configured, the public network and public gateway are used. Because the public gateway has limited bandwidth, jobs may experience slowdowns or fail to run properly during execution.
Configuring a VPC and selecting the corresponding vSwitch and security group improves network bandwidth, stability, and security. The cluster running your jobs can also directly access services within this VPC.
Important
- When using a VPC, ensure that the job resource group instances, dataset storage (OSS), and code repository are all within the same VPC in the same region, and that network connectivity is established.
- When using a CPFS dataset, you must configure a VPC that matches the CPFS VPC. Otherwise, the submitted DLC training job may remain in the Preparing environment state for an extended period.
- When submitting DLC jobs using Lingjun spot resources, you must configure a VPC.
Additionally, you can configure Internet Access Gateway. The following two options are supported:
- Public Gateway: Bandwidth is limited. During periods of high concurrency or when downloading large files, network speed may not meet demand.
- Private Gateway: To overcome the public gateway bandwidth limitation, you can create a NAT gateway in the DLC VPC, associate an EIP, and configure SNAT entries. For details, see Improve internet access with a private gateway.

Fault tolerance and diagnostics

Parameter	Description
Automatic Fault Tolerance	After enabling the Automatic Fault Tolerance feature and configuring the relevant parameters, the system provides job monitoring and control capabilities. It can detect algorithm-level errors in real time and take corrective action, improving GPU utilization. For details, see AIMaster: An elastic and automatic fault tolerance engine. Note When automatic fault tolerance is enabled, the system launches an AIMaster instance alongside the job instances, which consumes additional compute resources. AIMaster instance resource usage details: Resource quota: 1 CPU core and 1 GiB memory. Public resources: Uses the ecs.c6.large instance type.
Sanity Check	After enabling the Sanity Check feature, the health check thoroughly inspects the resources involved in training, automatically isolates faulty nodes, and triggers background automated operations. This effectively reduces the likelihood of issues during the early stages of training and improves the training success rate. For details, see SanityCheck: Compute health checks. Note The health check feature is supported only for PyTorch training jobs submitted with Lingjun resource quotas and when the GPU count is greater than 0.

Roles and permissions

Instance RAM role configuration is described below. For details, see Configure a RAM role for DLC.

Instance RAM role	Description
Default Role of PAI	Operates based on the service role AliyunPAIDLCDefaultRole, which has access only to MaxCompute and OSS, with more granular permissions. Temporary credentials issued based on the PAI default role: When accessing MaxCompute tables, it has the same permissions as the DLC instance owner. When accessing OSS, it can only access the default OSS bucket configured for the current workspace.
Custom Role	Select or specify a custom RAM role. When accessing cloud services from within the instance using STS temporary credentials, the permissions are consistent with those of the custom role.
Does Not Associate Role	No RAM role is associated with the DLC job. This is the default option.

FAQ

Q: What should I do if I get a BadRequest error about selecting a NAS data source with a mount target when creating a training job?

This error occurs when the NAS data source lacks a mount target. Create a new mount target in the NAS console, then select it as the data source in the Mount dataset or Mount storage configuration when creating the training job, and resubmit the job.

Q: Can DLC training jobs export metrics to Weights & Biases (W&B)?

Yes, DLC supports exporting training metrics to W&B, but the following prerequisites must be met:

Network connectivity: Ensure that PAI training nodes can access the public endpoint api.wandb.ai. If using a VPC, configure a NAT gateway.
Key management: Inject the W&B API key through environment variables. Do not hardcode it in your code.
Compliance: Upload only non-sensitive scalar metrics. Do not upload raw data or sensitive model details.

Q: Does a failed DLC job still appear in the job list?

Whether it appears depends on the failure reason:

Jobs that fail due to insufficient resources temporarily appear in the DLC console job list and are removed after deletion.
When the API returns an error directly (with a RequestID) due to invalid request parameters, no record is kept in the job list. The error message is displayed only on the operation page.

Q: What should I do if I see an "account security protection triggered" or "resource allocation suspended" message when creating a DLC job?

This message is typically triggered by account risk control policies. Use the Alibaba Cloud self-service diagnostic tool (https://smartservice.console.alibabacloud.com/service/self-service-center/search?questionId=27591) to diagnose why payment or resource allocation cannot proceed, and submit an appeal. Once the appeal is approved, you can retry creating the job.

Appendix

Create a job by using the SDK or CLI

Python SDK

Step 1: Install the Alibaba Cloud Credentials tool

When you call OpenAPI by using an Alibaba Cloud SDK to manage resources, you must install the Credentials tool to configure your credentials. Requirements:

Python version >= 3.7.
Use V2.0 Alibaba Cloud SDKs.

pip install alibabacloud_credentials

Step 2: Obtain the AccessKey of your account

This example uses AccessKey (AK) information to configure access credentials. To prevent credential leakage, we recommend that you store your AccessKey in environment variables. The environment variable names for the ID and secret are ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET.

For information about how to obtain an AccessKey, see Create an AccessKey.
For information about how to configure environment variables, see Configure environment variables.
For other credential configuration methods, see Install the Credentials tool.

Step 3: Install the Python SDK

Install the workspace SDK.

pip install alibabacloud_aiworkspace20210204==3.0.1

Install the DLC SDK.

pip install alibabacloud_pai_dlc20201203==1.4.17

Step 4: Submit a job

Submit a job by using public resources

The following sample code shows how to create and submit a job:

Sample code for creating and submitting a job

#!/usr/bin/env python3

from __future__ import print_function

import json
import time

from alibabacloud_tea_openapi.models import Config
from alibabacloud_credentials.client import Client as CredClient
from alibabacloud_pai_dlc20201203.client import Client as DLCClient
from alibabacloud_pai_dlc20201203.models import (
    ListJobsRequest,
    ListEcsSpecsRequest,
    CreateJobRequest,
    GetJobRequest,
)

from alibabacloud_aiworkspace20210204.client import Client as AIWorkspaceClient
from alibabacloud_aiworkspace20210204.models import (
    ListWorkspacesRequest,
    CreateDatasetRequest,
    ListDatasetsRequest,
    ListImagesRequest,
    ListCodeSourcesRequest
)

def create_nas_dataset(client, region, workspace_id, name,
                       nas_id, nas_path, mount_path):
    '''Create a NAS dataset.
    '''
    response = client.create_dataset(CreateDatasetRequest(
        workspace_id=workspace_id,
        name=name,
        data_type='COMMON',
        data_source_type='NAS',
        property='DIRECTORY',
        uri=f'nas://{nas_id}.{region}{nas_path}',
        accessibility='PRIVATE',
        source_type='USER',
        options=json.dumps({
            'mountPath': mount_path
        })
    ))
    return response.body.dataset_id

def create_oss_dataset(client, region, workspace_id, name,
                       oss_bucket, oss_endpoint, oss_path, mount_path):
    '''Create an OSS dataset.
    '''
    response = client.create_dataset(CreateDatasetRequest(
        workspace_id=workspace_id,
        name=name,
        data_type='COMMON',
        data_source_type='OSS',
        property='DIRECTORY',
        uri=f'oss://{oss_bucket}.{oss_endpoint}{oss_path}',
        accessibility='PRIVATE',
        source_type='USER',
        options=json.dumps({
            'mountPath': mount_path
        })
    ))
    return response.body.dataset_id

def wait_for_job_to_terminate(client, job_id):
    while True:
        job = client.get_job(job_id, GetJobRequest()).body
        print('job({}) is {}'.format(job_id, job.status))
        if job.status in ('Succeeded', 'Failed', 'Stopped'):
            return job.status
        time.sleep(5)
    return None

def main():

    # Please make sure that your primary account has authorized DLC and has sufficient permissions.
    region_id = 'cn-hangzhou'
    # The Alibaba Cloud account AccessKey has access permissions to all APIs. It is recommended to use RAM users for API access or daily operations and maintenance.
    # It is highly recommended not to save AccessKey ID and AccessKey Secret directly in the project code to prevent leak risks, which could threaten the security of all resources under your account.
    # This example uses the Credentials SDK to read the AccessKey from environment variables by default for identity authentication.
    cred = CredClient()

    # 1. create client;
    workspace_client = AIWorkspaceClient(
        config=Config(
            credential=cred,
            region_id=region_id,
            endpoint="aiworkspace.{}.aliyuncs.com".format(region_id),
        )
    )

    dlc_client = DLCClient(
         config=Config(
            credential=cred,
            region_id=region_id,
            endpoint='pai-dlc.{}.aliyuncs.com'.format(region_id),
         )
    )

    print('------- Workspaces -----------')
    # Get the list of workspaces. You can also specify the workspace name you created in the workspace_name parameter.
    workspaces = workspace_client.list_workspaces(ListWorkspacesRequest(
        page_number=1, page_size=1, workspace_name='',
        module_list='PAI'
    ))
    for workspace in workspaces.body.workspaces:
        print(workspace.workspace_name, workspace.workspace_id,
              workspace.status, workspace.creator)

    if len(workspaces.body.workspaces) == 0:
        raise RuntimeError('found no workspaces')

    workspace_id = workspaces.body.workspaces[0].workspace_id

    print('------- Images ------------')
    # Get the list of images.
    images = workspace_client.list_images(ListImagesRequest(
        labels=','.join(['system.supported.dlc=true',
                         'system.framework=Tensorflow 1.15',
                         'system.pythonVersion=3.6',
                         'system.chipType=CPU'])))
    for image in images.body.images:
        print(json.dumps(image.to_map(), indent=2))

    image_uri = images.body.images[0].image_uri

    print('------- Datasets ----------')
    # Get the dataset.
    datasets = workspace_client.list_datasets(ListDatasetsRequest(
        workspace_id=workspace_id,
        name='example-nas-data', properties='DIRECTORY'))
    for dataset in datasets.body.datasets:
        print(dataset.name, dataset.dataset_id, dataset.uri, dataset.options)

    if len(datasets.body.datasets) == 0:
        # Create the dataset if the current dataset does not exist.
        dataset_id = create_nas_dataset(
            client=workspace_client,
            region=region_id,
            workspace_id=workspace_id,
            name='example-nas-data',
            # NAS file system ID.
            # General-purpose NAS: 31a8e4****.
            # Extreme NAS: Must start with "extreme-", e.g., extreme-0015****.
            # CPFS: Must start with "cpfs-", e.g., cpfs-125487****.
            nas_id='***',
            nas_path='/',
            mount_path='/mnt/data/nas')
        print('create dataset with id: {}'.format(dataset_id))
    else:
        dataset_id = datasets.body.datasets[0].dataset_id

    print('------- Code Sources ----------')
    # Get the list of code sources.
    code_sources = workspace_client.list_code_sources(ListCodeSourcesRequest(
        workspace_id=workspace_id))
    for code_source in code_sources.body.code_sources:
        print(code_source.display_name, code_source.code_source_id, code_source.code_repo)

    print('-------- ECS SPECS ----------')
    # Get the list of ECS specifications for DLC.
    ecs_specs = dlc_client.list_ecs_specs(ListEcsSpecsRequest(page_size=100, sort_by='Memory', order='asc'))
    for spec in ecs_specs.body.ecs_specs:
        print(spec.instance_type, spec.cpu, spec.memory, spec.memory, spec.gpu_type)

    print('-------- Create Job ----------')
    # Create a DLC job.
    create_job_resp = dlc_client.create_job(CreateJobRequest().from_map({
        'WorkspaceId': workspace_id,
        'DisplayName': 'sample-dlc-job',
        'JobType': 'TFJob',
        'JobSpecs': [
            {
                "Type": "Worker",
                "Image": image_uri,
                "PodCount": 1,
                "EcsSpec": ecs_specs.body.ecs_specs[0].instance_type,
            },
        ],
        "UserCommand": "echo 'Hello World' && ls -R /mnt/data/ && sleep 30 && echo 'DONE'",
        'DataSources': [
            {
                "DataSourceId": dataset_id,
            },
        ],
    }))
    job_id = create_job_resp.body.job_id

    wait_for_job_to_terminate(dlc_client, job_id)

    print('-------- List Jobs ----------')
    # Get the list of DLC jobs.
    jobs = dlc_client.list_jobs(ListJobsRequest(
        workspace_id=workspace_id,
        page_number=1,
        page_size=10,
    ))
    for job in jobs.body.jobs:
        print(job.display_name, job.job_id, job.workspace_name,
              job.status, job.job_type)
    pass

if __name__ == '__main__':
    main()

Submit a job by using a subscription resource quota

Log on to the PAI console.
View the workspace ID: In the left-side navigation pane, click Workspace List. Find the target workspace, click the ⓘ icon next to its name, and view and copy the Workspace ID in the information panel that appears.
View the resource quota ID of your dedicated resource group. In the left-side navigation pane, choose AI Computing Resources > Resource Quota. Click the General Computing Resources tab, and obtain the Quota ID of the target resource quota from the Name/ID column in the resource quota list.

Use the following code to create and submit a job. For the list of public images that you can use, see Step 2: Prepare an image.

from alibabacloud_pai_dlc20201203.client import Client
from alibabacloud_credentials.client import Client as CredClient
from alibabacloud_tea_openapi.models import Config
from alibabacloud_pai_dlc20201203.models import (
    CreateJobRequest,
    JobSpec,
    ResourceConfig, GetJobRequest
)

# Initialize a Client to access the DLC API.
region = 'cn-hangzhou'
# The Alibaba Cloud account AccessKey has access permissions to all APIs. It is recommended to use RAM users for API access or daily operations and maintenance.
# It is highly recommended not to save AccessKey ID and AccessKey Secret directly in the project code to prevent leak risks, which could threaten the security of all resources under your account.
# This example uses the Credentials SDK to read the AccessKey from environment variables by default for identity authentication.
cred = CredClient()
client = Client(
    config=Config(
        credential=cred,
        region_id=region,
        endpoint=f'pai-dlc.{region}.aliyuncs.com',
    )
)

# Declare the resource configuration for the job. For image selection, refer to the public image list in the documentation, or you can pass in your own image address.
spec = JobSpec(
    type='Worker',
    image=f'registry-vpc.cn-hangzhou.aliyuncs.com/pai-dlc/tensorflow-training:1.15-cpu-py36-ubuntu18.04',
    pod_count=1,
    resource_config=ResourceConfig(cpu='1', memory='2Gi')
)

# Declare the execution content of the job.
req = CreateJobRequest(
        resource_id='<Replace with your own resource quota ID>',
        workspace_id='<Replace with your own WorkspaceID>',
        display_name='sample-dlc-job',
        job_type='TFJob',
        job_specs=[spec],
        user_command='echo "Hello World"',
)

# Submit the job.
response = client.create_job(req)
# Get the job ID.
job_id = response.body.job_id

# Query the job status.
job = client.get_job(job_id, GetJobRequest()).body
print('job status:', job.status)

# View the command executed by the job.
job.user_command

Submit a job by using spot resources

SpotDiscountLimit (Spot discount)

#!/usr/bin/env python3

from alibabacloud_tea_openapi.models import Config
from alibabacloud_credentials.client import Client as CredClient

from alibabacloud_pai_dlc20201203.client import Client as DLCClient
from alibabacloud_pai_dlc20201203.models import CreateJobRequest

region_id = '<region-id>'  # The region ID where the DLC job is located, for example, cn-hangzhou for China (Hangzhou).
cred = CredClient()
workspace_id = '12****'  # The workspace ID where the DLC job is located.

dlc_client = DLCClient(
    Config(credential=cred,
           region_id=region_id,
           endpoint='pai-dlc.{}.aliyuncs.com'.format(region_id),
           protocol='http'))

create_job_resp = dlc_client.create_job(CreateJobRequest().from_map({
    'WorkspaceId': workspace_id,
    'DisplayName': 'sample-spot-job',
    'JobType': 'PyTorchJob',
    'JobSpecs': [
        {
            "Type": "Worker",
            "Image": "dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/pytorch-training:1.12-cpu-py39-ubuntu20.04",
            "PodCount": 1,
            "EcsSpec": 'ecs.g7.xlarge',
            "SpotSpec": {
                "SpotStrategy": "SpotWithPriceLimit",
                "SpotDiscountLimit": 0.4,
            }
        },
    ],
    'UserVpc': {
        "VpcId": "vpc-0jlq8l7qech3m2ta2****",
        "SwitchId": "vsw-0jlc46eg4k3pivwpz8****",
        "SecurityGroupId": "sg-0jl4bd9wwh5auei9****",
    },
    "UserCommand": "echo 'Hello World' && ls -R /mnt/data/ && sleep 30 && echo 'DONE'",
}))
job_id = create_job_resp.body.job_id
print(f'jobId is {job_id}')

SpotPriceLimit (Spot price)

#!/usr/bin/env python3

from alibabacloud_tea_openapi.models import Config
from alibabacloud_credentials.client import Client as CredClient

from alibabacloud_pai_dlc20201203.client import Client as DLCClient
from alibabacloud_pai_dlc20201203.models import CreateJobRequest

# The region ID where the DLC job is located, for example, cn-hangzhou.
region_id = '<region-id>'
# Initialize the credentials client to handle authentication.
cred = CredClient()
# The workspace ID where the DLC job is located.
workspace_id = '12****'

# Initialize the DLC client with the required configuration.
dlc_client = DLCClient(
    Config(credential=cred,
           region_id=region_id,
           endpoint='pai-dlc.{}.aliyuncs.com'.format(region_id),
           protocol='http'))

# Create a DLC job with spot instance configuration (SpotWithPriceLimit).
create_job_resp = dlc_client.create_job(CreateJobRequest().from_map({
    'WorkspaceId': workspace_id,
    'DisplayName': 'sample-spot-job',
    'JobType': 'PyTorchJob',
    'JobSpecs': [
        {
            "Type": "Worker",
            "Image": "dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/pytorch-training:1.12-cpu-py39-ubuntu20.04",
            "PodCount": 1,
            "EcsSpec": 'ecs.g7.xlarge',
            "SpotSpec": {
                "SpotStrategy": "SpotWithPriceLimit",
                "SpotPriceLimit": 0.011,
            }
        },
    ],
    'UserVpc': {
        "VpcId": "vpc-0jlq8l7qech3m2ta2****",
        "SwitchId": "vsw-0jlc46eg4k3pivwpz8****",
        "SecurityGroupId": "sg-0jl4bd9wwh5auei9****",
    },
    "UserCommand": "echo 'Hello World' && ls -R /mnt/data/ && sleep 30 && echo 'DONE'",
}))

# Get and print the created job ID.
job_id = create_job_resp.body.job_id
print(f'jobId is {job_id}')

The key configurations are described as follows:

Parameter	Description
SpotStrategy	The bidding strategy. The bidding type takes effect only when this parameter is set toSpotWithPriceLimit.
SpotDiscountLimit	Uses spot discount as the bidding method. Note SpotDiscountLimit and SpotPriceLimit cannot be set at the same time. For Lingjun spot resources, only SpotDiscountLimit is supported as the bidding type.
SpotPriceLimit	Uses spot price as the bidding method.
UserVpc	Required when you submit a job by using Lingjun spot resources. Configure the VPC, vSwitch, and security groupID of the region where the job runs.