Develop custom models

Development machine overview

The development machine is a cloud-based development environment provided by the platform. It runs on preconfigured container images and includes commonly used AI frameworks and tools. Key features of the development machine include:

• Image-driven: Select different official images to get the corresponding development environment (such as GR00T, openpi, etc.).

• Shared resources: All development machines share the working directory, datasets, models, and simulation assets, allowing seamless access to the same resources across scenarios.

• Persistent storage: The working directory /mnt/workspace uses persistent storage. Data is retained after a restart.

Create a development environment

1. In the left-side navigation pane, expand Models and click Model Development.

2. Click Create Development Environment.

3. Configure the following parameters in the right-side panel:

   Parameter	Required	Description
   Name	Required	The name of the development environment.
   Development Image	Required	Select an official image or a custom image from the drop-down list.
   Resource Specification	Required	Select a CPU or GPU specification.
   Port Mapping	Optional	The ports to expose.
   Environment Variables	Optional	Environment variables in Key-Value format.
   SSH Public Key	Optional	The public key for SSH logon.

4. Click OK. After the environment is created, the status changes to Running.

Log on to the development machine

After the development machine is created, log on through SSH:

1. In the development environment list, copy the SSH Address of the target development machine.

2. Use an SSH client to log on:

   ssh -i <private key path> root@<SSH address>

Development machine directory structure

After logging on, the development machine provides the following mount paths:

Environment setup

The development machine has base dependencies installed by default. To install additional Python packages:

• With public network access (NAT gateway configured): Use pip install directly.

• Without public network access: Use the Alibaba Cloud PyPI mirror to install:

  pip install <package> -i https://mirrors.aliyun.com/pypi/simple/

Upload data

Upload local data to the development machine:

• Through SCP:

  scp -i <private key path> -r /path/to/local/data root@<SSH address>:/mnt/workspace/

• Through ossutil: Upload data to OSS first, and then download it on the development machine by using ossutil. For more information, see Upload data by using AnalyticDB lake storage OSS.

Training visualization

During training, you can use TensorBoard to view training metrics. Start TensorBoard on the development machine and access it through port mapping in your browser.

Build images

After development is complete, you can package the development machine environment as a custom image for subsequent training tasks and deployment.

1. Make sure that you have configured an ACR image repository in the AnalyticDB console. For more information, see Create and log on to the Embodied Intelligence Platform.

2. Build and push the image to ACR on the development machine.

3. After the push is complete, the custom image appears on the image management page of the platform.

Register models

Register trained models from the development machine working directory to the platform:

• Model files are stored in the /mnt/workspace or /model directory.

• After you register a model on the model management page, you can use it for subsequent training tasks or deploy it as an inference service directly.

Submit custom training tasks

Submit distributed training tasks by using custom models and images:

1. In the left-side navigation pane, click Model Training.

2. Click Start Training.

3. Switch Model Source to Custom Model.

4. Configure the following parameters:

   Parameter	Required	Description
   Model Path Query	Optional	If a model is registered on the platform, select the target model to view its storage path. You must manually configure the path in the startup command or environment variables.
   Dataset Path Query	Optional	Select a target dataset and the platform displays its storage path. You must manually configure the path in the startup command or environment variables.
   Image	Required	Select the image built during the model development phase.
   Startup Command	Required	The startup command for the model training task.
   Environment Variables	Optional	Custom environment variables.
   TensorBoard Log Path	Optional	Specify the TensorBoard log storage path. The platform automatically reads and displays training metrics.
   Training Resources	Required	The platform submits training jobs through Ray Job. If your training code is not based on Ray, only single-machine training is supported and you only need to configure the Head resource. If it is based on Ray, you can configure Ray Head and Worker resources as needed.

   After configuration, you can click Save Template to save the configuration as a template for quick reuse in future training tasks.

5. Click OK.

Monitor training

The monitoring method for custom training tasks is the same as that for built-in model training:

• Logs and resource usage: View through the Ray Dashboard link in the training task list. The Ray Dashboard is only available during training and is deleted after training ends. To retain logs long-term, we recommend writing log files to /mnt/workspace.

• Training metrics: View through More > View Metrics in the training task list. You must specify the TensorBoard log path when submitting the training task.

Download training results

After training is complete, you can download model weight files:

• Browser download: Suitable for files smaller than 200 MB. Click Download in the training task list.

• Export to OSS: Suitable for large files. Export the training results to OSS and then download them by using ossutil.

Related documentation

• Train and deploy built-in models

• Manage simulation environment and evaluation

• Enable public network access for development machines

• Create and log on to the Embodied Intelligence Platform