Mount a dataset, OSS, NAS, or CPFS

Mount a dataset

1. Create a dataset

   Log on to the PAI console. In the navigation pane on the left, choose AI Asset Management > Dataset. On the page that appears, create a custom dataset or a public dataset. For more information, see Create and manage datasets.

2. Mount the dataset

   On the DSW instance configuration page, find the Dataset Mounting parameter. You can access this page when creating an instance or by clicking Change Settings on an existing one. Click Custom, then select the created dataset and enter a Mount Path.

Mount a storage path directly

This section uses mounting an OSS storage path as an example.

1. Create an OSS bucket

   Activate OSS and create a bucket.

   Important

   The bucket's region must be the same as the PAI region. A bucket's region cannot be changed after the bucket is created.

2. Mount the OSS path

   On the DSW instance configuration page (opened when creating an instance or by clicking Change Settings for an existing instance), find the Mount storage parameter. Click OSS, select the created OSS bucket path, and enter a Mount Path. The Advanced Configurations field is empty by default. You can configure it as needed. For more information, see Advanced mount configuration.

Prerequisites

1. Install the PAI Python SDK. In the DSW instance Terminal, run the following command. Python 3.8 or later is required.

   python -m pip install pai>=0.4.11

2. Configure access credentials for the SDK to access PAI.

   • Method 1: Configure the DSW instance to use the default PAI role or a custom RAM role. Open the instance configuration page, click Show More at the bottom, and select an instance RAM role. For more information, see Configure a RAM role for a DSW instance.

   • Method 2: Manually configure by using the command-line tool provided by the PAI Python SDK. In the Terminal, run the following command to configure access parameters. For an example, see Initialization.

     python -m pai.toolkit.config

Examples

• Mount to the default path

  The data is mounted to the default mount path within the instance. For official pre-built instance images, the default path is /mnt/dynamic/.

  from pai.dsw import mount
  # Mount an OSS path.
  mount_point = mount("oss://<YourBucketName>/Path/Data/Directory/") 
  # Mount a dataset. The input parameter is the dataset ID.
  # mount_point = mount("d-m7rsmu350********")

• Mount to a specified path

  Dynamic mounting requires mounting data to a specific path (or a subdirectory) inside the container. You can get the dynamic mount path by using the API provided by the SDK.

  from pai.dsw import mount, default_dynamic_mount_path
  # Get the default mount path of the instance.
  default_path = default_dynamic_mount_path()
  mount_point = mount("oss://<YourBucketName>/Path/Data/Directory"
                , mount_point=default_path + "tmp/output/model")

• Dynamically mount NAS

  from pai.dsw import mount, default_dynamic_mount_path
  # Get the default mount path of the instance.
  default_path = default_dynamic_mount_path()
  # Mount NAS. The NAS endpoint and the instance must be in the same VPC. Replace <region> with the region ID, such as cn-hangzhou.
  mount("nas://06ba748***-xxx.<region>.nas.aliyuncs.com/", default_path+"mynas3/")

• View all mount configurations in the instance

  from pai.dsw import list_dataset_configs
  print(list_dataset_configs())

• Unmount data

  from pai.dsw import mount, unmount
  mount_point = mount("oss://<YourBucketName>/Path/Data/Directory/")
  # The input parameter is the mount path, which is the MountPath value returned by list_dataset_configs.
  # After running the unmount command, it takes a few seconds for the change to take effect.
  unmount(mount_point)

Q: Mounted OSS files not visible in JupyterLab

This is because the DSW file browser shows the instance's working directory by default, which is typically /mnt/workspace. The mount path you specified for OSS (for example, /mnt/data) is not under the default working directory, so it does not appear in the file browser.

Solutions:

• Access through code: Your files have actually been mounted successfully. In your code, you need to use the full mount path to access them, such as open('/mnt/data/my_file.csv').

• Change the mount point: To see the files easily in the UI, you can set the mount path to a subdirectory under the working directory when you configure the mount, such as /mnt/workspace/my_oss_data. After the mount is complete, you can see your OSS files in the my_oss_data folder in the file browser.

• Access through the Terminal: In the DSW Terminal, you can use the cd /mnt/data command to navigate to the mount directory, and then use commands like ls to view and manage files.

Q: How to resolve OSS mount connection errors?

This error indicates the mount connection between the DSW instance and OSS is lost. Possible causes and solutions are as follows:

1. RAM role permission issue: Check if the RAM role configured for your DSW instance has been granted permission to access OSS (for example, AliyunPAIDLCAccessingOSSRole). Insufficient permissions are a common cause of failure to read from OSS.

2. Insufficient resources for the mount service: During high-intensity random read/write or large-scale small file operations, the ossfs or JindoFuse process responsible for the mount may crash due to an out-of-memory (OOM) error. In the Advanced Configurations of the mount settings, you can disable the metadata cache or increase the memory configuration. For more information, see JindoFuse.

3. Restore the connection:

   • For startup mounting, the easiest way to restore the connection is to restart the DSW instance. The system will automatically re-establish the mount.

   • You can also run a dynamic mounting command using the PAI SDK to remount the path without restarting the instance.

Q: What data types are supported for mounting?

DSW supports using cloud storage services like OSS, NAS, and CPFS by creating a dataset or directly mounting a storage path. 

• Alibaba Cloud Drive is not supported: DSW does not currently support directly mounting a personal Alibaba Cloud Drive. It is recommended to store data that needs processing in OSS.

• Mounting MaxCompute tables is not supported: Table data from MaxCompute (formerly ODPS) cannot be "mounted" to a DSW directory like a file system. You can read and write this data in your DSW code by using the SDK or API provided by PAI. For more information, see Use PyODPS to read and write MaxCompute tables.

Q: How to ensure data persistence and migration?

The system disk of a DSW instance provides temporary storage. For a public resource group, data is cleared if the instance is stopped for more than 15 days. For a dedicated resource group, data on the system disk is cleared as soon as the instance is stopped or deleted.

To store data and code persistently or to migrate them between instances, you must use external storage. 

• Persistence solution: Save all your important data, code, and models to a mounted OSS or NAS path. This way, even if the DSW instance is deleted, all your assets remain safely stored in your own OSS or NAS.

• Migration solution: When you need to migrate data from one DSW instance to another, simply mount the same OSS or NAS path to the new instance. This is the most convenient data migration method.

Q: Exporting data from a failed DSW instance

If a DSW instance cannot start due to insufficient resources or other reasons, the data on its system disk is not immediately lost. You can try the following methods to export the data:

1. Change the instance specification and restart: On the instance list page, click "Change Settings", select another available instance specification, and then restart the instance. Changing specifications does not cause data loss on the system disk. Once the instance starts successfully, immediately back up your data to OSS or NAS.

2. Switch the region or availability zone: If the current region has insufficient resources and your data is already stored in external storage, you can try creating a new instance and mounting the same OSS/NAS storage to recover the data.

3. Contact technical support: If the methods above do not resolve the issue, submit a ticket to technical support for assistance.