Use OSS

Mounting methods

Default configuration limitations

If you leave the Advanced Configurations parameters empty, the default configuration is applied. The default configuration has the following limitations:

• To accelerate OSS file reads, metadata (directory and file lists) is cached when OSS is mounted.

  In a distributed training, if multiple nodes attempt to create the same directory, the metadata cache can cause each node to attempt the creation. Only one node will succeed, and the others will report an error.

• By default, the OSS MultiPart API is used to create files. The object becomes visible in OSS only after the write operations are complete.

• Concurrent read and write operations on the same file are not supported.

• Random write operations on files are not supported.

Common JindoFuse configurations

You can also customize JindoFuse parameters in the advanced configuration based on your use case.

This topic provides JindoFuse configuration suggestions for some common scenarios. These settings may not provide optimal performance for all workloads. For more flexible configuration options, see the JindoFuse User Guide.

• Quick Read/write: ensures quick reads and writes. However, data inconsistency may occur during concurrent reads or writes. You can mount training data and models to the mount path of this mode. We recommend that you do not use the mount path of this mode as the working directory.

  {
    "fs.oss.download.thread.concurrency": "Twice the number of CPU cores",
    "fs.oss.upload.thread.concurrency": "Twice the number of CPU cores",
    "fs.jindo.args": "-oattr_timeout=3 -oentry_timeout=0 -onegative_timeout=0 -oauto_cache -ono_symlink"
  }
  

• Incremental Read/Write: ensures data consistency during incremental writing. If original data is overwritten, data inconsistency may occur. The reading speed is slightly slow. You can use this mode to save the model weight files for training data.

  {
    "fs.oss.upload.thread.concurrency": "Twice the number of CPU cores",
    "fs.jindo.args": "-oattr_timeout=3 -oentry_timeout=0 -onegative_timeout=0 -oauto_cache -ono_symlink"
  }
  

• Consistent Read/write: ensures data consistency during concurrent reads or writes and is suitable for scenarios that require high data consistency and do not require quick reads. You can use this mode to save the code of your projects.

  {
    "fs.jindo.args": "-oattr_timeout=0 -oentry_timeout=0 -onegative_timeout=0 -oauto_cache -ono_symlink"
  }
  

• Read-only: allows only reads. You can use this mode to mount public datasets.

  {
    "fs.oss.download.thread.concurrency": "Twice the number of CPU cores",
    "fs.jindo.args": "-oro -oattr_timeout=7200 -oentry_timeout=7200 -onegative_timeout=7200 -okernel_cache -ono_symlink"
  }

Other common configuration operations include:

Mounting methods

Common ossfs configurations

In Advanced Configuration, you can set advanced parameters using fs.ossfs.args. Separate multiple parameters with a comma (,). For more information about these advanced parameters, see ossfs 2.0. The following examples show several common use cases:

• Static data source during the job: For files that are not modified during the job, configure a long cache timeout to reduce metadata requests. A typical use case is reading a batch of existing files and generating a new set of files after processing.

  {
      "mountType":"ossfs",
      "fs.ossfs.args": "-oattr_timeout=7200" 
  }

• Fast read/write: Use a shorter metadata cache timeout to balance cache efficiency and data freshness.

  {
      "mountType":"ossfs",
      "fs.ossfs.args": "-oattr_timeout=3, -onegative_timeout=0"
  }

• Consistent read/write for distributed jobs: By default, ossfs updates file data based on the metadata cache. Use the following configuration to ensure a consistent view across multiple nodes.

  {   
      "mountType":"ossfs",
      "fs.ossfs.args": "-onegative_timeout=0, -oclose_to_open"
  }

• Out-of-memory (OOM) from too many open files in DLC or DSW: High job concurrency in DLC and DSW can open many files simultaneously, potentially causing Out-of-memory (OOM) issues. The following configuration can help reduce memory pressure.

  {
      "mountType":"ossfs",
      "fs.ossfs.args": "-oreaddirplus=false, -oinode_cache_eviction_threshold=300000"
  }

• Failure to write large files: -oupload_buffer_size sets the buffer size (in bytes) for multipart uploads. This parameter determines the maximum writable file size, calculated as upload_buffer_size * 10000.

  By default, ossfs 2.0 uses a part size of 8 MiB, which limits the maximum writable file size to 78.125 GiB. When you write a file that exceeds this limit, the operation fails. To increase the maximum supported file size, configure the -oupload_buffer_size option to increase the part size. For example, setting the part size to 32 MiB (33,554,432 bytes) supports a maximum file size of 312.5 GiB. Note that a larger -upload_buffer_size consumes more memory. You can control memory usage by configuring -total_mem_limit. For more information, see Mount options.

  {
      "mountType":"ossfs",
      "fs.ossfs.args": "-oupload_buffer_size=33554432"
  }
  

Limitations

• Official images: The OSS Connector for AI/ML is available only in DLC jobs and DSW instances that use an official image of PyTorch 2.0 or later.

• Custom images: Only PyTorch 2.0 or later is supported. For custom images that use a supported version, install the OSS Connector for AI/ML by running the following command:

  pip install -i http://yum.tbsite.net/aliyun-pypi/simple/ --extra-index-url http://yum.tbsite.net/pypi/simple/ --trusted-host=yum.tbsite.net osstorchconnector

• Python version: Only Python 3.8–3.12 is supported.

Prerequisites

1. Configure a credential file.

   You can configure credentials in one of the following ways:

   • Configure password-free access to OSS for your DLC job. For more information, see Configure a DLC RAM role. With this method, the DLC job obtains a temporary credential from STS to securely access OSS and other cloud resources, eliminating the need for explicit authentication information and reducing the risk of access key leakage.

   • Configure a credential file in your code project to manage authentication information. The following code provides a sample configuration:

     Note

     Storing access key information in plaintext poses a security risk. We recommend using a RAM role to automatically configure credentials within a DLC instance. For more information, see Configure a DLC RAM role.

     When you use the OSS Connector for AI/ML, you can specify the path to the credential file to automatically retrieve authentication information for signing OSS data requests.

     {
       "AccessKeyId": "<Access-key-id>",
       "AccessKeySecret": "<Access-key-secret>",
       "SecurityToken": "<Security-Token>",
       "Expiration": "2024-08-20T00:00:00Z"
     }

     The following table describes the fields.

     Parameter	Required	Description	Example
     AccessKeyId	Yes	The access key ID and access key secret of an Alibaba Cloud account or a RAM user. Note When you use a temporary credential from STS to access OSS, set these parameters to the temporary access key ID and access key secret.	NTS****
     AccessKeySecret	Yes		7NR2****
     SecurityToken	No	The security token from STS. This parameter is required only when you use a temporary credential from STS to access OSS.	STS.6MC2****
     Expiration	No	The credential's expiration time. If this field is empty, the credential does not expire. The OSS Connector for AI/ML re-reads the credential file when the credential expires.	2024-08-20T00:00:00Z

2. Configure a config.json file. The following code provides a sample configuration:

   This file is used to manage settings in your project, such as concurrency levels, prefetch parameters, and log file locations. When using the OSS Connector for AI/ML, specify the path to this config.json file. The connector then automatically applies your configured settings and writes logs related to OSS data requests to the specified log file.

   {
       "logLevel": 1,
       "logPath": "/var/log/oss-connector/connector.log",
       "auditPath": "/var/log/oss-connector/audit.log",
       "datasetConfig": {
           "prefetchConcurrency": 24,
           "prefetchWorker": 2
       },
       "checkpointConfig": {
           "prefetchConcurrency": 24,
           "prefetchWorker": 4,
           "uploadConcurrency": 64
       }
   }

   The following table describes the fields.

   Parameter		Required	Description	Example
   logLevel		Yes	The log level. The default value is 1 (INFO). Valid values: 0: DEBUG 1: INFO 2: WARN 3: ERROR	1
   logPath		Yes	The log path for the connector. The default path is /var/log/oss-connector/connector.log.	/var/log/oss-connector/connector.log
   auditPath		Yes	The audit log for Connector I/O records read and write requests with a latency greater than 100 milliseconds. The default path is /var/log/oss-connector/audit.log.	/var/log/oss-connector/audit.log
   DatasetConfig	prefetchConcurrency	Yes	The number of concurrent tasks for prefetching data from OSS when using a dataset. The default value is 24.	24
   	prefetchWorker	Yes	The number of vCPUs available for prefetching data from OSS when using a dataset. The default value is 2.	2
   checkpointConfig	prefetchConcurrency	Yes	The number of concurrent tasks for prefetching data from OSS during a checkpoint read. The default value is 24.	24
   	prefetchWorker	Yes	The number of vCPUs available for prefetching data from OSS during a checkpoint read. The default value is 4.	4
   	uploadConcurrency	Yes	The number of concurrent tasks for uploading data during a checkpoint write. The default value is 64.	64

Usage

The OSS Connector for AI/ML provides two dataset access interfaces, OssMapDataset and OssIterableDataset, which extend PyTorch's Dataset and IterableDataset interfaces, respectively. OssIterableDataset is optimized with prefetching for higher training efficiency. The data reading order of OssMapDataset is determined by the DataLoader and supports shuffle operations. You can choose a dataset access interface based on the following recommendations:

• If you have limited memory, a large dataset, and require only sequential reads with low parallelism, use OssIterableDataset.

• If you have sufficient memory, a smaller dataset, and require random access and parallel processing, use OssMapDataset.

The OSS Connector for AI/ML also provides the OssCheckpoint interface for loading and saving models. Currently, the OssCheckpoint feature is available only in general-purpose computing resource environments.

The following sections describe how to use these three interfaces.

checkpoint = OssCheckpoint(endpoint="{oss_endpoint}", cred_path=cred_path, config_path=config_path)

checkpoint_read_uri = "{checkpoint_path}"
checkpoint_write_uri = "{checkpoint_path}"
with checkpoint.reader(checkpoint_read_uri) as reader:
    state_dict = torch.load(reader)
    model.load_state_dict(state_dict)
with checkpoint.writer(checkpoint_write_uri) as writer:
    torch.save(model.state_dict(), writer)

Code example

The following code provides a full example of how to use the OSS Connector for AI/ML to access OSS data:

from osstorchconnector import OssMapDataset, OssCheckpoint
import torchvision.transforms as transforms
import accimage
import torchvision.models as models
import torch

# The default credential path after you configure a RAM role for a DLC job or DSW instance.
cred_path = "/mnt/.alibabacloud/credentials"  
config_path = "config.json"
checkpoint = OssCheckpoint(endpoint="{oss_endpoint}", cred_path=cred_path, config_path=config_path)
model = models.__dict__["resnet18"]()

epochs = 100  # Specify the number of epochs.
checkpoint_read_uri = "{checkpoint_path}"
checkpoint_write_uri = "{checkpoint_path}"
with checkpoint.reader(checkpoint_read_uri) as reader:
    state_dict = torch.load(reader)
    model.load_state_dict(state_dict)


def read_and_transform(data):
    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])
    transform = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        normalize,
    ])

    try:
        img = accimage.Image((data.read()))
        value = transform(img)
    except Exception as e:
        print("read failed", e)
        return None, 0
    return value, 0
dataset = OssMapDataset.from_prefix("{oss_data_folder_uri}", endpoint="{oss_endpoint}", transform=read_and_transform, cred_path=cred_path, config_path=config_path)
data_loader = torch.utils.data.DataLoader(
    dataset, batch_size="{batch_size}",num_workers="{num_workers"}, pin_memory=True)

for epoch in range(args.epochs):
    for step, (images, target) in enumerate(data_loader):
        # batch processing
        # model training
    # save model
    with checkpoint.writer(checkpoint_write_uri) as writer:
        torch.save(model.state_dict(), writer)

Key points in this example:

• Build a dataset with OssMapDataset directly from the given OSS URI, following the standard PyTorch DataLoader pattern.

• Use this dataset to create a standard PyTorch DataLoader, and run a normal training loop to process each batch, train the model, and save checkpoints.

• This enables on-demand loading, which eliminates the need to mount the dataset in the container environment or download it to local storage beforehand.

OSS Python SDK

Use the OSS Python SDK to read data from and write data to OSS:

1. Install the OSS Python SDK. For more information, see Installation (Python SDK V1).

2. Configure access credentials for the OSS Python SDK. For more information, see Configure access credentials (Python SDK V1).

3. Read and write data in OSS.

   # -*- coding: utf-8 -*-
   import oss2
   from oss2.credentials import EnvironmentVariableCredentialsProvider
   
   # Configure access credentials using the RAM user access key from environment variables.
   auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
   bucket = oss2.Bucket(auth, '<Endpoint>', '<your_bucket_name>')
   # Read a complete file.
   result = bucket.get_object('<your_file_path/your_file>')
   print(result.read())
   # Read data by range.
   result = bucket.get_object('<your_file_path/your_file>', byte_range=(0, 99))
   # Write data to OSS.
   bucket.put_object('<your_file_path/your_file>', '<your_object_content>')
   # Append data to an appendable file.
   result = bucket.append_object('<your_file_path/your_file>', 0, '<your_object_content>')
   result = bucket.append_object('<your_file_path/your_file>', result.next_position, '<your_object_content>')
   

   Modify the following parameters as needed:

   Parameter	Description
   <Endpoint>	The endpoint for your bucket's region. For example, for the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. For more information about how to obtain an endpoint, see Regions and endpoints.
   <your_bucket_name>	The name of your bucket.
   <your_file_path/your_file>	The path of the file to read or write. Specify the full path of the object but exclude the bucket name. For example, testfolder/exampleobject.txt.
   <your_object_content>	The content to write or append. Replace as needed.

OSS Python API

You can use the OSS Python API to conveniently store training data and models in OSS. Before you begin, ensure you have installed the OSS Python SDK and configured your access credentials. For more information, see Installation (Python SDK V1) and Configure access credentials (Python SDK V1).

• Load training data

  You can store your data in an OSS bucket, with the data paths and corresponding labels in an index file in the same bucket. By creating a custom Dataset, you can use the DataLoader API in PyTorch to read data in parallel across multiple processes. The following code is an example.

  import io
  import oss2
  from oss2.credentials import EnvironmentVariableCredentialsProvider
  import PIL
  import torch
  
  class OSSDataset(torch.utils.data.dataset.Dataset):
      def __init__(self, endpoint, bucket, auth, index_file):
          self._bucket = oss2.Bucket(auth, endpoint, bucket)
          self._indices = self._bucket.get_object(index_file).read().split(',')
  
      def __len__(self):
          return len(self._indices)
  
      def __getitem__(self, index):
          img_path, label = self._indices(index).strip().split(':')
          img_str = self._bucket.get_object(img_path)
          img_buf = io.BytesIO()
          img_buf.write(img_str.read())
          img_buf.seek(0)
          img = Image.open(img_buf).convert('RGB')
          img_buf.close()
          return img, label
  
  
  # Obtain access credentials from environment variables. Before you run this code sample,
  # make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
  auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
  dataset = OSSDataset(endpoint, bucket, auth, index_file)
  data_loader = torch.utils.data.DataLoader(
      dataset,
      batch_size=batch_size,
      num_workers=num_loaders,
      pin_memory=True)
  

  This table describes the key parameters.

  Parameter	Description
  endpoint	The endpoint for your bucket's region. For example, for the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. For more information about how to obtain an endpoint, see Regions and endpoints.
  bucket	The name of your bucket.
  index_file	The path of the index file. Note In the example, the index file uses a comma (,) to separate each sample, and a colon (:) to separate the sample path from the label.

• Save or load models

  Use the OSS Python API to save or load PyTorch models. For more information about how to save and load models in PyTorch, see PyTorch. The following code provides examples:

  • Save a model

    from io import BytesIO
    import torch
    import oss2
    from oss2.credentials import EnvironmentVariableCredentialsProvider
    
    auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
    # bucket_name
    bucket_name = "<your_bucket_name>"
    bucket = oss2.Bucket(auth, endpoint, bucket_name)
    buffer = BytesIO()
    torch.save(model.state_dict(), buffer)
    bucket.put_object("<your_model_path>", buffer.getvalue())
    

    Where:

    • endpoint is the endpoint for your bucket's region. For example, for the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.

    • <your_bucket_name> is the name of the OSS bucket, without the oss:// prefix.

    • <your_model_path> is the model path. Replace it as needed.

  • Load a model

    from io import BytesIO
    import torch
    import oss2
    from oss2.credentials import EnvironmentVariableCredentialsProvider
    
    auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
    bucket_name = "<your_bucket_name>"
    bucket = oss2.Bucket(auth, endpoint, bucket_name)
    buffer = BytesIO(bucket.get_object("<your_model_path>").read())
    model.load_state_dict(torch.load(buffer))

    Where:

    • endpoint is the endpoint for your bucket's region. For example, for the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.

    • <your_bucket_name> is the name of the OSS bucket, without the oss:// prefix.

    • <your_model_path> is the model path. Replace it as needed.