Configure OSS accelerator with ossutil, ossfs, and SDKs-Object Storage Service(OSS)-阿里云帮助中心

Solutions

Use these tools and SDKs to access data with millisecond latency and high throughput:

ossutil: Add the accelerated endpoint to the ossutil configuration file.
ossfs: Specify the accelerated endpoint in the mount command.
OSS Connector for AI/ML: Set endpoint to the accelerated endpoint when building a dataset.
OSS SDKs: Set endpoint to the accelerated endpoint when initializing OSSClient. This example uses OSS SDK for Java.

Important

Use the accelerated endpoint as your OSS endpoint:

In ossutil, set endpoint to the accelerated endpoint.
In ossfs, set -ourl to the accelerated endpoint.

Prerequisites

You have created an accelerator. Create an OSS accelerator.

Solution 1: Use OSS accelerator with ossutil

Use OSS accelerator with ossutil

This example uses ossutil on 64-bit Linux x86. For more information about how to install ossutil on other operating systems, see Install ossutil.

Install ossutil.

Download the ossutil package.

curl -o ossutil-2.0.3-beta.09261200-linux-amd64.zip https://gosspublic.alicdn.com/ossutil/v2-beta/2.0.3-beta.09261200/ossutil-2.0.3-beta.09261200-linux-amd64.zip

Unzip the package:

unzip ossutil-2.0.3-beta.09261200-linux-amd64.zip

Make the binary executable:
```
chmod 755 ossutil
```

Add ossutil to the system PATH:

sudo mv ossutil /usr/local/bin/ && sudo ln -s /usr/local/bin/ossutil /usr/bin/ossutil

Verify the installation.
```
ossutil version
```

Configure ossutil with the accelerated endpoint.

Create an ossutil configuration file.
```
touch ossutilconfig
```

Set the parameters. For more information, see Configure ossutil.

#[profile dev]                  The name of the section. You can specify the name of the section. 
#accessKeyId, accessKeySecret  The AccessKey ID and AccessKey secret of the RAM user.  
#region                        The region in which the bucket is located. 
# buckets=dev-bucket            Use the parameters in the dev-bucket section. 
#[buckets dev-bucket]          The name of the section. You can specify the name of the section. 
#bucketname                    The name of the bucket.  
#endpoint                      The accelerated endpoint of the accelerator.  

[profile dev]
accessKeyId=LTA*********************ICT
accessKeySecret=V0qK**********************OOKp
region=cn-hangzhou
buckets=dev-bucket

[buckets dev-bucket]
bucket-name =
#Specify the endpoint of the accelerator. 
  endpoint=cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com

Use the accelerator.

This example downloads an object to the test directory. For more information about ossutil commands, see ossutil commands.

cp downloads objects. -r enables recursive operations on the bucket-name bucket. -c specifies the configuration file path. --profile selects the dev section.
```
ossutil cp -r oss://bucket-name/ ./test/ -c ./ossutilconfig --profile dev
```

Solution 2: Use OSS accelerator with ossfs

Use OSS accelerator with ossfs

This example uses ossfs on Ubuntu 16.04 x86. For other operating systems, Install ossfs.

Install ossfs.

sudo wget https://gosspublic.alicdn.com/ossfs/ossfs_1.91.3_ubuntu16.04_amd64.deb
sudo apt-get update
sudo apt-get install gdebi-core
sudo gdebi ossfs_1.91.3_ubuntu16.04_amd64.deb

Add the mime.types file.

Ensure content types match file extensions:
```
sudo apt-get install mime-support
```
Configure ossfs and mount the bucket with the accelerated endpoint. For more information about how to perform mount operations by using ossfs, see Mount a bucket.
1. Store the AccessKey ID and AccessKey secret in /etc/passwd-ossfs.
```
echo bucket-test:LTAI****************2CT:V0************************OKp > /etc/passwd-ossfs
```
2. Set the permissions of the configuration file to 640.
```
chmod 640 /etc/passwd-ossfs 
```
3. Create a mount directory.
```
mkdir bucket-test
```
4. Mount bucket-test to /tmp/bucket-test/ with the accelerated endpoint.
```
ossfs bucket-test /tmp/bucket-test/ -ourl=cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com
```
Use the accelerator.

Reads and writes on the mounted bucket are now accelerated.

Solution 3: Use OSS accelerator with OSS Connector for AI/ML

Use OSS accelerator with OSS Connector for AI/ML

This example uses Python 3.12. Adjust the version number for other Python versions.

Prerequisites:

Operating system: 64-bit x86 Linux
glibc: 2.17 or later
Python: 3.8 to 3.12
PyTorch: 2.0 or later
To use the OSS checkpoint feature, the Linux kernel must support userfaultfd.

Note
On Ubuntu, run sudo grep CONFIG_USERFAULTFD /boot/config-$(uname -r) to check userfaultfd support. CONFIG_USERFAULTFD=y means supported. CONFIG_USERFAULTFD=n means unsupported, and the checkpoint feature is unavailable.

Install OSS Connector for AI/ML.
1. On your Linux operating system or in a container built from a Linux image, run the pip3.12 install osstorchconnector command to install the OSS Connector for AI/ML.
```
pip3.12 install osstorchconnector
```
2. Run pip3.12 show osstorchconnector to verify the installation.
```
pip3.12 show osstorchconnector
```
  A successful installation displays the version information for osstorchconnector. The output is similar to the following:
```
Name: osstorchconnector
Version: 1.0.0rc1
Summary: OSS connector for AI/ML
Home-page: 
Author: 
Author-email: 
License: 
Location: /usr/local/lib/python3.12/dist-packages
Requires: torch
Required-by:
```
Configure OSS Connector for AI/ML.
1. You can create a configuration file for access credentials.
```
mkdir -p /root/.alibabacloud && touch /root/.alibabacloud/credentials
```
2. You can add the access credential configuration and save the file.
  Replace <Access-key-id> and <Access-key-secret> in the example with the AccessKey ID and AccessKey secret of a Resource Access Management (RAM) user. For more information about how to create an AccessKey ID and an AccessKey secret, see Create an AccessKey. For more information about configuration items and how to use temporary access credentials, see Configure access credentials.
```
{
  "AccessKeyId": "LTAI************************",
  "AccessKeySecret": "At32************************"
}
```
3. You can create the OSS Connector configuration file.
```
mkdir -p /etc/oss-connector/ && touch /etc/oss-connector/config.json
```
4. You can add the OSS Connector configuration and save the file. For more information about the configuration items, see Configure OSS Connector.
  In most cases, you can use the following default configurations.
```
{
    "logLevel": 1,
    "logPath": "/var/log/oss-connector/connector.log",
    "auditPath": "/var/log/oss-connector/audit.log",
    "datasetConfig": {
        "prefetchConcurrency": 24,
        "prefetchWorker": 2
    },
    "checkpointConfig": {
        "prefetchConcurrency": 24,
        "prefetchWorker": 4,
        "uploadConcurrency": 64
    }
}
```

Use the accelerator.

Build a dataset with OssMapDataset.from_prefix and set Endpoint to the accelerated endpoint.

from osstorchconnector import OssMapDataset

# Replace ENDPOINT with the accelerated endpoint of the accelerator. 
ENDPOINT = "cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com"
# Specify the path of the OSS Connector for AI/ML configuration file. 
CONFIG_PATH = "/etc/oss-connector/config.json"
# Specify the path of the access credential configuration file. 
CRED_PATH = "/root/.alibabacloud/credentials"
# Specify the path of the directory of the bucket. 
OSS_URI = "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/"

# Use the from_prefix method of OssMapDataset to build a dataset.
map_dataset = OssMapDataset.from_prefix(oss_uri=OSS_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)

# Access objects in the created dataset at random.
item = map_dataset[0]
print(item.key)
content = item.read()
print(item.size)
print(len(content))

# Traverse the objects in the dataset.
for item in map_dataset:
    print(item.key)
    print(item.size)
    content = item.read()
    print(len(content))

Solution 4: Use OSS accelerator with OSS SDKs

Use OSS accelerator with OSS SDKs

This example uses OSS SDK for Java. For other SDKs, set the accelerated endpoint when initializing OSSClient.

Prerequisites:
- Java 1.7.0 or later is installed.
- Run java -version to verify.

Add the OSS SDK for Java dependency to your Maven project (Installation).

Add these dependencies to pom.xml (OSS SDK for Java 3.17.4):

<dependency>
    <groupId>com.aliyun.oss</groupId>
    <artifactId>aliyun-sdk-oss</artifactId>
    <version>3.17.4</version>
</dependency>

If you use Java 9 or later, add these JAXB dependencies:

<dependency>
    <groupId>javax.xml.bind</groupId>
    <artifactId>jaxb-api</artifactId>
    <version>2.3.1</version>
</dependency>
<dependency>
    <groupId>javax.activation</groupId>
    <artifactId>activation</artifactId>
    <version>1.1.1</version>
</dependency>
<!-- no more than 2.3.3-->
<dependency>
    <groupId>org.glassfish.jaxb</groupId>
    <artifactId>jaxb-runtime</artifactId>
    <version>2.3.3</version>
</dependency>

Use the accelerator.

Upload an object by setting endpoint to the accelerated endpoint.

import com.aliyun.oss.ClientBuilderConfiguration;
import com.aliyun.oss.OSS;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.common.auth.CredentialsProviderFactory;
import com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider;
import com.aliyun.oss.common.comm.SignVersion;

import java.io.ByteArrayInputStream;

public class OssCliectTest {
    public static void main(String[] args) throws Exception {
        // Specify the accelerated endpoint of the accelerator. 
        String endpoint = "cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com";
        // Specify the ID of the region that maps to the endpoint. Example: cn-hangzhou.
        String region = "cn-hangzhou";

        // Specify access credentials.
        String alibaba_cloud_access_key_id = "LT********************CT";
        String alibaba_cloud_access_key_secret = "V0q***************************OKp";

        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "BucketName";
        // Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
        String objectName = "Test.txt";

        // Create an OSSClient instance.
        // Call the shutdown method to release resources when the OSSClient is no longer in use. 
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        // Explicitly declare the use of the V4 signature algorithm.
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
                .endpoint(endpoint)
                .credentialsProvider(credentialsProvider)
                .clientConfiguration(clientBuilderConfiguration)
                .region(region)
                .build();
        
        String content = "##################";
        // Upload the configurations of the content variable to the Test.txt file in the bucket.
        ossClient.putObject(bucketName,objectName,new ByteArrayInputStream(content.getBytes()));
        
        // Shut down the OSSClient instance. 
        ossClient.shutdown();
    }
}