Accelerate model deployment with OSS Connector for AI/ML

High performance

OSS Connector for AI/ML significantly improves large model loading from OSS. With sufficient bandwidth, throughput can exceed 10 GB/s. Performance testing.

How it works

OSS Connector for AI/ML addresses performance bottlenecks when loading large models from OSS.

• FUSE-based mount solutions often cannot fully utilize OSS bandwidth, causing slow model loading. OSS Connector intercepts I/O requests from the inference framework and converts them directly into HTTP(s) requests to OSS.

• Using `LD_PRELOAD`, it prefetches and caches model data in memory — no code changes to your inference application required.

Deployment environment

• Operating system: Linux x86-64

• glibc: >=2.17

Install OSS Connector

1. Download the installation package.

   • oss-connector-lib-1.1.0rc7.x86_64.rpm: For Red Hat-based Linux distributions

     https://gosspublic.alicdn.com/oss-connector/oss-connector-lib-1.1.0rc7.x86_64.rpm

   • oss-connector-lib-1.1.0rc7.x86_64.deb: For Debian-based Linux distributions

     https://gosspublic.alicdn.com/oss-connector/oss-connector-lib-1.1.0rc7.x86_64.deb

2. Install OSS Connector.

   Install using the downloaded .rpm or .deb package. The library `libossc_preload.so` is installed to /usr/local/lib/.

   • Install oss-connector-lib-1.1.0rc7.x86_64.rpm

     yum install -y oss-connector-lib-1.1.0rc7.x86_64.rpm

   • Install oss-connector-lib-1.1.0rc7.x86_64.deb

     dpkg -i oss-connector-lib-1.1.0rc7.x86_64.deb

3. Verify that `/usr/local/lib/libossc_preload.so` exists and the version is correct.

   nm -D /usr/local/lib/libossc_preload.so | grep version

Configure OSS Connector

• Configuration file

  The configuration file controls log output, cache policy, and prefetch concurrency.

  Default configuration at /etc/oss-connector/config.json:

  {
      "logLevel": 1,
      "logPath": "/var/log/oss-connector/connector.log",
      "auditPath": "/var/log/oss-connector/audit.log",
      "expireTimeSec": 120,
      "prefetch": {
          "vcpus": 16,
          "workers": 16
      }
  }
  

  Parameter	Description
  logLevel	Controls the detail level of log output.
  logPath	Runtime log output path.
  auditPath	Audit log path for security and compliance tracking.
  expireTimeSec	Cache file release delay in seconds after all references are closed. Default: 120.
  prefetch.vcpus	Number of vCPUs for prefetching. Default: 16.
  prefetch.workers	Number of workers per vCPU for concurrency. Default: 16.

• Configure environment variables

  Environment variable KEY	Description
  OSS_ACCESS_KEY_ID	AccessKey pair of an Alibaba Cloud account or RAM user. When using a temporary access token, set these to the temporary credential's AccessKey pair. OSS Connector requires the `oss:ListObjects` permission on the target bucket directory. For anonymous-access buckets, leave `OSS_ACCESS_KEY_ID` and `OSS_ACCESS_KEY_SECRET` unset or empty.
  OSS_ACCESS_KEY_SECRET
  OSS_SESSION_TOKEN	Temporary access token. Required when using STS temporary credentials to access OSS. Set to an empty string when using permanent AccessKey credentials.
  OSS_ENDPOINT	OSS endpoint. Example: http://oss-cn-beijing-internal.aliyuncs.com. Defaults to HTTPS if no protocol is specified. Use HTTP on internal networks for better performance.
  OSS_REGION	OSS region ID. Example: cn-beijing. Authentication may fail if not specified.
  OSS_PATH	OSS model path. Format: `oss://bucketname/path/`. Example: oss://examplebucket/qwen/Qwen3-8B/.
  MODEL_DIR	Local model directory for the inference framework. Empty the directory before use. Temporary data downloaded during loading can be deleted afterward. Note The `MODEL_DIR` path must match the inference framework's model path (`--model` for vllm, `--model-path` for sglang). `MODEL_DIR` requires read/write permissions. Its directory structure mirrors `OSS_PATH`. Model files are prefetched and cached in memory during loading. The cache releases after 120 seconds by default (configurable via `expireTimeSec`). Use this directory exclusively for connector model loading. Do not create this directory on an existing OSS mount (such as ossfs).
  LD_PRELOAD	Path to the preloaded library: /usr/local/lib/libossc_preload.so. Set as a temporary environment variable. Example: LD_PRELOAD=/usr/local/lib/libossc_preload.so ENABLE_CONNECTOR=1 ./myapp
  ENABLE_CONNECTOR	OSS Connector process role. Set as a temporary environment variable. `ENABLE_CONNECTOR=1`: Primary connector role. `ENABLE_CONNECTOR=2`: Secondary connector role. Each instance allows only one primary connector process. Assign the primary role to the main process (entrypoint). All other connector processes must use the secondary role. the ray+vllm example for multi-node startup.

Start the model service

LD_PRELOAD=/usr/local/lib/libossc_preload.so \
ENABLE_CONNECTOR=1 OSS_ACCESS_KEY_ID=${OSS_ACCESS_KEY_ID} \
OSS_ACCESS_KEY_SECRET=${OSS_ACCESS_KEY_SECRET} \ OSS_ENDPOINT=${OSS_ENDPOINT} \
OSS_REGION=${OSS_REGION} \
OSS_PATH=${OSS_PATH} \
MODEL_DIR=/tmp/model \
python3 -m vllm.entrypoints.openai.api_server --model /tmp/model --trust-remote-code --tensor-parallel-size 1 --disable-custom-all-reduce

LD_PRELOAD=/usr/local/lib/libossc_preload.so \
ENABLE_CONNECTOR=1 OSS_ACCESS_KEY_ID=${OSS_ACCESS_KEY_ID} \
OSS_ACCESS_KEY_SECRET=${OSS_ACCESS_KEY_SECRET} \ OSS_ENDPOINT=${OSS_ENDPOINT} \
OSS_REGION=${OSS_REGION} \
OSS_PATH=${OSS_PATH} \
MODEL_DIR=/tmp/model \
python3 -m sglang.launch_server --model-path /tmp/model --port 8000 

export OSS_ACCESS_KEY_ID=${OSS_ACCESS_KEY_ID}
export OSS_ACCESS_KEY_SECRET=${OSS_ACCESS_KEY_SECRET}
export OSS_ENDPOINT=${OSS_ENDPOINT}
export OSS_REGION=${OSS_REGION}
export OSS_PATH=oss://examplebucket/
export MODEL_DIR=/tmp/models

LD_PRELOAD=/usr/local/lib/libossc_preload.so ENABLE_CONNECTOR=1 ray start --head --dashboard-host 0.0.0.0 --block

LD_PRELOAD=/usr/local/lib/libossc_preload.so ENABLE_CONNECTOR=1 ray start --address='172.24.176.137:6379' --block     // 172.24.176.137 is the pod IP. Change this to the IP address of the head pod. The command to join the cluster is provided in the output after you run `ray start` on Pod A.

LD_PRELOAD=/usr/local/lib/libossc_preload.so ENABLE_CONNECTOR=2 python3 -m vllm.entrypoints.openai.api_server --model ${MODEL_DIR}/qwen/Qwen2___5-72B/ --trust-remote-code --served-model-name ds --max-model-len 2048 --gpu-memory-utilization 0.98 --tensor-parallel-size 32

LD_PRELOAD=/usr/local/lib/libossc_preload.so \
ENABLE_CONNECTOR=1 OSS_ACCESS_KEY_ID=${OSS_ACCESS_KEY_ID} \
OSS_ACCESS_KEY_SECRET=${OSS_ACCESS_KEY_SECRET} \ OSS_ENDPOINT=${OSS_ENDPOINT} \
OSS_REGION=${OSS_REGION} \
OSS_PATH=${OSS_PATH} \
MODEL_DIR=/tmp/model \
python3 -m sglang.launch_server --model-path /tmp/model --port 8000 --dist-init-addr 192.168.1.1:20000 --nnodes 2 --node-rank 0 

LD_PRELOAD=/usr/local/lib/libossc_preload.so \
ENABLE_CONNECTOR=1 OSS_ACCESS_KEY_ID=${OSS_ACCESS_KEY_ID} \
OSS_ACCESS_KEY_SECRET=${OSS_ACCESS_KEY_SECRET} \ OSS_ENDPOINT=${OSS_ENDPOINT} \
OSS_REGION=${OSS_REGION} \
OSS_PATH=${OSS_PATH} \
MODEL_DIR=/tmp/model \
python3 -m sglang.launch_server --model-path /tmp/model --port 8000 --dist-init-addr 192.168.1.1:20000 --nnodes 2 --node-rank 1 

apiVersion: v1
kind: ConfigMap
metadata:
  name: connector-config
data:
  config.json: |
    {
        "logLevel": 1,
        "logPath": "/var/log/oss-connector/connector.log",
        "auditPath": "/var/log/oss-connector/audit.log",
        "expireTimeSec": 120,
        "prefetch": {
            "vcpus": 16,
            "workers": 16
        }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-connector-deployment
spec:
  selector:
    matchLabels:
      app: model-connector
  template:
    metadata:
      labels:
        app: model-connector
    spec:
      imagePullSecrets:
        - name: acr-credential-beijing
      hostNetwork: true
      containers:
      - name: container-name
        image: {IMAGE_ADDRESS}
        imagePullPolicy: Always
        resources:
          requests:
            cpu: "24"
            memory: "700Gi"
          limits:
            cpu: "128"
            memory: "900Gi"
        command: 
          - bash
          - -c
          - ENABLE_CONNECTOR=1 python3 -m vllm.entrypoints.openai.api_server --model /var/model --trust-remote-code --tensor-parallel-size 1 --disable-custom-all-reduce
        env:
        - name: LD_PRELOAD
          value: "/usr/local/lib/libossc_preload.so"
        - name: OSS_ENDPOINT
          value: "oss-cn-beijing-internal.aliyuncs.com"
        - name: OSS_REGION
          value: "cn-beijing"
        - name: OSS_PATH
          value: "oss://examplebucket/qwen/Qwen1.5-7B-Chat/"
        - name: MODEL_DIR
          value: "/var/model/"
        - name: OSS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: oss-access-key-connector
              key: key
        - name: OSS_ACCESS_KEY_SECRET
          valueFrom:
            secretKeyRef:
              name: oss-access-key-connector
              key: secret
        volumeMounts:
          - name: connector-config
            mountPath:  /etc/oss-connector/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: connector-config
        configMap:
          name: connector-config

Performance testing