Quickly deploy an inference service based on the DeepSeek model

Step 1: Set up the network environment

1. Create an ECS instance on the Elastic Compute Service (ECS) instances page, using the ecs.g8i.48xlarge instance type.

   

2. Create an Elastic Network Interface (ENI) on the ENIs page. The ENI must be in the same virtual private cloud (VPC) as the ECS instance. Aggregating bandwidth from multiple ENIs lets you exceed the single-ENI bandwidth cap when accessing OSS within the VPC.

3. Attach the ENI to the ECS instance.

4. Note the IP addresses of the attached ENIs — you will need them when configuring OSS Connector.

   

Step 2: Install the inference framework

This tutorial uses llama.cpp, which supports efficient loading and inference of GGUF-format models.

To build from source:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release

Step 3: Install OSS Connector

1. Download and install the package.

      wget https://gosspublic.alicdn.com/oss-connector/oss-connector-lib-1.1.0rc7.x86_64.rpm
      yum install -y oss-connector-lib-1.1.0rc7.x86_64.rpm

2. Edit the configuration file at /etc/oss-connector/config.json.

      {
          "logLevel": 1,
          "logPath": "/var/log/oss-connector/connector.log",
          "auditPath": "/var/log/oss-connector/audit.log",
          "expireTimeSec": 120,
          "prefetch": {
              "vcpus": 32,
              "workers": 32
          }
      }

Step 4: Configure OSS Connector

Set the following environment variables so OSS Connector can authenticate and locate the model:

export OSS_ACCESS_KEY_ID=<your-access-key-id>
export OSS_ACCESS_KEY_SECRET=<your-access-key-secret>
export OSS_REGION=cn-beijing
export OSS_ENDPOINT=oss-cn-beijing-internal.aliyuncs.com
export OSS_PATH=oss://<BUCKET-NAME>/deepseek/DeepSeek-R2-Q8_0/
export MODEL_DIR=/tmp/model/DeepSeek-R2-Q8_0/
export HTTP_SOURCE_IP=172.xx.x.xxx,172.xx.x.xxx

Step 5: Start the inference service

LD_PRELOAD=/usr/local/lib/libossc_preload.so ENABLE_CONNECTOR=1 llama-server \
  -m /tmp/model/DeepSeek-R2-Q8_0/DeepSeek-R2.Q8_0-00001-of-00015.gguf \
  -t 48 -c 1024 --host 0.0.0.0 --port 9090 --numa isolate

LD_PRELOAD loads the OSS Connector library before the process starts, intercepting I/O calls transparently.

Step 6: Verify performance

After the service starts, internal network bandwidth on the ECS instance saturates within about one minute:

Both ENIs run at capacity, with combined bandwidth held continuously at 100 Gbit/s:

The following table compares end-to-end startup time across loading tools. Startup time is measured from process launch to when llama.cpp logs that the model is ready.

Overview

When hundreds of inference nodes start simultaneously and all pull the same model from OSS, concurrent back-to-origin traffic can exceed single-account bandwidth limits and slow startup across the cluster.

OSS Connector for AI/ML can work with a peer-to-peer (P2P) distribution system designed for this scenario. Nodes share model data with each other, reducing the load on OSS while preserving the connector's low-latency, high-throughput initial data access. The system supports on-demand distribution and can perform random and streaming reads of large files.

Key properties:

• Peak traffic reduction: Node-to-node sharing cuts direct OSS requests during concurrent startup, reducing instantaneous pressure on the central data source.

• Sustained performance: Raw data is still pulled from OSS via the connector, preserving low latency and high throughput for initial access.

• Horizontal scaling: Supports clusters of hundreds of nodes with startup times close to single-node performance.

• Reduced back-to-origin traffic: Local caching and P2P sharing minimize repeated data pulls from OSS, improving overall resource utilization.

Test environment

Test results

Key findings

The average startup time across 50 nodes was approximately 1.5x that of a single node (127s vs. 81s). Total back-to-origin data volume was about 1,262 GB — roughly twice the 664 GB model size — because each unique data block was fetched from OSS once and then distributed via P2P to the remaining nodes.

Even in this high-concurrency environment, startup times remained stable and did not increase proportionally with the number of nodes, demonstrating effective horizontal scaling.