LLM intelligent routing workbench

Feature overview

You can enable the user management feature in the workbench as needed. When this feature is disabled, the workbench provides the following modules:

• Real-time monitoring: Displays near-real-time performance metrics such as throughput, latency, and TTFT, along with status code distribution and the operational status of the gateway, scheduler, and inference instances, helping you quickly detect anomalies and identify performance bottlenecks.

• Configuration center: Supports hot updates for core parameters, including traffic splitting proxy, scheduling policy, request throttling, and traffic mirroring. Changes take effect immediately without a service restart, making it ideal for dynamic tuning in production environments. For more information, see configuration center.

• Smart chat: Provides a visual interface for chat testing. You can send text or image requests directly to the LLM intelligent routing service and use HTTP logs for debugging, allowing you to quickly validate the behavior of the traffic splitting proxy.

Enabling user management creates a comprehensive user system, ideal for enterprises where multiple teams share an LLM service. This adds the following modules:

• Data overview: Displays token consumption and request volume distribution by user and department. You can filter the data by time range and user dimension to help admins identify high-usage users.

• User management: You can create, retrieve, update, and delete users, as well as enable or disable them and manage their API Keys. You can also perform these operations through an API. For more information, see user management API reference.

• Audit log: Records management operations such as logins and logouts, user modifications, API Key resets, and configuration changes. It logs the time, operator, and resource object for each action. You can filter logs by time, operation type, and operator for compliance audits and issue tracing.

• User center: Users can manage their profiles, view and reset their own API Key, and review their personal request statistics and token consumption records.

The modules available to each role are as follows:

Access the workbench

Prerequisites: You have deployed an LLM intelligent routing service, or an aggregated service that includes an LLM intelligent routing service.

Procedure:

1. Log on to the PAI console and go to the EAS inference services tab.

2. Find the target LLM intelligent routing service.

3. In the Invocation / Logs / Monitoring column of the service, click the web app icon.

4. In the pop-up window, find the WebUI workbench and open the admin workbench.

   After an admin creates a user on the user management page, the user can log on to the WebUI workbench through the User Login portal with their API Key.

Configuration center

The configuration center supports hot updates for the core configurations of LLM intelligent routing, which take effect immediately without a service restart. The following diagram shows the stages where each configuration is applied.

User management API reference

Authentication: All API calls must include the system admin API Key in the request header: Authorization: Bearer <admin_api_key>.

When making calls, replace https://gateway.example.com in the examples with your LLM intelligent routing endpoint, such as http://xxx.cn-shanghai.pai-eas.aliyuncs.com/api/predict/ai_assistant.ai_gw.

curl -X GET "https://gateway.example.com/api/users/check?employee_id=12345" \
-H "Authorization: Bearer <admin_api_key>"

curl -X POST "https://gateway.example.com/api/users" \
-H "Authorization: Bearer <admin_api_key>" \
-H "Content-Type: application/json" \
-d '{"name":"John Doe","email":"john.doe@company.com","employee_id":"12345","department":"Engineering"}'

# Retrieve all users
curl -X GET "https://gateway.example.com/api/users" \
-H "Authorization: Bearer <admin_api_key>"

# Retrieve a single user
curl -X GET "https://gateway.example.com/api/users/user_xxx" \
-H "Authorization: Bearer <admin_api_key>"

curl -X POST "https://gateway.example.com/api/users/user_xxx/regenerate-key" \
-H "Authorization: Bearer <admin_api_key>"

curl -X PUT "https://gateway.example.com/api/users/user_xxx" \
-H "Authorization: Bearer <admin_api_key>" \
-H "Content-Type: application/json" \
-d '{"department":"New Product","email":"john.doe@newcompany.com"}'

curl -X DELETE "https://gateway.example.com/api/users/user_xxx" \
-H "Authorization: Bearer <admin_api_key>"

curl -X POST "https://gateway.example.com/api/provision" \
  -H "Authorization: Bearer <admin_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "John Doe",
    "email": "john.doe@company.com",
    "employee_id": "12345",
    "department": "Engineering"
  }'

curl -X POST "https://gateway.example.com/api/provision/reset" \
-H "Authorization: Bearer <admin_api_key>" \
-H "Content-Type: application/json" \
-d '{"employee_id":"12345"}'

# 1. Create a user and get the provisioning link.
curl -X POST "https://gateway.example.com/api/provision" \
-H "Authorization: Bearer <admin_api_key>" \
 -H "Content-Type: application/json" \
 -d '{"name":"John Doe","email":"john.doe@company.com","employee_id":"12345","department":"Engineering"}'
# 2. Send the returned provision_url to the user (for example, by email or instant message).

# 3. The user opens the link, and the page automatically calls /api/provision/verify to retrieve the API Key.

# 1. Check if the user already exists.
curl -X GET "https://gateway.example.com/api/users/check?employee_id=12345" \
-H "Authorization: Bearer <admin_api_key>"

# 2. If the user does not exist, create the user.
curl -X POST "https://gateway.example.com/api/users" \
-H "Authorization: Bearer <admin_api_key>" \
-H "Content-Type: application/json" \
-d '{"name":"John Doe","employee_id":"12345","department":"Engineering"}'

# 3. Send the returned api_key to the user.

# Disable the user (recommended).
curl -X PUT "https://gateway.example.com/api/users/user_xxx" \
-H "Authorization: Bearer <admin_api_key>" \
-H "Content-Type: application/json" \
-d '{"enabled": false}'

# Alternatively, delete the user.
curl -X DELETE "https://gateway.example.com/api/users/user_xxx" \
-H "Authorization: Bearer <admin_api_key>"

# Reset the key by employee ID and send a new provisioning link.
curl -X POST "https://gateway.example.com/api/provision/reset" \
-H "Authorization: Bearer <admin_api_key>" \
-H "Content-Type: application/json" \
-d '{"employee_id":"12345"}'

# Send the returned provision_url to the user.

# Check for the user by employee ID.
curl -X GET "https://gateway.example.com/api/users/check?employee_id=12345" \
-H "Authorization: Bearer <admin_api_key>"

# Use the returned user_id to reset the API Key.
curl -X POST "https://gateway.example.com/api/users/user_xxx/regenerate-key" \
-H "Authorization: Bearer <admin_api_key>"

FAQ