Smart model selection

Step 1: Configure the classifier service for the smart routing service

Edit the JSON configuration of the LLM smart routing service and add the llm_gateway.model_selection section:

{
    "properties": {
        "llm_gateway.model_selection": {
            "classifier": {
                "api_key": "<classifier_service_credentials>",
                "endpoint": "http://<classifier_service_IP>:<port>",
                "model_name": "<classifier_service_model_name,default:model>"
            }
        }
    }
}

Step 2: Enable smart model selection

1. In the left-side navigation pane of InferX, click Configuration Center.

2. On the configuration center page, find the Smart model selection card and click the toggle on the right side of the card title to enable the feature.

Step 3: Configure basic parameters

After enabling the feature, the card automatically enters edit mode. Configure the following parameters and click Save.

Default tier

The fallback tier when the classifier service is unavailable or classification fails. L2 is recommended.

Candidate models per tier

Select candidate models for each of the three tiers (L3, L2, L1). The dropdown lists models reachable by the gateway; you can also enter model names manually. Within a tier, models are prioritized from left to right — if the first model is unavailable, the gateway automatically falls through to the next candidate.

If no candidate models are configured for a tier, that tier becomes unavailable.

Recommended models per tier:

After configuration, click Save. A change confirmation dialog appears showing a JSON diff of the before and after states. Click OK to submit.

Step 4: Call smart model selection

After configuration, set model to auto in your request to delegate model selection to the gateway. Example:

from openai import OpenAI

client = OpenAI(
    # Gateway endpoint
    base_url="http://xxxx/api/predict/group_llmgw_with_smart_model_selector/v1",
    api_key="<credentials>",
)

resp = client.chat.completions.create(
    model="auto",  # auto = delegate model selection to the gateway
    messages=[{"role": "user", "content": "Introduce yourself in one sentence"}],
)

print("Model selected by gateway:", resp.model)
print("Response:", resp.choices[0].message.content)

Overview — Smart model selection tier distribution

On the Overview page, the Smart model selection tier distribution card shows the basic distribution of model selection requests:

• Selection request count: Request count and token count distribution across tiers (L3/L2/L1).

• Selection coverage: The proportion of requests whose model was determined by smart model selection out of total requests within the selected time range. This metric uses the global scope and is not affected by user or department filters.

Real-time monitoring — Smart model selection health section

Displayed only when the feature is enabled. Includes a health status banner and six core metrics:

Health status banner

Six core metrics

Configuration conflicts

Smart model selection cannot be enabled simultaneously with Weight-based traffic distribution in the traffic proxy. If both are enabled, the configuration center shows a conflict warning and the backend rejects the save.

If both are forcefully enabled through backend configuration updates or other means, smart model selection takes no effect and traffic continues to follow the weight-based distribution logic.

Classifier service issues

Smart model selection depends on the classifier service. When the classifier service has issues, the health status is shown on the "Smart model selection" panel in real-time monitoring, with the fail-open ratio > 0 (displayed in red). Affected requests in the access logs carry the source=fail_open annotation.

• The gateway probes the classifier periodically (approximately every 10 seconds). Three consecutive failures trigger degradation; a single success restores normal operation.

• During degradation, each request that times out or fails during classification automatically falls through to the default tier. No requests are blocked or rejected. Normal model selection resumes automatically after recovery.