Appendix: Configure dedicated gateway timeouts

Recommended configuration

The fully managed gateway uses the following fixed idle connection timeout values. These values can't be modified.

• Gateway as server (facing clients): Fixed at 600 seconds. If a client-to-gateway connection is idle for longer than this period, the gateway closes it.

• Gateway as client (facing backend inference services): Fixed at 30 seconds. If a gateway-to-backend connection is idle for longer than this period, the gateway closes it.

The following diagram illustrates the connection flow:

To ensure that the client manages and closes connections, configure each link as follows:

Leave a safety margin between upstream and downstream values. Avoid setting related timeouts to the same value.

Default values by access method

The following table lists the default idle connection timeout for each access method, along with recommended client and backend inference service configurations.

Model Gallery template defaults and NLB considerations

The backend inference service server idle timeout must be greater than the gateway client idle timeout.

When you deploy a vLLM or SGLang inference service with a Model Gallery template, the template sets the engine HTTP Keep-Alive to 60 seconds by default.

For the fully managed gateway and ALB dedicated gateway, the default configuration works as follows:

• Model Gallery engine server idle timeout 60 seconds > ALB gateway client idle timeout 15 seconds

• Model Gallery engine server idle timeout 60 seconds > fully managed gateway client idle timeout 30 seconds

The Model Gallery template default of 60 seconds meets the requirements for both the ALB dedicated gateway and the fully managed gateway.

NLB requires special attention:

Model Gallery engine server idle timeout 60 seconds < NLB client idle timeout 900 seconds

With this configuration, the backend inference service may close idle connections first, while NLB or upstream connection management still considers the connection available. Subsequent requests that reuse such a connection may encounter EOF, RST, connection resets, or intermittent 502/503 errors.

When using NLB with vLLM or SGLang services deployed through Model Gallery, set the backend inference service server idle timeout to a value greater than 900 seconds with a safety margin. For example:

# vLLM, example value
VLLM_HTTP_TIMEOUT_KEEP_ALIVE=1000

# SGLang, example value
SGLANG_TIMEOUT_KEEP_ALIVE=1000

Client configuration examples

The client idle timeout must be less than the gateway server idle timeout. The following examples show how to manage or set the client idle connection timeout in different programming languages.

import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.client.config.RequestConfig;
import java.util.concurrent.TimeUnit;

public class HttpClientIdleTimeout {
    public static void main(String[] args) throws InterruptedException {
        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        // Example value: max connections per route
        cm.setDefaultMaxPerRoute(20);
         // Example value: max total connections
        cm.setMaxTotal(100);

        RequestConfig requestConfig = RequestConfig.custom()
                // Example value: connect timeout 5 seconds
                .setConnectTimeout(5000)
                // Example value: read timeout 10 seconds
                .setSocketTimeout(10000)
                .build();

        try (CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(cm)
                .setDefaultRequestConfig(requestConfig)
                .build()) {

            // Start a background thread to clean up idle connections periodically
            Thread cleanerThread = new Thread(() -> {
                try {
                    while (!Thread.currentThread().isInterrupted()) {
                        // Example value: check every 5 seconds
                        Thread.sleep(5000);
                        // Close connections idle for more than 500 seconds (less than gateway idle timeout of 600 seconds)
                        cm.closeIdleConnections(500, TimeUnit.SECONDS);
                        // Close expired connections (e.g., connections closed by the server)
                        cm.closeExpiredConnections();
                    }
                } catch (InterruptedException e) {
                    // Restore interrupted status
                    Thread.currentThread().interrupt();
                }
            });
            // Set as daemon thread so it exits when the main thread exits
            cleanerThread.setDaemon(true);
            cleanerThread.start();

            // Execute HTTP requests...
            // Example: httpClient.execute(new HttpGet("http://your-gateway-url"));

            // Simulate the program running for a period. Example value: run for 1 minute
            Thread.sleep(60000);

            // Stop the cleaner thread (in production, gracefully shut it down during application shutdown)
            cleanerThread.interrupt();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

package main

import (
    "net/http"
    "time"
)

func main() {
    // Create a custom Transport
    tr := &http.Transport{
        MaxIdleConns:        100,              // Maximum number of idle connections
        IdleConnTimeout:     500 * time.Second, // Idle connection timeout, e.g. 500 seconds (less than 600 seconds)
        DisableKeepAlives:   false,            // Enable Keep-Alive
    }
}

import com.aliyun.openservices.eas.predict.http.HttpConfig;

public class EasSdkTimeoutJava {
    public static void main(String[] args) {
        // 1. Global client configuration
        HttpConfig httpConfig = new HttpConfig();
        // Enable idle connection cleanup, in milliseconds. Set based on your client requirements.
        httpConfig.setConnectionCleanupInterval(5000);
        // Set the idle connection timeout to 500 seconds, less than the fully managed gateway idle timeout of 600 seconds
        httpConfig.setIdleConnectionTimeout(500000);
        ...
    }
}

Recommended configuration

Adjust the request timeout based on your business requirements. In production, balance fault tolerance, responsiveness, model processing time, and network jitter.

To prevent the client from timing out while the server is still processing, set the client request timeout slightly higher than the gateway request timeout.

• A common formula: client timeout = server request timeout + 1–5 seconds.

• For fast-fail scenarios, set a shorter client timeout and rely on idempotence and retry strategy to handle failures.

For long-running scenarios that exceed 10 minutes, use one of these approaches instead:

• Streaming, such as AI content generation and large file downloads.

• WebSocket, for persistent bidirectional communication.

• Asynchronous inference or task polling.

The fully managed gateway has a default request timeout of 10 minutes (600 seconds). The following diagram illustrates the request timeout flow:

Request timeout by access method

Client configuration examples

The following examples show how to configure the client request timeout in different programming languages.

import org.apache.http.client.config.RequestConfig;

public class ApacheHttpClientTimeout {
    public static void main(String[] args) {
        // Set the client request timeout slightly above the gateway request timeout.
        // If the gateway default is 600 seconds, use 610 seconds here.
        RequestConfig requestConfig = RequestConfig.custom()
                // Example value: connection timeout in milliseconds
                .setConnectTimeout(5000)
                // Data transfer timeout (read timeout) in milliseconds (610 seconds)
                .setSocketTimeout(610000)
                .build();
    }
}

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "time"
)

func main() {
    // Recommended: set client request timeout slightly greater than or equal to the gateway request timeout (default 600 seconds). Use 610 seconds here.
    client := &http.Client{
        Timeout: 610 * time.Second, // Timeout for the entire request
    }

    req, err := http.NewRequest("GET", "http://your-gateway-url", nil)
    if err != nil {
        fmt.Println("Error creating request:", err)
        return
    }

    // You can also set a shorter timeout for individual requests
    ctx, cancel := context.WithTimeout(req.Context(), 610*time.Second) // 610 seconds
    defer cancel()
    req = req.WithContext(ctx)

    resp, err := client.Do(req)
    if err != nil {
            fmt.Println("Error sending request:", err)
            // Check if it is a timeout error
            if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() {
                    fmt.Println("Request timed out!")
            }
            return
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        fmt.Println("Error reading response body:", err)
        return
    }
    fmt.Printf("Response Status: %s\n", resp.Status)
    fmt.Printf("Response Body: %s\n", body)
}

import com.aliyun.openservices.eas.predict.http.HttpConfig;

public class EasSdkTimeoutJava {
    public static void main(String[] args) {
        // 1. Global client configuration
        HttpConfig httpConfig = new HttpConfig();
        // Connection timeout
        httpConfig.setConnectTimeout(5);
        // Read timeout. Set the client request timeout slightly above the gateway request timeout.
        // If the gateway default is 600 seconds, use 610 seconds here.
        httpConfig.setReadTimeout(610);

    }
}

Idle connection timeout misconfiguration

Request timeout misconfiguration