MSE cloud-native gateways provide AI-optimized traffic handling, security protection, and a plugin ecosystem for LLM applications.
AI gateway traffic has three distinguishing features:
-
Persistent connections: WebSocket and SSE protocols dominate AI workloads, creating many persistent connections. Gateway configuration updates must not drop these connections.
-
High latency: LLM inference latency far exceeds standard applications, making AI services vulnerable to low-cost slow-request attacks that consume significant server resources.
-
High bandwidth: LLM context transmission over high-latency connections consumes far more bandwidth than standard traffic. Without efficient streaming and memory recovery, gateway memory usage spikes rapidly.
MSE cloud-native gateways are well suited for AI traffic:
-
Lossless hot updates for persistent connections: MSE gateways use Envoy for seamless configuration hot updates without dropping connections, unlike Nginx which requires reloads that can cause disconnections.
-
Security gateway capabilities: Multi-dimensional CC attack mitigation based on IP, cookies, and other factors. For AI workloads, throttling extends beyond QPS to token throughput.
-
Efficient streaming: Full stream forwarding on the Envoy (C++) data plane minimizes memory usage in high-bandwidth scenarios. While memory costs less than GPUs, poor memory management can still trigger OOM errors, causing service outages and significant losses.
Out-of-the-box AI plugins cover security, multi-model adaptation, observability, caching, and prompt engineering:
-
AI proxy plugin: Supports multiple protocols across 15 LLM providers, covering most major model vendors.
-
AI content moderation plugin: Integrates Alibaba Cloud Content Moderation to block harmful, misleading, discriminatory, illegal, or non-compliant content.
-
AI statistics plugin: Calculates token throughput, generates real-time Prometheus metrics, and logs data to access logs and Tracing Analysis spans.
-
AI throttling plugin: Throttles based on token throughput for backend protection. Supports precise per-tenant call quota limits.
-
AI developer plugin set: Provides capabilities such as LLM result caching and prompt decoration for AI application development.
