Overview of Flink AI capabilities
Realtime Compute for Apache Flink integrates AI capabilities for a full range of scenarios, from SQL function calls and Python data processing to building streaming agents. You can perform large model inference, vectorization, multimodal analysis, and agent orchestration all within a single Flink job.
On the operations side, Flink AI Native Ops provides full-link AI O&M capabilities such as the AI assistant, Health Patrol, and skill management. For details, see Flink AI Native Ops.
Built-in model service
The platform includes built-in large models from the Qwen series, which are ready to use out of the box with no API key required.
Capability | Built-in model |
Text generation, visual understanding | qwen3.6-plus, qwen3.6-flash, qwen3.5-plus, qwen3.5-flash |
Text vectorization | text-embedding-v4 |
Multimodal vectorization | qwen3-vl-embedding |
SQL AI functions
Directly call large models in Flink SQL without writing application code.
Text: Complete sentiment analysis, entity extraction, tagging and classification, translation, summary generation, and more with a single SQL statement.
Vector: Perform text vectorization and vector search. Vectorization results can be cached to reduce the overhead of repeated calls.
Multimodal: Classify images, extract information, and generate content descriptions. The model takes an image URL as input and outputs structured results.
Use cases: Real-time data cleansing and annotation, streaming text analysis, online feature engineering, and real-time image classification and information extraction.
Python DataFrame API
A high-level AI data processing interface for Python developers. Use chained calls to connect data ingestion, preprocessing, AI inference, and output.
Core capabilities:
Multimodal support: Accepts both text and images as input.
Built-in operators: Includes built-in operators for large model inference and vectorization that can be integrated with a single line of code.
Custom preprocessing: Supports custom UDFs for data preprocessing.
Use cases: E-commerce product image analysis and description generation, multimodal content understanding, and intelligent batch data processing.
Flink Agents
Build event-driven, streaming AI agents using the open-source Apache Flink Agents framework.
Core capabilities:
Distributed coordination: The framework handles event routing, partitioning, state consistency, and automatic fault recovery.
Stateful memory: Agent memory is automatically persisted and restored at each checkpoint.
Stream processing: Agents continuously respond to incoming events with high throughput and low latency.
Dual-mode agents: Use Workflow agents for scenarios with well-defined processes and ReAct agents for scenarios that require flexible decision-making.
Supports both Python and Java development. Use cases: Real-time user review analysis, intelligent customer service, and automated business decision-making.
Agent Skills (natural language O&M)
Use Alibaba Cloud Flink Agent Skills to manage Flink resources in natural language in AI agent clients such as Claude Code and Cursor.
Instance management: Create instances, query status, and manage namespaces.
Workspace O&M: Create and validate SQL drafts, deploy and start/stop jobs, manage Session Clusters, and diagnose jobs.
No manual OpenAPI calls are required. Operations can be handled through conversation.
AI assistant
A conversational AI assistant built into the Realtime Compute console that lets you manage the entire lifecycle of your Flink jobs through natural language interaction.
Intelligent code generation: Automatically generates Flink SQL code from your business requirements. It supports building complete logic, including source table declarations, dimension table joins, and window calculations.
Troubleshooting: Analyzes job runtime exceptions and error logs to identify root causes and recommend fixes.
Performance tuning: Identifies bottlenecks such as data skew and oversized states based on Flink best practices, and suggests parameter tuning and SQL rewriting.
Health Patrol
A job health diagnosis service that combines scheduled and manual patrols. It patrols all jobs automatically, generates reports with health scores, root cause analysis, and fix suggestions, and can push reports to DingTalk or Feishu groups.
Third-party model services
Connect to your self-managed or third-party models through inference services like Triton Inference Server or PAI-EAS to perform online inference in your Flink jobs. This approach is ideal for scenarios when you have existing model assets and need to manage the inference services yourself.