Stream computing quick start
This guide helps you get started with ApsaraMQ for Kafka stream computing quickly and simply.
Available Regions
Live: Hangzhou, Beijing, Shanghai, Singapore
Key Terms
Reserved CU
Reserved CU is the minimum resource guarantee for a stream task—the number of compute units the instance continuously reserves for the task and never reclaims.
Purpose: Reserved CU is the task's resource baseline—locked regardless of traffic, ensuring stable off-peak operation without slowdown or interruption. It defines the task's guaranteed compute floor and the fixed base of billing: even when idle, reserved CUs remain occupied and billed.
Sizing advice: Set reserved CU to your business's average daily load—enough to handle most periods steadily. Too low triggers frequent elastic scale-out, adding scheduling overhead and potential latency; too high leaves resources idle off-peak, raising fixed costs. Calibrate by observing Busy percentage and CU count in monitoring over time: if daily Busy stays low, lower the reserved CU.
CU Cap
CU cap is the maximum compute boundary a stream task can scale to—the upper limit of compute units the instance may occupy during auto scaling.
Purpose: CU cap puts a ceiling on elasticity. When traffic rises beyond reserved CU, the system scales out automatically between reserved CU and the cap, absorbing peaks and suppressing backpressure and latency; when traffic falls, it scales back in. The cap preserves burst-handling capacity while preventing runaway resource expansion and uncontrolled costs from abnormal traffic or job issues. Elastic resources above reserved CU are billed by actual usage.
Sizing advice: Set the cap to your peak load plus headroom, so the task can still scale during promotions or traffic spikes. Too low means the task hits the ceiling at peak—backpressure rises and latency accumulates; too high does not affect normal operation but may bring unexpected costs during abnormal surges, so also configure resource usage alerts. The CU cap must be ≥ reserved CU.
Usage Limits
DML: UPDATE / DELETE not supported; INSERT INTO supports append or UPSERT mode only.
JOIN: JOIN semantics not currently supported.
Exactly-once: End-to-end exactly-once depends on the sink's idempotency or transactional capability (e.g., Kafka transactions).
Window job resources: Jobs using window functions consume more CU; configure at least 4 CU.
Overall throughput constrained by external systems: Performance reference values reflect internal compute only; actual throughput is limited by upstream/downstream connection counts, TPS, and quotas.
Resource cap: The resource cap ensures enough compute to process data promptly when real-time traffic rises. Keep sufficient account balance to avoid tasks stopping due to overdue payment.
Source and sink: Kafka stream computing must use an ApsaraMQ for Kafka instance as the source or sink.
Quick Start
Enable and purchase a compute instance: Enable stream computing and purchase a compute instance in the ApsaraMQ for Kafka console. No charges are incurred at this step.
Deploy the instance: Deploy the instance in the console. Deployment mainly confirms the network architecture for the instance's future tasks. Wait until the status becomes Running.
Configure members and permissions: Add the Alibaba Cloud accounts or RAM users as members and grant the corresponding permissions via roles (the stream computing instance's API list must be supplemented).
Create a stream task: Create a stream task and use streaming SQL to define the input table, output table, and processing logic. ApsaraMQ for Kafka stream computing is compatible with Flink SQL and supports two creation methods:
Create via connector: Select an existing source or sink connector; the system generates the corresponding SQL template, and you add processing logic on top.
Create manually: Write the SQL by hand, or copy and modify an existing stream task's SQL.
Run the stream task:
Debug run: The console displays partial execution results for reference; data is not written to the sink. Compute metering and billing still apply.
Run: The task is deployed and executed per the SQL; data is written to the sink. Compute metering and billing apply.
Observe and tune: Watch CU utilization, throughput, lag, and backpressure via instance-level and task-level monitoring metrics; enable elastic scaling or adjust CU as needed.