ApsaraMQ for Kafka is a fully managed, distributed messaging service built on Apache Kafka that delivers high throughput and elastic scalability for real-time data pipelines, without the operational overhead of self-managed clusters.
Common workloads include log collection, monitoring data aggregation, streaming data processing, and large-scale online and offline analytics.
%E8%8B%B1%E6%96%87%E7%89%88Kafka-%E4%BF%AE%E6%94%B9%E7%AC%AC%E4%B8%89%E7%89%88.mp4
Prerequisites
Before you begin, you should understand the following concepts:
Why use a managed Kafka service
Running Apache Kafka in production requires significant operational investment. ApsaraMQ for Kafka handles these tasks so you can focus on your applications:
-
Cluster provisioning and configuration -- Deploy ready-to-use Kafka clusters without manual setup.
-
Patching and version upgrades -- Apply security patches and Kafka version updates with minimal disruption.
-
Monitoring and alerting -- Track cluster health and throughput through integrated tools.
-
Elastic scaling -- Scale broker capacity and storage to match changing workloads.
Compared to self-managed Apache Kafka, ApsaraMQ for Kafka reduces infrastructure costs, improves reliability, and scales with less effort.
Use cases
ApsaraMQ for Kafka supports a range of big data and real-time processing scenarios.
-
User behavior analysis -- Analyze website user actions and clickstream data to understand usage patterns.
-
Log aggregation -- Collect logs from distributed applications and infrastructure into a central pipeline for search, alerting, and analysis.
-
Monitoring and observability -- Aggregate monitoring data from multiple sources and stream it to dashboards and alerting systems for real-time operational visibility.
-
Stream processing -- Build event-driven architectures that react to data in real time.
-
Offline data analysis -- Buffer high-volume event streams and load them into data warehouses for historical analysis and reporting.
Ecosystem integrations
ApsaraMQ for Kafka integrates with Alibaba Cloud services and open-source frameworks for data movement and processing.
Data integration
Move messages from Kafka topics into downstream storage and analytics systems:
| Target service | Use case |
|---|---|
| MaxCompute | Large-scale data warehousing and batch analytics |
| Object Storage Service (OSS) | Long-term storage and archiving |
| ApsaraDB RDS | Relational database synchronization |
| Hadoop / HBase | Distributed storage for big data workloads |
Stream processing engines
Connect Kafka to real-time compute engines for continuous data processing:
| Engine | Use case |
|---|---|
| Realtime Compute for Apache Flink | Managed stream processing |
| E-MapReduce (EMR) | Managed Hadoop and Spark clusters for batch and streaming |
| Spark | In-memory analytics and streaming |
| Storm | Low-latency, event-by-event stream processing |
Get started
Choose a starting point based on your goal:
-
Create your first instance -- Follow the quick start guide to provision a Kafka instance and produce your first message.
-
Understand core concepts -- Learn about topics, partitions, consumer groups, and how ApsaraMQ for Kafka organizes data.
-
Connect your applications -- Use the SDK to integrate Kafka producers and consumers into your applications.
-
Set up data pipelines -- Configure connectors to move data between Kafka and other Alibaba Cloud services.
Customer testimonials
Read what our customers say about ApsaraMQ for Kafka.
XMiles
Activate ApsaraMQ for Kafka
Click the button below to activate ApsaraMQ for Kafka.