The Lindorm compute engine is a high-performance, cost-effective, and reliable distributed computing service built on the core capabilities of Lindorm. It handles computing demands within Lindorm, including data production, interactive analytics, machine learning, and graph computing, and supports concurrent processing of massive data. You own the resources. The compute engine fully supports the open source Spark computing model and programming interfaces and is deeply integrated with the Lindorm storage engine, leveraging its data storage features and indexing capabilities for efficient distributed computing.
Features
|
Core capabilities |
Description |
|
Multiple connection types |
Analyze data interactively using Java Database Connectivity (JDBC). You can also submit JAR packages to customize distributed computing tasks. |
|
Data lake analytics |
Supports creating and writing data to column-oriented tables in Iceberg and Hudi formats. It also supports batch and online analytics based on data lake tables. |
|
Out-of-the-box |
Access permissions are shared across the compute engine and storage engines, including LindormTable, LindormTSDB, and the search engine, eliminating complex component configuration. Developers need only SQL knowledge and Spark development experience to use the service. |
|
Fully managed |
You do not need to manage cluster O&M operations, such as configuration, upgrades, downgrades, or scaling. Simply manage jobs through the console and the Spark UI. |
|
Elasticity |
Elastic scaling reduces computing costs for workloads with significant peaks and troughs, minimizes idle resource waste, and handles traffic spikes quickly. |
|
Pay-as-you-go |
You are billed based on the actual compute resources used. |
|
Heterogeneous computing power |
Supports elastic computing power from both CPUs and GPUs. |
Before you activate the compute engine, contact Lindorm technical support (DingTalk ID: s0s3eg3) to have the required permissions enabled. Otherwise, a permission error will occur.
Service architecture
The following figure illustrates the Lindorm compute engine architecture:
The compute engine is a cloud-native distributed computing service whose compute nodes run in Alibaba Cloud Container Service for Kubernetes (ASK). It supports community edition computing models and programming interfaces and is deeply integrated with the Lindorm storage engine, leveraging its data storage features and indexing capabilities to efficiently complete distributed jobs. The engine delivers high-performance computing for data production, interactive analytics, and machine learning. During Spark job execution, you can use the Spark web UI (SparkUI) to monitor and manage Spark jobs.