The Lindorm compute engine is a high-performance, cost-effective, and reliable distributed computing service built on the core capabilities of Lindorm. The service handles computing demands for scenarios within Lindorm, such as data production, interactive analytics, machine learning, and graph computing. It supports the concurrent processing of massive data. You own the resources. The compute engine fully supports the open source Spark computing model and programming interfaces. It is also deeply integrated with the Lindorm storage engine. This integration leverages the underlying data storage features and indexing capabilities to efficiently complete distributed computing tasks.
Features
Core capabilities | Description |
Multiple connection types | Analyze data interactively using Java Database Connectivity (JDBC). You can also submit JAR packages to customize distributed computing tasks. |
Data lake analytics | Supports creating and writing data to column-oriented tables in Iceberg and Hudi formats. It also supports batch and online analytics based on data lake tables. |
Out-of-the-box | Access permissions are integrated between the compute engine and storage engines, including LindormTable, LindormTSDB, and the search engine. This integration eliminates complex configurations of underlying components. Developers can use the service with only SQL knowledge and Spark development experience. |
Fully managed | You do not need to manage cluster O&M operations, such as configuration, upgrades, downgrades, or scaling. Simply manage jobs through the console and the Spark UI. |
Elasticity | The compute engine provides elastic scaling. For business workloads with significant peaks and troughs, this feature can greatly reduce computing costs, minimize idle resource waste, and quickly handle traffic spikes. |
Pay-as-you-go | The compute engine is billed based on the actual amount of compute resources used. |
Heterogeneous computing power | Supports elastic computing power from both CPUs and GPUs. |
Before you activate the compute engine, contact Lindorm technical support (DingTalk ID: s0s3eg3) to have the required permissions enabled. Otherwise, a permission error will occur.
Service architecture
The following figure shows the architecture of the Lindorm compute engine:
The compute engine is a distributed computing service based on a cloud-native architecture. Its compute nodes run in Alibaba Cloud Container Service for Kubernetes (ASK). The compute engine supports community edition computing models and programming interfaces and is deeply integrated with the Lindorm storage engine. This integration leverages underlying data storage features and indexing capabilities to efficiently complete distributed jobs. It provides high-performance computing for scenarios such as data production, interactive analytics, and machine learning. During Spark job execution, the compute engine provides a job management interface. You can use the Spark web UI (SparkUI) for comprehensive monitoring and maintenance of Spark jobs.