E-MapReduce (EMR) is an open source big data platform and processing solution that runs on Alibaba Cloud. EMR is built on open source frameworks, such as Apache Hadoop and Apache Spark, allowing you to use other systems in the Hadoop and Spark ecosystems to analyze and process data. You can also use EMR to transfer data to and from other Alibaba Cloud data storage and database systems, such as Object Storage Service (OSS) and ApsaraDB RDS (RDS).
Product introduction
Alibaba Cloud EMR provides EMR on ECS, EMR on ACK, and Serverless options to meet the needs of different users.
Type | Description |
EMR on ECS | EMR installs and deploys components from the open source Hadoop ecosystem on Elastic Compute Service (ECS) instances and starts the corresponding services. You can perform O&M operations on the cluster's ECS instances and services in the EMR console. For more information about EMR on ECS, see What is EMR on ECS?. |
EMR on ACK | First, install and deploy a Container Service for Kubernetes (ACK) cluster. After the ACK cluster is ready, EMR installs and deploys big data service components based on the ACK resources. The components run inside containers. For more information about EMR on ACK, see What is EMR on ACK?. |
EMR Serverless StarRocks | EMR Serverless StarRocks is a fully managed service for the open source StarRocks on Alibaba Cloud. It enables you to flexibly create and manage instances and data. This topic describes the core features of StarRocks and details the enhanced features and service advantages of EMR Serverless StarRocks. For more information about EMR Serverless StarRocks, see What is EMR Serverless StarRocks?. |
EMR Serverless Spark | EMR Serverless Spark is a high-performance lakehouse product for data and AI. It provides a one-stop data platform for enterprises with features such as task development, debugging, scheduling, and operations and maintenance (O&M). This simplifies the entire process of data processing and model training. The product is 100% compatible with the open source Spark ecosystem and can be seamlessly integrated into your existing data platform. EMR Serverless Spark allows enterprises to focus on optimizing data processing, analysis, and model training to improve work efficiency. For more information about EMR Serverless Spark, see What is EMR Serverless Spark?. |
Service architecture
Benefits
EMR on ECS
EMR provides a convenient and manageable enterprise-grade open source big data service. You can quickly set up open source big data services, such as Hadoop, Spark, Flink, Kafka, and HBase.
Uses 100% community open source components. These components are adapted and optimized for performance that far exceeds the open source versions.
Provides time-based elastic scaling. You can use Spot Instances to further reduce costs.
Decouples computing from storage. This architecture allows for elastic resource utilization.
You can create and scale out clusters in minutes without having to manually deploy or start services.
EMR on ACK
Cost-effective: You do not need to purchase a separate ACK cluster.
Simplified O&M: A unified O&M and cluster management system covers various services, including big data and online services.
Optimized experience: Supports two Infrastructure as a Service (IaaS) resource models, ECS and ACK, for a seamless switching experience.
Deep integration: Uses a fully cloud-native data lake architecture. Computing uses Alibaba Cloud ACK, which allows for infinite scaling of computing resources.
EMR Serverless StarRocks
EMR Serverless StarRocks provides the following enhancements for enterprise-grade features:
A fully managed service that simplifies operations and maintenance (O&M) and usage, greatly reducing complexity and costs.
A visual management console for StarRocks instances that makes overall instance O&M and management more convenient.
Visual monitoring and O&M capabilities.
Support for automatic major and minor version upgrades, which simplifies version management for StarRocks.
The addition of EMR StarRocks Manager, which provides enterprise-grade management capabilities for StarRocks:
Security: Supports user and permission management.
Diagnostics and analysis: Supports visualization for slow SQL statements and SQL query analysis.
Data management: Provides query capabilities for databases, tables, partitions, shards, and tasks to facilitate O&M.
EMR Serverless Spark
Cloud-native high-speed compute engine
Built-in Fusion Engine (Spark Native Engine): Delivers a 300% performance improvement over the open source version and significantly accelerates big data computing tasks. The engine optimizes computing efficiency with a vectorized engine and batch data processing technology. It also reduces memory usage, which improves overall performance.
Built-in Celeborn (Remote Shuffle Service): Supports petabyte-scale shuffle data processing, which greatly improves the stability and performance of large shuffle tasks. Compute nodes do not require large disks. The service fully utilizes Spark's dynamic resource scaling capabilities to reduce storage costs. The total cost of computing resources can be reduced by up to 30%.
Open data lake architecture
On-demand elastic scaling: Supports a compute-storage decoupled architecture. Computing resources can scale elastically within seconds, with a minimum granularity of one core. Resources are metered at a fine-grained task or queue level. Storage uses a pay-as-you-go model to prevent resource waste and significantly reduce operational costs.
Seamless migration and compatibility: Integrates with OSS-HDFS and is fully compatible with HDFS cloud storage, which supports a smooth migration of your business to the cloud. It uses DLF to fully integrate lakehouse metadata. This ensures data access consistency and complete permission management, which helps you easily build a modern data lakehouse architecture.
One-stop developer experience
End-to-end development support: Provides a one-stop development experience from task development, debugging, and publishing to scheduling. This meets the high standards for enterprise-level development and release. The built-in version management feature records the complete history of each release and supports source code and configuration difference comparisons to ensure that changes are traceable.
Efficient collaboration and stability: Development and production environments are strictly isolated to ensure business stability. This helps teams collaborate efficiently and deliver stable results.
Serverless resource platform
Out-of-the-box: You can start task development quickly without manual management or complex infrastructure setup.
Second-level elasticity: Dynamically pulls resources and starts pods based on the resource requirements of Spark tasks. Resources are released immediately after the computation is complete. Billing is based only on the amount of resources that are actually used, which further reduces the total computing cost.
Cost estimation: Provides task-level resource metering and cost estimation to help you achieve fine-grained operations.