Data lake lifecycle management overview
Managing data lakes built on modern table formats such as Delta Lake and Apache Iceberg requires systematic operations similar to database management. The Spark engine of AnalyticDB for MySQL provides lifecycle management for data lake tables, enabling query performance comparable to a data warehouse.
Data mutation (DML)
The managed Spark engine of AnalyticDB for MySQL supports insert, delete, and update operations on data lake tables stored on immutable media such as Object Storage Service (OSS).
|
Core capability |
Technical details |
Importance |
|
Upsert (merge) |
Similar to the SQL |
A core capability for Change Data Capture (CDC) synchronization. Essential for processing change logs (binary logging) from sources like Salesforce or MySQL to build a real-time data lake. |
|
Logical deletion (delete) |
Even on append-only storage, specific rows can be deleted by marking them or rewriting files. This supports requirements such as the "right to be forgotten" under the General Data Protection Regulation (GDPR). |
Corrects incorrect data or responds to user requests to delete data. |
|
Overwrite |
Atomically replaces the data of an entire partition or table. |
Idempotence. If a data processing task fails or produces incorrect results, the ETL task can be rerun to automatically overwrite old data and prevent data duplication. |
Storage layout optimization
Optimizing the storage layout is essential for reducing query latency from hours to seconds and resolving the "small file problem". By using the managed Spark engine of AnalyticDB for MySQL together with Alibaba Cloud scheduling tools such as DataWorks or DMS Airflow, you can regularly optimize the file layout to achieve data-warehouse-level analytics performance on your data lake.
|
Core capability |
Technical details |
Importance |
|
Small file compaction (compression/bin-packing) |
Merges thousands of small, kilobyte-sized files into larger files, such as 128 MB to 1 GB. |
Reduces I/O latency. Object storage is sensitive to |
|
Data clustering (Z-ordering) |
Physically stores related data, such as records with the same |
Data Skipping allows queries to skip up to 90% of files, greatly reducing the amount of data scanned. |
|
File cleanup (vacuuming/expiring snapshots) |
Physically deletes old version history files that are no longer needed. |
Cost control. Prevents historical data versions from causing unbounded growth in OSS costs. Also physically removes data as required for compliance. |
Metadata and schema management (data governance)
|
Core capability |
Technical details |
Importance |
|
Schema evolution |
Supports adding new columns, renaming columns, or changing column types without rewriting historical data. |
Adapts to business changes. When new fields are needed, the data lake schema can evolve without rewriting historical data, preventing frequent pipeline failures. |
|
Schema enforcement |
Rejects writes of data that do not conform to the schema, such as data with mismatched types. |
Data quality gatekeeping. Prevents dirty data from contaminating "gold" or "silver" tables, avoiding errors when downstream analysts query data. |
|
Partition management |
Dynamically generates new partitions or replaces data in specific partitions. |
Query pruning. Ensures that a |
Time travel and versioning
With the managed Spark engine of AnalyticDB for MySQL, you can query historical snapshots of data lake tables and roll back data to a previous version.
|
Core capability |
Technical details |
Importance |
|
History rollback (time travel) |
Queries the state of a table at a specific point in the past, such as "yesterday at 2:00 PM", by using syntax like |
Disaster recovery and debugging. If user error corrupts data, you can quickly roll back to a correct version for investigation or recovery. |
|
Audit log |
Records commit logs for data change operations, including who performed the operation, when, and what was changed. |
Security and traceability. Provides a full history of data changes to help investigate the root cause of data anomalies. |