The Data+AI module of Data Management (DMS) provides a unified interface for data integration, data development, and data services. It is designed for data developers, business developers, analysts, and data operators.
Feature overview
The DMS Data+AI feature supports multiple computing and storage engines. It provides real-time and offline integration, development, and services for structured, semi-structured, and unstructured data. This feature meets various enterprise needs for data transformation, integration, development, and services.
You can use unified batch and stream data integration to load and transform online data. You can use the Data Studio module to develop data warehouse data in layers. Then, you can provide data analytics and usage services using DataService Studio and visualization features. You can also integrate these features as basic capabilities into your own data platform to build a custom enterprise data platform.
Benefits
| Benefit | Description |
|---|---|
| Low-code development | Reduces the engineering effort required to build data pipelines. |
| Flink and Spark compatibility | Full compatibility lets you use familiar frameworks without modification. |
| Built-in data security | Security controls apply consistently across data chains and during data development. |
| Minute-level scheduling | Fine-grained control over task execution timing. |
| Multi-environment management | Isolate development, testing, and production configurations. |
Unified batch and stream technology that supports real-time and offline integration for over 20 types of data.
Use cases
Data integration
O&M: Data disaster recovery, active geo-redundancy, data archiving, data migration, test data generation, O&M metric monitoring, and business metric monitoring.
Development: Real-time reports, log analysis, offline wide tables, T+1 data snapshots, data aggregation, data cleaning, and data masking.
Data Development
Database development: Cross-database development, scheduled tasks, data archiving, data migration, and report development.
Data warehouse development: Data loading, data cleaning, data transformation, data tiering, report development, and wide table development.
Notebook
Notebook integrates with large language models (LLM) to accelerate data delivery and self-service data analysis. Use it to present query results, test data, and data change trends in document form — for business developers, data developers, analysts, and data operators. After a document is published, the DMS AI Agent provides around-the-clock Q&A on its content.
For more information, see Notebook.
DTS
DTS is the data transfer component of DMS. It covers four capabilities: data synchronization, data migration, change tracking, and data verification.
Data synchronization
Data synchronization replicates data between sources in real time. It is suited for active geo-redundancy, geo-disaster recovery, zone-disaster recovery, cross-border data synchronization, cloud business intelligence (BI) systems, and real-time data warehousing.
For more information, see Overview of data synchronization solutions.
Data migration
Data migration moves data between homogeneous or heterogeneous data sources in a single operation. Common scenarios include migration to Alibaba Cloud, migration between instances within Alibaba Cloud, and database splitting and scale-out.
For more information, see Overview of data synchronization solutions.
Change tracking
Change tracking surfaces real-time incremental data from your databases, enabling downstream consumers to react to changes as they occur.
For more information, see Overview of change tracking scenarios.
Data verification
Data verification monitors differences between source and destination databases. Run it without downtime to detect inconsistencies in data and schemas as early as possible.
For more information, see What is data verification?
Data integration
Offline integration
Offline integration is a low-code data development tool. You can combine various task nodes to form a data stream and use a recurring schedule to run the stream for data transformation and data synchronization. This lets you transform data from online databases and data warehouses and synchronize it to the destination. For more information, see Offline integration overview.
Streaming ETL
You can configure streaming data transformation jobs in two ways: using drag-and-drop operations in a visual interface, or using SQL statements that are 100% compatible with Flink. This provides capabilities for extracting, transforming, and loading streaming data. It is suitable for various real-time data development scenarios, such as log processing, real-time transformation of online data, and real-time statistical reports. For more information, see Streaming ETL.
Data development
Task orchestration
Task orchestration is mainly used to orchestrate and schedule various tasks. You can create a task flow composed of one or more task nodes to implement complex task scheduling and improve data development efficiency. For more information, see Overview.
Data warehouse development
Data warehouse development is mainly used for immersive data warehouse development. You can create a project, select a data warehouse engine and environment, and then create, publish, and run multiple data warehouse development tasks. This implements complex data warehouse development processes and improves development efficiency and management capabilities. For more information, see Create a project.
The task orchestration feature lets you compose task flows from one or more task nodes to implement complex scheduling logic. Use it to coordinate multi-step data development pipelines and reduce manual coordination overhead.
For more information, see Overview.
Data applications
DataService Studio
DMS DataService Studio supports making enterprise data hosted on DMS rapidly available to external systems. For more information, see DataService Studio.
DataAnalysis
DMS DataAnalysis provides datasets and dashboards. You can visually analyze data and render the results intuitively in dashboards. For more information, see DataAnalysis.