Data Integration DI Agent

Description

DI Agent allows you to create and manage four types of sync tasks through natural language: batch single-table, real-time single-table, batch full-database, and real-time full-database. Instead of filling out form fields one by one for data sources, field mappings, and scheduling policies, you only need to describe your sync requirements and the Agent completes the entire process from intent understanding to deployment. Key capabilities include:

• Natural-language task creation: Describe your sync requirements in one sentence (for example, "Sync the entire RDS order database to Hologres in real time, partitioned by day"), and the Agent automatically completes data source identification, schema detection, field mapping, resource group assignment, and scheduling policy configuration.

• Intent understanding and multi-turn conversations: The Agent can understand vague or incomplete requirement descriptions and asks follow-up questions to fill in key parameters (such as partitioning strategy and incremental conditions) to ensure that the generated task configuration is accurate and usable.

• Full task lifecycle management: You can edit, pause, resume, delete, and rerun tasks through conversation, covering the complete lifecycle from task creation to decommissioning.

• Batch task operations: You can create or modify multiple sync tasks in a single conversation, which is suitable for scenarios such as full-database migration and bulk scheduling policy adjustments.

Example

The following example demonstrates how to create a sync task through conversation:

1. Go to the DI Agent conversation interface.

2. Enter your sync requirements. For example: "Sync the entire RDS order database to Hologres in real time, partitioned by day".

3. The Agent automatically detects the source schema, displays the list of tables to be synced, and continues after you confirm.

4. The Agent automatically completes field mapping, resource group assignment, and scheduling policy configuration.

5. Preview the task configuration summary and deploy the task after confirming everything is correct.

6. Check the task running status and verify that data is being synced properly.

Description

DI Agent allows you to embed large language model inference as an ETL operator in data pipelines to perform batch AI processing on structured and unstructured data (images, videos, audio files, documents, and more) from any source. Processing operations include recognition, classification and tagging, transcription, translation, summarization, vectorization, and custom processing logic. Results are written directly to any destination supported by data integration (such as MaxCompute, Hologres, or Data Lake Formation (DLF)), enabling all types of data to enter the enterprise data asset system in the appropriate format. Key capabilities include:

• Natural-language processing logic definition: Describe your processing intent in natural language (for example, "Identify product categories in images and output labels"), and the Agent automatically generates the corresponding processing pipeline configuration.

• Three-stage pipeline orchestration: The Agent automatically orchestrates a Source → Transform (large model inference) → Sink pipeline without requiring you to write scripts or set up inference services.

• Direct result writing to various destinations: For example, processing results are written directly to MaxCompute, Hologres, or DLF data lakes without additional data transfer.

• Data embedding: Vectorize processed data and write it to a specified storage destination for subsequent RAG use.

• Batch and periodic execution: Supports batch processing of millions of files and can be configured for scheduled periodic execution to continuously process incremental data.

Example

The following example demonstrates how to process product images in OSS and write the results to MaxCompute:

1. Go to the AI data processing entry point.

2. Enter your processing requirements. For example: "Read product images from an AWS S3 bucket → Call a large model to identify classification labels → Write the results to a MySQL table".

3. The Agent automatically orchestrates a three-stage pipeline: Source (S3) → Transform (large model inference) → Sink (MySQL).

4. ChatDB automatically identifies the target table and generates the corresponding SQL statement.

5. View the returned chart results (line charts, bar charts, and other chart types are automatically matched).

6. If you need to vectorize data in a feature engineering database into Hologres for RAG use, you can define the embedding in natural language: "Continue with embedding → Call a large model for vectorization → Write the results to a Hologres table".

7. The Agent automatically orchestrates a three-stage pipeline: Source (MySQL) → Transform (large model vectorization) → Sink (Hologres).

8. ChatDB automatically identifies the target table and generates the corresponding SQL statement.

9. View the returned results.

Query data source schemas and business semantics

ChatDB allows you to explore database metadata and interpret business semantics using natural language. After you specify a target data source, the Agent automatically connects and drills down through the complete structure of databases, tables, and fields, including table names, field types, primary and foreign key relationships, index constraints, and other metadata details. The Agent also interprets the business semantics of databases and tables based on table names, field naming conventions, and data characteristics, helping you quickly understand the purpose of each table, the business concepts represented by fields, and relationships between tables.

Use cases: Quickly understanding the full data landscape when taking over an unfamiliar system, and helping non-technical team members understand data structures during cross-team collaboration.

Data source sample data preview

ChatDB allows you to view sample data from any table using natural language. After you specify a target table, the Agent automatically retrieves representative data samples with explanations, so you can visually confirm the data content and format stored in the table without writing SQL.

Use cases: Quickly verifying data content and understanding actual field values, and helping non-technical team members understand data semantics during cross-team collaboration.

Statistical analysis with chart visualization

ChatDB supports natural-language-driven data analysis and visualization. Describe your analysis intent (such as "show the sales trends by category over the past three months" or "compare order proportions across regions"), and the Agent automatically locates the data source, generates the query logic, and presents the analysis results as line charts, bar charts, pie charts, or other chart types — no SQL or specialized visualization tools required.

Use cases: Business teams quickly validating operational hypotheses, and technical teams obtaining trend data on the fly during troubleshooting.

Schema diagnostics and optimization recommendations

ChatDB supports intelligent diagnostics of database and table schemas for a specified data source. The Agent automatically scans the table schema designs of the target database and reviews them across multiple dimensions, including naming conventions, field type selection, index coverage, primary and foreign key integrity, redundant fields, and partitioning strategies. It identifies potential design defects and provides specific optimization recommendations and improvement directions.

Use cases: Design reviews for newly created databases and tables, technical debt assessments for existing systems, and root cause analysis for database performance issues.

Database performance troubleshooting and optimization

ChatDB supports database performance diagnostics and slow SQL root cause analysis. After you describe a performance issue or specify a target data source, the Agent automatically analyzes the factors causing slow queries, including full table scans, index invalidation, suboptimal JOIN ordering, deeply nested subqueries, implicit data type conversions, stale statistics, lock contention, and hot tables. It explains the root cause of each slow SQL statement based on the actual execution plan and provides targeted optimization recommendations such as index adjustments, query rewrites, and table schema refactoring.

Use cases: Emergency troubleshooting for sudden production performance incidents, and proactive management of potential slow queries during routine inspections.

Overview

After you receive a sync task alert, describe the issue or paste the alert information in the Agent conversation interface. The Agent automatically aggregates multi-dimensional information such as task logs, resource monitoring metrics, and source database status, identifies the root cause, and provides a remediation plan — reducing troubleshooting time from hours to minutes. Each diagnostic result is automatically archived as a knowledge base entry, so the next time a similar issue occurs, the existing solution is matched directly, preventing teams from starting from scratch when personnel changes occur. Key capabilities include:

• Multi-dimensional information aggregation: Automatically collects and correlates task logs, resource monitoring metrics, source database status, network bandwidth, and other multi-dimensional information, presenting the complete context in a single view.

• Root cause identification: Automatically analyzes the root cause based on aggregated information (such as insufficient CUs, source-side table locks, network bandwidth bottlenecks, or configuration errors) and provides a clear conclusion.

• Remediation recommendations with one-click execution: After identifying the root cause, the Agent provides actionable remediation suggestions (such as scaling up resource groups, restarting instances, or adjusting concurrency). Once confirmed, you can execute the fix with one click.

• Automatic knowledge accumulation: Each diagnostic result is automatically archived as a knowledge base entry, so the next time a similar issue occurs, the existing solution is matched directly.

Usage example

1. After an alert is triggered, open the Agent conversation interface and describe the issue or paste the alert information.

2. The Agent automatically aggregates multi-dimensional information such as task logs, resource monitoring metrics, and source-side status.

3. The Agent outputs a root cause analysis report (for example: insufficient CUs causing the consumption rate to fall below the production rate).

4. The Agent provides remediation recommendations (for example: scaling up the resource group CU quota or restarting failed instances). After you confirm, the fix is executed.

Overview

DI Agent provides automated intelligent inspection tasks created and managed by the AI Agent. You do not need to write monitoring scripts or configure alert rules. After you create an inspection task and enable automatic scheduling, the Agent proactively inspects the running status of sync tasks at the configured frequency and generates a detailed result report after each run, shifting the O&M model from reactive response to proactive governance. Key capabilities include:

• Configurable run frequency: The run frequency is configured using Cron expressions, supporting one-time, hourly, daily, weekly, monthly, and custom Cron schedules. The default is 08:00 every day. You must configure the run frequency before enabling automatic scheduling.

• Full-pipeline proactive scanning: The Agent automatically scans the running status of all sync pipelines on a periodic basis, proactively identifying potential risks without manual item-by-item checks.

• Detailed result reports: After each inspection, a detailed result report is generated. You can view the report in the task's run history. Reports can be opened in a new window or downloaded for team sharing and archiving.

• Priority ranking and action recommendations: High-risk tasks are ranked by impact scope and urgency, with each item accompanied by specific action recommendations and expected benefit assessments.

Usage example

1. Log on to the Data Integration console. In the left-side navigation pane, click Intelligent Inspection Tasks under AI Native.

2. Create an inspection task. Click Create Intelligent Inspection Task and select Create via Agent Chat or Create Manually from the drop-down menu. Alternatively, click a recommended scenario card at the top of the page to quickly create a task from a built-in scenario template. To create a task manually, specify the task name, description, resource group, run frequency, and task content (describe the inspection content in natural language).

3. After you turn on Auto Scheduling, the Agent automatically runs inspections at the specified frequency. You can also manually trigger a one-time inspection by choosing More > Execute Now in the task list.

4. On the run history page of a task, view the status and Result Details Report of each run. Reports support Open in New Tab and Download.

Overview

Inspections address periodic health checks, while the watch capability addresses real-time responses at the moment an exception occurs. The watch feature uses scenarios as management units, where a scenario is a collection of events. You first define the task types that the scenario applies to, then configure monitoring events and their trigger thresholds within the scenario. After the scenario is applied to specific tasks, you can override thresholds on a per-task basis. When a task matches an event, the Agent automatically collects runtime information, analyzes the root cause, and continuously tracks the task's recovery status. Key capabilities include:

• Built-in watch scenarios: Three built-in scenarios are provided: real-time sync latency watch, real-time sync runtime watch, and batch sync runtime watch. You can directly apply these scenarios to sync tasks of the applicable types.

• Event-level threshold management: A scenario can contain multiple monitoring events, each with an independently configured trigger threshold. After a scenario is applied to tasks, you can override thresholds for individual tasks to fine-tune monitoring for critical tasks.

• Trigger-based analysis: When business latency continuously exceeds the threshold, the Agent automatically analyzes the source, sync pipeline, and destination to identify potential performance bottlenecks. When a task fails or experiences frequent failovers, the Agent automatically analyzes the root cause. When a batch sync task or batch incremental instance fails, the Agent automatically collects runtime information and provides remediation suggestions.

• Recovery tracking and watch records: After completing the analysis, the Agent continuously tracks the task's recovery status. Each triggered event is recorded as a watch record that shows the trigger time, the tasks that matched the event, the specific events matched, the analysis conclusion, and whether the task has recovered. This forms a traceable O&M ledger.