Data Integration DI Agent

Capability description

DI Agent allows you to create and manage four types of sync tasks through natural language: batch single-table, real-time single-table, batch full-database, and real-time full-database. Instead of filling out form fields one by one for data sources, field mappings, and scheduling policies, you simply describe your sync requirements, and the Agent completes the entire process from intent understanding to deployment. Key capabilities include:

• Natural language task creation: Describe your sync requirements in one sentence (for example, "Sync the entire RDS order database to Hologres in real time, partitioned by day"), and the Agent automatically completes data source identification, schema detection, field mapping, resource group assignment, and scheduling policy configuration.

• Intent understanding and multi-turn conversations: The Agent understands vague or incomplete requirement descriptions and asks follow-up questions to fill in key parameters (such as partition strategy and incremental conditions), ensuring that the generated task configuration is accurate and usable.

• Full task lifecycle management: Supports editing, pausing, resuming, deleting, and rerunning tasks through conversation, covering the complete lifecycle from task creation to decommissioning.

• Batch task operations: Supports creating or modifying multiple sync tasks in a single conversation, suitable for large-scale scenarios such as full-database migration and bulk scheduling policy adjustments.

Usage example

The following example demonstrates how to create a sync task through conversation:

1. Open the DI Agent conversation interface.

2. Enter your sync requirements. For example: "Sync the entire RDS order database to Hologres in real time, partitioned by day".

3. The Agent automatically detects the source schema, displays the list of tables to sync, and continues after you confirm.

4. The Agent automatically completes field mapping, resource group assignment, and scheduling policy configuration.

5. Preview the task configuration summary and deploy after confirming that everything is correct.

6. Check the task running status and verify that data is synced properly.

Capability description

DI Agent supports embedding large language model inference as an ETL operator in data pipelines to perform batch AI processing on structured data as well as unstructured data (images, videos, audio files, documents, and more) from source systems. Supported processing includes recognition, classification and tagging, transcription, translation, summarization, vectorization, and custom processing logic. Processing results are written directly to all data integration-supported destinations (such as MaxCompute, Hologres, or Data Lake Formation (DLF)), enabling various types of data to enter the enterprise data asset system in appropriate formats. Key capabilities include:

• Natural language processing logic definition: Describe your processing intent in natural language (for example, "Identify product categories in images and output tags"), and the Agent automatically generates the corresponding processing pipeline configuration.

• Three-stage pipeline orchestration: The Agent automatically orchestrates a Source → Transform (LLM inference) → Sink pipeline without requiring you to write scripts or set up inference services.

• Direct result writing to various destinations: For example, processing results are written directly to MaxCompute, Hologres, or DLF data lake without additional data movement.

• Data embedding: Supports vectorizing processed data and writing it to designated storage destinations for subsequent RAG use.

• Batch and periodic execution: Supports batch processing of millions of files and can be configured to run on a scheduled basis for continuous incremental processing of new data.

Usage example

The following example demonstrates how to process product images in OSS and write the results to MaxCompute:

1. Open the AI data processing entry point.

2. Enter your processing requirements. For example: "Read product images from an AWS S3 bucket → call a large model to identify classification tags → write results to a MySQL table".

3. The Agent automatically orchestrates a three-stage pipeline: Source (S3) → Transform (LLM inference) → Sink (MySQL).

4. ChatDB automatically identifies the target table and generates the corresponding SQL.

5. View the returned chart results (line charts, bar charts, and other formats are automatically matched).

6. If you need to vectorize data in the feature engineering database into Hologres for RAG use, you can define the embedding in natural language: "Continue with embedding → call a large model for vectorization → write results to a Hologres table".

7. The Agent automatically orchestrates a three-stage pipeline: Source (MySQL) → Transform (LLM vectorization) → Sink (Hologres).

8. ChatDB automatically identifies the target table and generates the corresponding SQL.

9. View the returned results.

Query data source schemas and business semantics

ChatDB allows you to explore database metadata and interpret business semantics through natural language. After you specify a target data source, the Agent automatically connects to it and progressively expands the full structure of databases, tables, and fields, including table names, field types, primary and foreign key relationships, index constraints, and other metadata details. The Agent also interprets the business semantics of schemas based on table names, field naming conventions, and data characteristics, helping you quickly understand each table's purpose, the business concepts that fields represent, and inter-table relationships.

Applicable scenarios: Quickly understanding the full data landscape when taking over an unfamiliar system, and helping non-technical team members understand data structures during cross-team collaboration.

Sample data preview for data sources

ChatDB allows you to preview sample data from any table using natural language. After you specify a target table, the Agent automatically fetches representative data samples with explanations, enabling you to visually confirm the content and format stored in the table without writing SQL.

Applicable scenarios: Quickly verifying data content, understanding actual field values, and helping non-technical team members understand data semantics during cross-team collaboration.

Statistical analysis with chart visualization

ChatDB supports natural-language-driven data analysis and visualization. Describe your analysis intent (for example, "show the sales trend by category over the last three months" or "compare the order proportion across regions"), and the Agent automatically locates the data source, generates the query logic, and presents the analysis results as line charts, bar charts, pie charts, or other visualizations—no SQL or specialized visualization tools required.

Applicable scenarios: Business users quickly validating business hypotheses, and technical teams instantly obtaining trend data during troubleshooting.

Schema diagnostics and optimization recommendations

ChatDB supports intelligent diagnostics for the schema of a specified data source. The Agent automatically scans the table schema design of the target database, audits it across dimensions such as naming conventions, field type selection, index coverage, primary and foreign key integrity, redundant fields, and partitioning strategies, identifies potential design flaws, and provides specific optimization recommendations and improvement directions.

Applicable scenarios: Design reviews for newly created schemas, technical debt assessments for existing systems, and root cause analysis for database performance issues.

Database performance troubleshooting and optimization

ChatDB supports database performance diagnostics and slow SQL root cause analysis. After you describe a performance issue or specify a target data source, the Agent automatically analyzes various factors causing slow queries, including full table scans, index invalidation, unreasonable JOIN order, deeply nested subqueries, implicit data type conversions, stale statistics, lock contention, and hot tables. It explains the root cause of each slow SQL statement based on the actual execution plan and provides targeted optimization recommendations such as index adjustments, query rewrites, and table schema refactoring.

Applicable scenarios: Emergency troubleshooting for sudden online performance incidents, and proactive remediation of potential slow queries during routine inspections.

Capability description

After you receive a sync task alert, describe the issue or paste the alert information in the Agent conversation interface. The Agent automatically aggregates multi-dimensional information including task logs, resource monitoring metrics, and source database status, identifies the root cause, and provides a remediation plan—reducing troubleshooting time from hours to minutes. Each diagnosis result is automatically archived as a knowledge base entry so that the next time a similar issue occurs, the existing solution is matched directly, eliminating the need to start from scratch when team members rotate. Key capabilities include:

• Multi-dimensional information aggregation: Automatically collects and correlates task logs, resource monitoring metrics, source database status, network bandwidth, and other multi-dimensional information, presenting the complete context in a single view.

• Root cause identification: Automatically analyzes the fault root cause (such as insufficient CUs, source-side table locks, network bandwidth bottlenecks, or configuration errors) based on the aggregated information, and provides a definitive conclusion.

• Remediation recommendations with one-click execution: After identifying the root cause, provides actionable remediation suggestions (such as scaling up resource groups, restarting instances, or adjusting concurrency), and supports one-click execution after confirmation.

• Automatic knowledge accumulation: Each diagnosis result is automatically archived as a knowledge base entry so that similar issues are directly matched to existing solutions in the future.

Usage example

1. After an alert is triggered, open the Agent conversation interface and describe the issue or paste the alert information.

2. The Agent automatically aggregates multi-dimensional information including task logs, resource monitoring metrics, and source-side status.

3. The Agent outputs a root cause analysis report (for example: insufficient CUs causing the consumption rate to fall below the production rate).

4. The Agent provides remediation recommendations (for example: scaling up the resource group CU quota or restarting failed instances), and executes the fix after confirmation.

Capability description

DI Agent supports proactive scanning of all sync pipelines on a configurable schedule, shifting the O&M model from reactive response to proactive governance. Inspection results are presented as structured reports, with high-risk tasks prioritized by impact scope and urgency. Each item includes specific recommended actions and an expected benefit assessment. Key capabilities include:

• Configurable inspection schedule: Supports four inspection frequencies—hourly, daily, weekly, and manual only—to accommodate different business requirements for health check timeliness.

• Full-pipeline proactive scanning: The Agent automatically scans the runtime status of all sync pipelines on schedule, proactively detecting potential risks without manual item-by-item checks.

• Structured daily health reports: Inspection results include a key metrics overview, an abnormal task list, and a risk level assessment, enabling the O&M team to grasp the overall health status within 3 minutes.

• Priority ranking with action recommendations: High-risk tasks are prioritized by impact scope and urgency, with each item accompanied by specific recommended actions and an expected benefit assessment.

Usage example

1. Go to the intelligent inspection configuration page and select an inspection schedule (hourly, daily, or weekly).

2. Configure the inspection scope (all pipelines or specific workspaces/resource groups).

3. The Agent automatically executes inspections on schedule and generates structured daily reports.

4. Review the key metrics overview and high-risk task list in the daily report, and address risk items by priority.