AI Data Preparation
AI Data Preparation is a data transmission service provided by AI-Native Database Service. It delivers unstructured and structured data to AI data infrastructure such as vector databases and lakehouse databases, building a high-quality data foundation for Retrieval-Augmented Generation (RAG) applications.
Overview
High-quality data is essential for model inference and knowledge retrieval in AI applications. AI Data Preparation provides a data pipeline from your data sources to AI data infrastructure. It helps you deliver multimodal enterprise data, including documents, images, and structured data, to vector databases or lakehouse databases. The service handles data cleansing, chunking, and vectorization to build reliable knowledge bases for RAG applications.
Scenarios
AI Data Preparation is designed for the following scenarios:
-
Enterprise knowledge base: Import unstructured data such as technical documents, product manuals, and FAQs into a vector database to build an enterprise-grade intelligent Q&A system that helps employees quickly find the information they need.
-
Intelligent customer service: Vectorize customer service knowledge bases and product documentation into a vector database. Combine the data with large language models (LLMs) to enable intelligent customer service responses and improve service quality and efficiency.
-
Content creation: Use industry data and research reports as a knowledge foundation to help LLMs generate high-quality content.
Features
AI Data Preparation provides the following core features:
-
Multimodal Data Preparation: Transmits structured data from data sources such as MySQL to AnalyticDB for PostgreSQL (ADB PG) for vectorization, providing data support for RAG applications.
-
RAGFlow knowledge base: Integrates RAGFlow knowledge base capabilities for intelligent chunking, vectorization, and retrieval of unstructured documents, enabling you to quickly build high-quality knowledge bases.
RAGFlow knowledge base
The RAGFlow knowledge base is a key component of AI Data Preparation. It provides end-to-end capabilities from document parsing to knowledge retrieval.
Supported vector databases
The RAGFlow knowledge base supports the following vector databases as knowledge storage backends:
|
Vector database |
Description |
|
AnalyticDB for PostgreSQL (ADB PG) |
A cloud-native data warehouse with high-performance vector retrieval capabilities, suitable for large-scale knowledge base scenarios. |
|
PolarSearch |
A full-text and vector retrieval engine built on PolarDB that provides hybrid retrieval capabilities. |
|
Lindorm |
A multi-model database that supports wide tables, time series, search, and vector retrieval. |
|
PolarDB for PostgreSQL |
A cloud-native relational database that supports vector storage and retrieval through extensions such as pgvector. |
Supported regions
The RAGFlow knowledge base is available in the following regions:
|
Area |
Region |
|
Chinese mainland |
China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen) |
|
Hong Kong (China) and overseas |
China (Hong Kong) |
Performance
|
Metric |
Value |
Notes |
|
Maximum single-file size |
500 MB |
Significantly higher than the 10 to 128 MB limit of typical open-source tools, thanks to deep optimization. |
|
File parsing throughput |
A single node parses a 20 MB file within 180 seconds |
You can scale out to up to 20 AI Data Preparation nodes to increase throughput. |
|
Retrieval QPS |
10 QPS (test scenario with 100 identical files) |
Actual QPS depends on multiple factors such as query embedding latency and vector retrieval performance. |
Enterprise-grade capabilities
|
Dimension |
Description |
|
Reliability |
24/7 service availability with a comprehensive L1/L2/L3 support structure and a dedicated support mechanism for enterprise customers. |
|
Security |
Supports OSS-hosted storage. Documents and parsed results are stored only in customer-specified spaces. File processing stays within the internal network. Fine-grained access control is available. |
|
Performance |
Provides monitoring of average response time and P95/P99 latency. Supports performance evaluation across different data scales. |
|
Ease of use |
Provides complete API access and supports billing details queries, including time ranges, usage, and costs. |
Free trial
|
Product |
Free trial policy |
|
RAGFlow knowledge base |
New users can enjoy a 15-day free trial on their first purchase. This offer applies to the Alibaba Cloud account and can be used only once. |
|
AnalyticDB for PostgreSQL vector database |
New users can enjoy a one-month free trial on their first purchase. |
Each free trial can be used only once. If you release an instance during the trial and purchase a new one, the free trial no longer applies. RAM users cannot directly use the free trial. The Alibaba Cloud account must create the instance and then grant access to RAM users.
Access AI Data Preparation
To access AI Data Preparation, perform the following steps:
-
Log on to the AI-Native Database Service console.
-
In the left-side navigation pane, choose AI Data Preparation > Multimodal Data Preparation.
-
On the Multimodal Data Preparation page, you can create a RAGFlow knowledge base instance or manage existing instances.
Before you use AI Data Preparation, make sure that you have activated AI-Native Database Service and created the required target database instance, such as an AnalyticDB for PostgreSQL instance.