Retrieval-Augmented Generation (RAG) enhances a large language model's (LLM) ability to answer questions about private, domain-specific knowledge. It retrieves relevant information from an external knowledge base and provides it as context to the LLM with the user's query. EAS provides a scenario-based deployment option to quickly build and deploy a rag-based llm chatbot, letting you select your preferred large language model and vector database. This topic shows you how to deploy a rag-based llm chatbot service and verify its inference.
This topic applies to RAG versions 0.3.4 and 0.3.5. We recommend that you use the latest version.
Step 1: Deploy the RAG service
-
Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
On the Inference Service tab, click Deploy Service. In the Scenario-based Model Deployment section, click RAG-based Smart Dialogue Deployment.
On the RAG-based LLM Chatbot Deployment page, configure these parameters:
Version: The RAG service supports two deployment modes. Select the mode that fits your use case and resources.
LLM-Integrated Deployment: Deploys the RAG service and a large language model (such as Qwen) in a single EAS service instance. This mode is simple to configure and ideal for quick validation and prototyping.
LLM-Separated Deployment: Deploys only the RAG service, while the large language model runs as a separate service. This mode allows the RAG service to connect to different LLM services, such as other LLMs deployed on EAS or Alibaba Cloud Model Studio services. It facilitates resource reuse and independent scaling and is suitable for production environments or when an LLM service already exists.
Deployment: Allocate appropriate compute resources for the service.
LLM-Integrated Deployment: The system automatically recommends a resource specification based on the selected Model Category. A specification lower than the recommendation may prevent the service from starting.
LLM-Separated Deployment: The RAG service itself consumes minimal resources. We recommend a specification with at least an 8-core CPU and 16 GB of memory, such as
ecs.g6.2xlargeorecs.g6.4xlarge.
Vector database settings:
Vector Database Type: Select FAISS to build a local vector database for quick practice. For production environments, we recommend using a production-grade vector database. For details, see Use an Alibaba Cloud vector database.
OSS Path: Select an existing Object Storage Service (OSS) directory in the current region to store uploaded knowledge base files. If no storage paths are available, see Console Quick Start to create one.
NoteIf you deploy a service with a self-hosted fine-tuned model, ensure that the selected OSS storage path does not conflict with the model's storage path.
VPC: Configure EAS to access public or private resources to ensure that the RAG service can communicate properly with the LLM service, vector database, and other cloud services.
Access public services: To access a vector database, an LLM service (such as Alibaba Cloud Model Studio), or web search over the internet, you must configure public network access for the EAS service.
Access private services:
To access a vector database by using a private endpoint, the RAG service and the vector database must be in the same VPC.
To access an LLM service by using a private endpoint, the RAG service and the LLM service must be in the same VPC.
NoteHologres, Elasticsearch, Milvus, and RDS PostgreSQL support access over both private and public networks. We recommend using private network access.
The FAISS vector database does not require network access. OpenSearch supports access only over the public network.
After you configure the parameters, click Deploy. The service is successfully deployed when the Service Status changes to Running. The deployment typically takes about 5 minutes, but the duration may vary depending on the model size and other factors.
Step 2: Configure and debug in the WebUI
After the RAG service is successfully deployed, you can configure its features and debug its performance on the WebUI. On the Inference Service tab, find the deployed RAG service and open its details page. Click Web applications in the upper-right corner to open the WebUI.
The WebUI varies by image version. You can view the RAG service version number in the Environment Information section on the service's Overview tab. The following steps use version pai-rag:0.3.4 as an example.
2.1 Configure the large language model
On the System Settings tab, select the Model and Storage Configuration tab to configure the parameters.
Use model IDs to distinguish between different Model Configurations. Select NEW to add a new model configuration.
Integrated deployment
You typically do not need to modify the model configuration. However, if you select vLLM Accelerated Deployment, the Model Name must match the name set in the run command.
On the Service Configuration tab of the service details page, view the run command in the script field. The model name is the value that follows --served-model-name.
Decoupled deployment
URL and Key: The Alibaba Cloud Model Studio service URL is pre-filled. You only need to obtain an API key and enter it. Alternatively, you can use a model service deployed on EAS by providing the endpoint and token of the EAS service.
NoteCalls to Alibaba Cloud Model Studio models are billed separately. For more information, see Alibaba Cloud Model Studio Billing.
To connect to an LLM service by using a public endpoint, you must configure a VPC with public network access for the RAG service.
To connect to an LLM service by using a VPC endpoint, the RAG service and the LLM service must be in the same VPC.
Model name:
If you use an EAS-deployed model service with the vLLM inference engine, you must specify the model name. You can obtain the model name by calling the
/v1/modelsAPI. If the inference engine is SGLang or BladeLLM, set the model name todefault.If you are using other model services, refer to the specific service's parameter instructions.
2.2 Manage the knowledge base
On the Knowledge Base tab, you can create, update, or delete knowledge base configurations, upload files, set prompts for Q&A, and debug and apply retrieval parameters.
1. Knowledge base settings
On the Knowledge Base Settings tab, configure parameters for the knowledge base vector model, vector database, and more. We recommend that you use the default configurations.
Knowledge base: Select NEW to add a new, independent knowledge base. You can use knowledge base names to isolate data.
NoteThe knowledge base name corresponds to the folder name on the File Management tab. Upload files to the corresponding folder and select the relevant knowledge base by name during a conversation.
Knowledge base-vector model: Only the Hugging Face source is supported. The system automatically configures parameters such as the vector dimension based on the selected vector model.
Vector dimension: The output vector dimension. This setting directly affects model performance. After you select a vector model, the system automatically configures the vector dimension.
Vector batch size: The batch size for processing.
Vector database configuration: By default, the service uses the vector database configured during deployment. You can also switch to a different vector database. After configuring the relevant parameters, click Update Knowledge Base.
ImportantIf the vector database parameters were configured incorrectly during deployment, you must modify them here. You cannot modify them by updating the service.
2. File management
After you configure the knowledge base, you can upload files on the File Management tab. For more information about how to upload knowledge base files, see Manage RAG knowledge bases.
After a file is uploaded, the system automatically processes it, vectorizes its content, and stores it in the vector database. Supported file types include .html, .htm, .txt, .pdf, .pptx, .md, Excel (.xlsx or .xls), .jsonl, .jpeg, .jpg, .png, .csv, and Word (.docx), such as rag_chatbot_test_doc.txt.
For vector databases other than FAISS, uploading a file with the same name overwrites the existing file.
Files uploaded via API are not stored permanently. The storage duration depends on the configuration of your chosen vector database, such as Object Storage Service (OSS), Elasticsearch, or Hologres. We recommend that you review the relevant documentation to understand storage policies and ensure long-term data preservation.
3. Retrieval testing
On the Retrieval Testing tab, adjust the retrieval parameters to optimize Q&A performance.
Retrieval mode:
Vector search: The default mode, which retrieves content based on semantic similarity. It is suitable for most complex scenarios.
Keyword search: Retrieves content based on keyword retrieval algorithms such as BM25. It performs better in niche domains with limited corpora or in scenarios that require exact matches.
Hybrid search: Combines the advantages of the preceding modes. It uses the reciprocal rank fusion (RRF) algorithm to calculate a weighted sum of the document ranks from different modes, improving overall retrieval accuracy and efficiency.
Return top-K text results: Retrieves the top K most relevant text segments. The value can range from 0 to 100.
Similarity score threshold: Filters out results with a similarity score below this threshold.
Reranking type: If you select Model-based reranking, the results from the initial retrieval are reranked to improve the accuracy of the final answer.
NoteWhen you use reranking for the first time, the model may take a long time to load.
2.3 Web search
To use web search for Q&A testing, you must configure it on the Web Search tab of the Application page. For more information, see Best practices for integrating RAG with web search to build an AI-powered Q&A system.
2.4 Dialogue testing
On the Dialogue tab, simulate real user Q&A interactions to verify the RAG system's performance.
Dialogue model ID: Select the LLM that you configured in System Settings.
Knowledge base name: Select the target knowledge base to query.
Select a tool:
Large model: Uses the LLM to answer questions directly.
Web search: Combines real-time web search results to provide an answer. You must configure a search engine beforehand.
Query Knowledge Base: Combines knowledge base content to provide an answer.
Query Database: Directly retrieves and returns the top K similar results from the database by using a natural language to SQL (NL2SQL) query.
Enter a question to test the system's response.
You can enable streaming output and Show References to enhance the debugging experience.
You can adjust the temperature (0 to 1) to control the randomness of the generated content. A lower temperature results in more deterministic output, while a higher temperature produces more diverse and creative results.
Step 3: Call the API
The RAG service provides APIs for tasks like knowledge base management and conversation, allowing for easy integration into your applications. For details, see RAG API reference.
Costs and risks
Cost components
The costs for this solution primarily include:
EAS resource fees: The computing resources (such as vCPUs, memory, and GPUs) consumed by the RAG service are billed on an hourly basis. Billing for these resources stops after you stop the service.
Vector database fees: If you use Elasticsearch, Milvus, Hologres, OpenSearch, or RDS PostgreSQL, you are charged instance fees for the corresponding products.
Object Storage Service (OSS) fees: These fees are incurred for storing the original knowledge base files.
Large language model call fees: If you use commercial model services such as Alibaba Cloud Model Studio, API call fees apply.
Internet NAT Gateway fees: If an EAS service needs to access the internet, it incurs NAT Gateway fees.
Web search service fees: If you enable the web search feature (such as Bing or Alibaba Cloud OpenSearch), fees are incurred for the corresponding search services.
Stopping an EAS service only stops billing for EAS resources. To stop billing for other products, refer to the documentation for those products and follow the instructions to stop or delete the relevant instances.
Key risks and limitations
Conversation length limit: The amount of text processed in a single conversation is limited by the context window size (token limit) of the selected LLM service.
File overwriting: When you use a vector database other than FAISS, uploading a file with the same name overwrites existing data. Proceed with caution.
API parameter limitations: Only a subset of feature parameters can be configured through API calls. Other advanced configurations, such as most retrieval tuning parameters, must be configured in the WebUI.
Usage notes
This solution is limited by server resource size and the LLM service's default token limit, which restricts the conversation length. Its purpose is to help you experience the basic retrieval functions of the rag-based llm chatbot.
FAQ
Isolating knowledge base data
In RAG service version pai-rag:0.3.4, you can use the following method to isolate data for different knowledge bases:
On the Knowledge Base tab of the WebUI, go to the Knowledge Base Settings tab, configure the following parameters, and then click Add Knowledge Base.
Knowledge base name: Select NEW.
New knowledge base name: Enter a custom name for the new knowledge base, such as INDEX_1.
On the File Management tab of the Knowledge Base page, upload the knowledge base files to the corresponding knowledge base directory. For more information, see Upload a knowledge base file.
On the Dialogue tab, select the corresponding knowledge base name when you start a conversation. The system then uses files from the target knowledge base for Q&A, isolating data between different knowledge bases.
Ineffective API parameters
Currently, the PAI-RAG service only supports configuring the parameters listed in the API reference document through the API. Other parameters must be configured in the WebUI. For details, see Step 2: Configure and debug the service in the WebUI.