Alibaba Cloud Elasticsearch (ES) integrates with the AI model capabilities provided by Alibaba Cloud Model Studio and the AI Search Open Platform to deliver efficient and flexible AI search solutions for diverse business scenarios. Common scenarios include semantic search, multi-modal retrieval, RAG knowledge bases, and search and recommendation Q&A assistants.
Background information
Whether in enterprise knowledge management or consumer-facing domains such as e-commerce, content, and entertainment, search technology effectiveness directly impacts business metrics such as click-through rate (CTR) and conversion rate. At its core, search aims to rapidly locate user needs from massive volumes of information. AI search goes further by prioritizing information accuracy and user satisfaction. Large Language Model (LLM) search holds immense potential but faces challenges including high performance requirements across scenarios, high model usage costs, and limited privacy and security controllability.
The Alibaba Cloud ES AI search feature builds on a robust ES foundation. It integrates diverse models and hybrid retrieval technology to enable a leap from traditional search to AI-powered semantic search. This feature leverages fine-grained data pre-processing, intelligent vectorization, multi-dimensional retrieval, and LLM-assisted generation to form a complete and efficient application framework for intelligent semantic search and RAG scenarios.
Alibaba Cloud AI Search Solution Overview
The Alibaba Cloud AI Search solution is an intelligent search solution designed for enterprise-level applications. It deeply integrates the capabilities of traditional search engines with the semantic understanding and generation capabilities of Large Language Models (LLMs) to create a new generation of search experience—accurate, answer-oriented, and user-friendly. The solution consists of two core building blocks: high-performance search engine and advanced AI model capabilities.
High-performance search engine (Elasticsearch)
Alibaba Cloud provides a high-performance, highly available, and scalable underlying search engine based on Enterprise Edition Elasticsearch and self-developed optimizations. It delivers the following key capabilities:
Millisecond-level response: The enhanced high-performance engine supports real-time indexing of massive data and low-latency queries.
Multi-modal retrieval: It supports hybrid retrieval of various content types, such as text, vectors, and structured data, significantly improving search precision and efficiency.
Semantic enhancement: It improves relevance sorting by combining vector retrieval (ANN), inverted index, and semantic matching.
Enterprise-grade features: These include access control, audit logs, high availability deployment, and rapid scaling.
NoteAlibaba Cloud ES is a managed service for Elasticsearch. It is compatible with the open-source ecosystem while deeply optimizing performance, stability, and cost efficiency.
Advanced AI model capabilities
AI Search provides model capabilities through two platforms:
Model Studio: This is an Alibaba Cloud platform for Large Language Model (LLM) development and application building. It provides the most comprehensive and advanced general-purpose model capabilities, including multi-modal models, text embedding models, and reranking models.
AI Search Open Platform Overview: This model service platform is specifically designed for search scenarios. It provides the atomic capabilities required for intelligent retrieval pipelines.
NoteThe Alibaba Cloud AI Search Open Platform is a model service platform specifically designed for search scenarios. It offers specific optimizations for text search, e-commerce images, and facial recognition, balancing both cost and effectiveness.
Driven by the dual engines of 'engine + model', the Alibaba Cloud AI Search solution helps enterprises move from 'keyword match' to the next-generation search paradigm of 'semantic understanding + intelligent generation', significantly improving user experience and business efficiency.
Alibaba Cloud ES AI Search Solution Overview
This topic describes the principles of two scenario-based solutions.
AI Semantic Search
The high-performance hybrid retrieval solution provided by Alibaba Cloud ES combines with the high-quality componentized services offered by the AI Search Open Platform in intelligent search scenarios to form a complete AI semantic search solution.
This process includes two core flows:
Data ingestion flow:
Use an advanced document segmentation model to perform fine-grained semantic segmentation, converting documents into semantic segment units.
Using a self-developed vectorization model, convert these segments into dense and sparse vectors, which greatly enhances information expressiveness and retrieval efficiency.
Vectors are used in the ES search engine to build efficient indexes, laying a solid foundation for subsequent information retrieval tasks. This reflects the effective transformation and utilization of complex data into structured knowledge.
Data query flow:
When processing search tasks, first convert the query information into dense and sparse vectors using a vectorization model. Then, perform hybrid retrieval in the ES index to recall the top N document contents.
RAG-based Knowledge Base Q&A
Retrieval-Augmented Generation (RAG) is an artificial intelligence technology that combines information retrieval techniques with generative Large Language Models (LLMs). It aims to improve the relevance, accuracy, and diversity of content generated by models. The following figure shows the end-to-end RAG deployment solution.
This process includes two core flows:
Data ingestion flow:
Use a document parsing model to transform various unstructured document contents into structured data.
Use an advanced document segmentation model to perform fine-grained semantic segmentation, converting documents into semantic segment units.
Using a self-developed vectorization model, convert these segment units into dense and sparse vectors. Ingest them into the ES index to serve as an online Q&A knowledge base, providing additional information input for subsequent Large Language Model (LLM) information retrieval tasks.
Data query flow:
When processing search tasks, first analyze the query (search content). Then, retrieve the most relevant segments from the knowledge base. Finally, input the retrieved information along with the query into the Large Language Model (LLM). The query acts as a prompt or context to guide the LLM in generating more accurate and comprehensive answers.
References
Scenario-based solutions:
AI semantic search scenarios:
RAG scenarios: