Configure and use a vector index-Simple Log Service(SLS)-阿里云帮助中心

Overview

Vector: In information retrieval and machine learning, vectors represent complex data, such as text, images, or audio, as points in a high-dimensional space. In this space, semantically similar data points are close to each other. Each data point is represented by a high-dimensional vector, where each dimension corresponds to a specific feature.
Embedding: The process of mapping multimodal data, such as text or images, into a high-dimensional vector space.
Distance metric: A metric, such as Euclidean distance or cosine similarity, used to quantify the similarity between vectors.
Vector index: A data structure, such as a tree or hash table, used to build an index for efficient searching. Common vector indexes include HNSW and IVF.

Benefits of the SLS vector index

Simple Log Service is a one-stop log data analysis platform that helps you collect, process, store, retrieve, and analyze log data. The rise of large language models (LLMs) has led to a sharp increase in the demand for natural language search. For example, you may need to audit, retrieve, and analyze user Q&A data and interaction logs between agents and LLMs. To address the need for semantic search in the field of large language models, Log Service has launched the vector index feature.

The vector index in Simple Log Service (SLS) offers the following benefits:

All-in-one embedding-to-index workflow: In traditional solutions, you must maintain separate systems for vector embedding and a vector database. This eliminates the need to manage a complex architecture; you only need to select an embedding model.
Low cost: The service is based on vector models hosted by SLS.
Simple search syntax (SPL): While the industry often uses JSON for search queries, SLS provides its own similarity and topk syntax. You can run queries directly in the search box without writing complex JSON or configuring numerous parameters.
Seamless pipeline integration: You can pass search results through a pipeline to a rerank model by using the Search & Process Language (SPL) or to SQL for statistical analysis.

Supported regions

The vector index feature is available only in the Chinese mainland.

Configure vector index

Prerequisites

A project and a Standard logstore are created, and logs are collected. For more information, see Manage projects, Create a basic logstore, and Data collection.
A full-text index and a field index have been created. For more information, see Create indexes.

Procedure

Log on to the Log Service console. In the Project list, click the target Project.
On the Log Storage > Logstores tab, click the target Logstore.
Click Query & Analysis > Properties. In the Query & Analysis panel, the vector index feature is disabled by default. The system automatically enables it after you select an embedding model.

Query notes

After you enable the vector index feature, it takes effect only on new data.
Vector indexes cannot be rebuilt.
Due to a 5-minute indexing delay, you can query new data only after this period.
If you get inaccurate query results due to caching, you can bypass the cache by adding a random condition, such as not abcdefg and, to the beginning of your query statement. The string abcdefg can be any random string.
The vector index feature depends on the statistics feature. Make sure to enable statistics on the right side of the panel. Because natural language text is often long, increase the maximum length of the statistical field. You can set it to up to 16 KB in the console or 64 KB by using an SDK.

Model parameters

sls-multilang-v3: The Qwen3 embedding model series is the latest proprietary model in the Qwen family, specifically designed for text embedding and ranking tasks.

Vector search syntax

Vector search in Simple Log Service SLS uses a retrieval syntax that returns data meeting specified conditions. The results are sorted by log time instead of similarity. This approach significantly reduces search costs without affecting the final accuracy. Because vector search results are generally reranked using a rerank model to improve accuracy, we recommend that you perform a secondary rerank on the search results.

The SLS vector retrieval syntax supports two filtering modes: distance-based filtering and top-k-based filtering.

Filter by distance

Syntax

similarity(field_name, "search_text") < distance

field_name: The field to search, such as msg.
search_text: The target text to search for, such as "tomato and egg".
distance: A value from 0 to 1. The smaller the value, the higher the similarity. A value of 0 indicates maximum similarity, and 1 indicates minimum similarity.

Example

The query similarity("input_semantic.topic","education") < 0.1 searches for data in the input_semantic.topic field that is related to "education" with a distance of less than 0.1.

Filter by top-k

Syntax

topk(field_name, "search_text") = $k

Note

The $k in topk specifies the number of top results to return per file. Results from multiple files are not merged. As a result, the total number of returned entries may be greater than $k. You need to rerank the search results.

field_name: The field to search, such as msg.
search_text: The target text to search for, such as "tomato and egg".
$k: An integer greater than or equal to 1. This parameter specifies the number of most similar results to return from a single file.

Example

The query topk("input_semantic.topic","education") = 2 searches for the two most similar data entries.

Multi-vector search

You can specify multiple vector retrieval conditions in a single query by using logical operators such as or, and, and not.

Note

Do not place the not operator at the beginning of a multi-vector search query.

Examples

Search for content related to "education" or "culture": topk(topic,"education") = 2 or topk(topic,"culture") = 2.
Search for content related to "education" and "Shanghai": topk(topic,"education") = 2 and topk(summary,"Shanghai") = 2.
Search for content related to "education" but not "Shanghai": topk(topic,"education") = 2 not topk(summary,"Shanghai") = 2.

Hybrid search

SLS supports hybrid search, which combines vector retrieval and keyword search. Both the vector retrieval and keyword search parts can be complex query statements. You can connect them with and, or, or not.

Note

A hybrid search query must consist of two parts: one for keyword search and one for vector retrieval.
Enclose both parts of the query in parentheses.

Examples

error and similarity("input_semantic.summary","query") <0.8
( (hello world) and topic:education) AND (topk(topic,"education") = 2 or topk(topic,"culture") = 2)

Vector clustering

After you enable a vector index, you can perform clustering based on vectors to classify long text into different categories. When you enable a vector index for a field, a hidden field is added in SQL/SPL. This hidden field is named by appending the __embedding__ suffix to the original field name. For example, if the original field is topic, the new hidden vector field is topic__embedding__. You can use this field, which has a data type of array(float), in SQL to perform clustering calculations.

Example

*| set session enable_remote_functions=true;
with t1 as (select  sentence1   as x, "sentence1__embedding__"  as embd from log    having embd is not null  limit 1000 ),
t2 as (select cluster(array_agg(embd),'dbscan','{
  "n_clusters": "${{cluster_num|5}}",
  "max_iter": "100",
  "tol": "0.0001",
  "metric":"cosine",
  "init_mode": "dbscan"
}') as cluster_res, array_agg(x) as xs from t1),
t3 as (select label,x from t2,unnest(cluster_res.assignments,xs) as t(label,x) )
select cast(label as bigint) as "Cluster ID",array_join(slice((array_agg(x)),1,3),concat(chr(10),'----',chr(10))) as "Topic",count(1) as pv from t3 group by label order by pv desc

Playground

Try out vector queries.
Try out vector clustering.