DocsICP FilingConsole
Documentation
Home Page DashVector Developer Reference Appendix DashText Quick start

Quick start

Updated at:
Copy as MD
Product
My Favorites

DashText is a sparse vector encoder recommended for DashVector. DashText can convert the original text into a sparse vector by using the Best Match 25 (BM25) algorithm. This greatly simplifies the use of the keyword-aware semantic search feature of DashVector.

Note

  1. You need to replace YOUR_API_KEY with your API key and YOUR_CLUSTER_ENDPOINT with the endpoint of your cluster in the sample code for the code to run properly.

  2. This topic describes how to use sparse vectors in a search. For simplicity, the number of dense vector dimensions is set to 4. In actual scenarios, set it as needed. For more information, see Vector introduction.

Step 1. Create a collection that supports sparse vectors

import dashvector

client = dashvector.Client(api_key='YOUR_API_KEY', endpoint='YOUR_CLUSTER_ENDPOINT')
assert client

ret = client.create('hybrid_collection', dimension=4, metric='dotproduct')
assert ret

collection = client.get('hybrid_collection')
assert collection
import com.aliyun.dashvector.DashVectorClient;
import com.aliyun.dashvector.DashVectorCollection;
import com.aliyun.dashvector.models.requests.CreateCollectionRequest;
import com.aliyun.dashvector.models.responses.Response;
import com.aliyun.dashvector.proto.CollectionInfo;

DashVectorClient client = 
  new DashVectorClient("YOUR_API_KEY", "YOUR_CLUSTER_ENDPOINT");

CreateCollectionRequest request = CreateCollectionRequest.builder()
            .name("hybrid_collection")
            .dimension(4)
            .metric(CollectionInfo.Metric.dotproduct)
            .dataType(CollectionInfo.DataType.FLOAT)
            .build();
      
Response<Void> response = client.create(request);
System.out.println(response);

DashVectorCollection collection = client.get("hybrid_collection");
Important

Only collections that use the dot product metric (metric='dotproduct') support sparse vectors.

Step 2. Create a sparse vector encoder

Use the built-in encoder

from dashtext import SparseVectorEncoder

encoder = SparseVectorEncoder.default("en")
  import com.aliyun.dashtext.encoder.SparseVectorEncoder;

SparseVectorEncoder encoder = SparseVectorEncoder.getDefaultInstance("en");
Note

The built-in encoder is trained on the Chinese Wikipedia corpus, and Jieba is used for Chinese text segmentation.

Create an encoder based on your own corpus

from dashtext import SparseVectorEncoder

encoder = SparseVectorEncoder()

# Your own corpus.
corpus = [
    "DashVector vector retrieval service is based on Alibaba Cloud's self-developed efficient vector engine Proxima kernel, providing cloud native and fully managed vector retrieval services with horizontal scalability",
    "DashVector leverages its powerful vector management, vector querying, and other diverse capabilities through a concise and easy-to-use SDK/API interface, making it easy for upper level AI applications to integrate quickly",
    "Thus, it provides the required efficient vector retrieval capability for various application scenarios, including large model ecology, multimodal AI search, and molecular structure analysis",
    "A simple, flexible, and out of the box SDK that enables vector management with minimal code",
    "Self developed vector similarity comparison algorithm for fast, efficient, and stable service",
    "Schema free design, using Schema to implement combined filtering queries under any conditions"
]

# Train the encoder by using your own corpus.
encoder.train(corpus)
import com.aliyun.dashtext.encoder.SparseVectorEncoder;
import java.util.*;

SparseVectorEncoder encoder = new SparseVectorEncoder();

// Your own corpus.
List<String> corpus = Arrays.asList(
  "DashVector vector retrieval service is based on Alibaba Cloud's self-developed efficient vector engine Proxima kernel, providing cloud native and fully managed vector retrieval services with horizontal scalability",
    "DashVector leverages its powerful vector management, vector querying, and other diverse capabilities through a concise and easy-to-use SDK/API interface, making it easy for upper level AI applications to integrate quickly",
    "Thus, it provides the required efficient vector retrieval capability for various application scenarios, including large model ecology, multimodal AI search, and molecular structure analysis",
    "A simple, flexible, and out of the box SDK that enables vector management with minimal code",
    "Self developed vector similarity comparison algorithm for fast, efficient, and stable service",
    "Schema free design, using Schema to implement combined filtering queries under any conditions"
);

// Train the encoder by using your own corpus.
encoder.train(corpus);
Note

  1. The built-in encoder is ready for use without the need to be trained on the original corpus, making it user-friendlier and more excellent in generalization. However, the built-in encoder has a low accuracy if the original corpus contains many terms.

  2. To create an encoder based on your own corpus, you must train the encoder on the full corpus in advance. This way, the encoder provides a higher accuracy. For more information, see Advanced use.

  3. You need to select the encoder based on your business requirements. We recommend that you create an encoder based on your own corpus if your business involves a large number of terms specific to a certain field.

Step 3. Insert a document containing a sparse vector

from dashvector import Doc

document = "DashVector, a vector retrieval service, is based on Alibaba Cloud's self-developed efficient vector engine Proxima kernel, providing cloud native and fully managed vector retrieval services with horizontal scalability."
doc_sparse_vector = encoder.encode_documents(document)

print(doc_sparse_vector)
# Output based on the built-in encoder: 
# {639348534: 0.3751617076326003, 765408119: 0.3751617076326003, 782729609: 0.3751617076326003, 920154565: 0.5456255879586078, 979706322: 0.3751617076326003, 1217996814: 0.3751617076326003, 1760434515: 0.3751617076326003, 1955147705: 0.6430155210643016, 2089228990: 0.3751617076326003, 2141666983: 0.3751617076326003, 2206567656: 0.3751617076326003, 2438044443: 0.3751617076326003, 2724419853: 0.3751617076326003, 3056142163: 0.3751617076326003, 3523129663: 0.3751617076326003, 3640852848: 0.3751617076326003, 3945092264: 0.3751617076326003, 3987047702: 0.3751617076326003, 4109964284: 0.3751617076326003}

collection.insert(Doc(
    id='A',
    vector=[0.1, 0.2, 0.3, 0.4],
    sparse_vector=doc_sparse_vector
))
String document = "DashVector, a vector retrieval service, is based on Alibaba Cloud's self-developed efficient vector engine Proxima kernel, providing cloud native and fully managed vector retrieval services with horizontal scalability.";
Map<Long, Float> sparseVector = encoder.encodeDocuments(document);

System.out.println(sparseVector);
// Output based on the built-in encoder: 
// {2141666983=0.37735847, 3523129663=0.37735847, 4109964284=0.37735847, 765408119=0.37735847, 2438044443=0.37735847, 3945092264=0.37735847, 2206567656=0.37735847, 639348534=0.37735847, 2724419853=0.37735847, 782729609=0.37735847, 1955147705=0.6451613, 3640852848=0.37735847, 3987047702=0.37735847, 3102935493=0.6451613, 1217996814=0.37735847, 979706322=0.37735847, 2089228990=0.37735847, 3056142163=0.37735847, 920154565=0.5479452, 1760434515=0.37735847}

Vector vector = Vector.builder().value(Arrays.asList(0.1f, 0.2f, 0.3f, 0.4f)).build();

// Build a Doc object containing a sparse vector.
Doc doc = Doc.builder()
  .id("28")
  .sparseVector(sparseVector)
  .vector(vector)
  .build();

// Insert the document containing a sparse vector.
Response<Void> response = collection.insert(InsertDocRequest.builder().doc(doc).build());

Step 4. Perform a keyword-aware semantic search

query = "What is vector retrieval service?"
sparse_vector = encoder.encode_queries(query)

print(sparse_vector)
# Output based on the built-in encoder: 
# {920154565: 0.22585031534902333, 1955147705: -0.06326246188143803, 2794577624: 0.5019280291881163, 3640852848: 0.3354841173442982}

docs = collection.query(
    vector=[0.1, 0.1, 0.1, 0.1],
    sparse_vector=sparse_vector
)
String query = "What is a vector retrieval service?";

Map<Long, Float> sparseVector = encoder.encodeQueries(query);

System.out.println(sparseVector);
// Output from the built-in encoder: 
// {1169440797: 0.2947158712590364, 2045788977: 0.7052841287409635}

Vector vector = Vector.builder().value(Arrays.asList(0.1f, 0.2f, 0.3f, 0.4f)).build();
      	
// Build a QueryDocRequest. 
QueryDocRequest request = QueryDocRequest.builder()
  .vector(vector)
  .sparseVector(sparseVector)
  .topk(100)
  .includeVector(true)
  .build();

Response<List<Doc>> response = collection.query(request);
System.out.println(response);

Step 5. Perform a weight-factored keyword-aware semantic vector search

from dashtext import combine_dense_and_sparse

query = "What is vector retrieval service?"
sparse_vector = encoder.encode_queries(query)

# Specify the weight factor.
alpha = 0.7
dense_vector = [0.1, 0.1, 0.1, 0.1]
scaled_dense_vector, scaled_sparse_vector = combine_dense_and_sparse(dense_vector, sparse_vector, alpha)

docs = collection.query(
    vector=scaled_dense_vector,
    sparse_vector=scaled_sparse_vector
)
String query = "What is a vector retrieval service?";

Map<Long, Float> sparseVector = encoder.encodeQueries(query);

System.out.println(sparse_vector);
// Output from the built-in encoder: 
// {1169440797: 0.2947158712590364, 2045788977: 0.7052841287409635}

Vector denseVector = Vector.builder().value(Arrays.asList(0.1f, 0.2f, 0.3f, 0.4f)).build();

// Scale the dense and sparse vectors using the alpha weight factor.
float alpha = 0.1;
sparse_vector.forEach((key, value) -> sparse_vector.put(key, value * (1 - alpha)));
denseVector = Vector.builder().value(
            denseVector.getValue().stream().map(number -> number.floatValue() * alpha).collect(Collectors.toList())
    ).build();

// Build a QueryDocRequest. 
QueryDocRequest request = QueryDocRequest.builder()
  .vector(denseVector)
  .sparseVector(sparseVector)
  .topk(100)
  .includeVector(true)
  .build();

Response<List<Doc>> response = collection.query(request);
System.out.println(response);
Note

The alpha parameter controls the weighted distances of dense and sparse vectors. If it is set to 0.0, only sparse vectors are used for distance measurement. If set to 1.0, only dense vectors are used for distance measurement.

API reference

For more information about DashText API, see the following:

Previous：Install DashText SDKNext：Advanced use