DocsICP FilingConsole
官方文档
首页

MaxCompute + API data source

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic describes how to add a table that uses MaxCompute as a data source.

Prerequisites

  • MaxCompute was formerly known as ODPS. For more information, see What is MaxCompute.

  • The Alibaba Cloud account that you use to log on to OpenSearch must have the required permissions on the MaxCompute table. These permissions include `describe`, `select`, `download`, and label permissions for fields.

The following statements show how to grant permissions:

-- Add a user.
add user ****@aliyun.com;

-- Grant the required table permissions to the user.
GRANT describe,select,download ON TABLE table_xxx TO USER ****@aliyun.com
GRANT describe,select,download ON TABLE table_xxx_done TO USER ****@aliyun.com

-- MaxCompute enables field-level permission verification. This may prevent you from accessing high-privilege fields when pulling data, which causes the index build to fail. In this case, you must grant field-level access permissions to the user.
-- Grant permissions on the entire project.
SET LABEL 3 to USER ****@aliyun.com

-- Grant permissions on a single table.
GRANT LABEL 3 ON TABLE table_xxx(col1, col2) TO ****@aliyun.com

  • The Vector Search Edition engine supports the following field types for MaxCompute tables: STRING, BOOLEAN, DOUBLE, BIGINT, and DATETIME.

Note

For more information about the CREATE TABLE statements and parameter settings, see CREATE TABLE statements for a MaxCompute data source.

Add a table

  1. On the Instance Details > Table Management page, click Add Table.

image.png

  1. Enter the basic information for the table and click Next.

image.png

Parameter description:

  • Table Name: A custom name for the table.

  • Number of Shards: Enter a positive integer up to 256. Shards improve the speed of full builds and the performance of individual queries. For some existing instances, the number of shards must be the same for all index tables. Alternatively, one index table can have one shard while the others have the same number of shards.

  • Data Update Resources: The number of resources used for data updates. By default, each index comes with a free quota of two 4-core 8 GB update resources. Resources that exceed the free quota are billed. For more information, see Billing overview for Vector Search Edition.

  • Scenario Template: Vector Search Edition provides three built-in templates: General, Vector - Image Search, and Vector - Text Semantics.

  1. Configure the data source for data synchronization. After the configuration is verified, click Next.

image.png

Parameter description:

  • Data Source Type: Select MaxCompute.

  • Project: The name of the destination MaxCompute project.

  • accesskeyId: The AccessKey ID of your Alibaba Cloud account or Resource Access Management (RAM) user.

  • accesskeySecret: The AccessKey secret that corresponds to the AccessKey ID.

  • Table: The name of the destination MaxCompute table.

  • Partition Key (partition): You must set a partition key for a MaxCompute data source. Example: ds=20170626.

  • Timestamp: Specifies the time range for retrieving incremental data from an API. The system can retrieve incremental API data from the last three days.

  • Auto Index Rebuild: Specifies whether to enable automatic index rebuilds. If enabled, the system automatically rebuilds the index for tables that reference this data source when a change is detected.

Note

  • If you enable automatic index rebuilds, you must create a done table. For instructions, see Auto index rebuild below.

  1. Configure the fields and click Next.

Note

  • The primary key field and the vector field are required. The primary key field must be of the INT or STRING type, and you must select the Primary Key checkbox for it. The vector field must be of the FLOAT type, and you must select the Vector Field checkbox for it.

  • By default, the vector field is a multi-value FLOAT type. The default multi-value separator is the ha3 separator `^]` (UTF-8 encoding: `\x1D`). You can also enter a custom multi-value separator.

  • If a field is missing or empty in the data, the system automatically populates it with a default value. The default value is 0 for numeric types and an empty string for the STRING type. You can also specify a custom default value.

  1. Configure the index schema and click Next.

image.png

This section describes how to configure the vector index:

  • The primary key field and vector field are required. The namespace field is optional and can be left empty.

  • You can only select from these three fixed fields. You cannot add new fields.

  • Vector Dimensions: Select the dimensions based on the vectors generated by your model.

  • Distance Metric: Select the metric based on the vectors generated by your model. The system supports two distance metrics: SquareEuclidean and InnerProduct.

  • Vector Index Algorithm: Select the algorithm based on the vectors generated by your model. The system supports the following vector index algorithms: Quantized Clustering, linear, and HNSW (Hierarchical Navigable Small World).

  • Real-time Index: Specifies whether to build a vector index in real time for incremental data from the API. The default value is true.

To configure other advanced settings, click to expand the section. For parameter descriptions, see General configurations for vector indexes.

image.png

  1. Review and confirm the settings. After you click Confirm Creation, the system automatically creates the table.

image.png

You can view the creation progress in the change history.

image.png

  1. When the table status changes to In Use, you can test queries on the Query Test page.

image.png

Auto index rebuild

Purpose of the done table: When you enable automatic index rebuilds for a data source, the Vector Search Edition instance automatically rebuilds the index based on changes to your done table.

For example, your MaxCompute data table is named `mytable` and is partitioned by `ds=20220113`. After you configure the data source and rebuild the index for the first time, a new partition that contains the full data is generated daily. If you want the Vector Search Edition instance to scan for the new partition and automatically rebuild the index to pull the new data, you must use the auto index rebuild feature with a done table.

Procedure:

  1. When you add the data source, enable Auto Index Rebuild.

image.png

  1. In MaxCompute, create a done table. If the data table is named `mytable` and the partition key is `ds`, the done table must be named `mytable_done` and its partition key must also be `ds`. The two tables are displayed in MaxCompute as follows:

odps:sql:xxx> show tables;
InstanceId: xxx  
SQL: .                                                                                              

ALIYUN$****@aliyun.com:mytable          # The full data source table
ALIYUN$****@aliyun.com:mytable_done     # The done table that controls automatic full rebuilds

The following figure shows the done table:

image.png

Statement to create the done table:

create table mytable_done (attribute string) partitioned by (ds string);

  1. After the data for the `mytable` partition `ds=20220114` is generated, configure the done table to trigger an index rebuild in the Vector Search Edition instance.

-- Add a partition.
alter table mytable_done add if not exists partition (ds="20220114");

-- Insert the signal data for the automatic full rebuild.
insert into table mytable_done partition (ds="20220114") select '{"swift_start_timestamp":1642003200}';

The final content of the done table is as follows:

odps:sql:xxx> select * from mytable_done where ds=20220114 limit 1;
InstanceId: xxx  
SQL: .                                                                                              
+-----------+----+
| attribute | ds |
+-----------+----+
| {"swift_start_timestamp":1642003200} | 20220114 |
+-----------+----+

After the signal data is inserted into the done table, the Vector Search Edition instance scans for the signal and automatically triggers an index rebuild.

Important

  • The done table must have at least one partition key, and the name of the partition key must be the same as the partition key of the data table. For example, if the data table's partition key is `ds`, the done table's partition key must also be `ds`.

  • The done table must have only one field of the STRING type, and the field name must be `attribute`.

  • The partition added to the done table must exist in the data table. For example, if the data table has partitions `ds="20220114"`, `ds="20220115"`, and `ds="20220116"`, the new partition added to the done table must be one of these.

  • When you insert data into the done table, the value of the `attribute` field must be a JSON string, such as {"swift_start_timestamp":1642003200}. The timestamp indicates the start offset for retrieving real-time incremental data.

Notes

  • MaxCompute does not support foreign tables. You must create an internal table.

  • The table configured for the MaxCompute data source must be a partitioned table.

  • The table generated in MaxCompute is used for full data builds. The API data source is used to push real-time data.

Previous：Add a tableNext：API data source