MaxCompute data source-OpenSearch(Open Search)-阿里云帮助中心

Prerequisites

You are familiar with MaxCompute, formerly known as ODPS. To learn more, click here.
Your OpenSearch account must have the required permissions on the MaxCompute table, including the describe, select, download, and label permissions on fields.

Use the following statements to grant permissions:

-- Add an account.
add user ****@aliyun.com;

-- Grant the required table permissions to the account.
GRANT describe,select,download ON TABLE table_xxx TO USER ****@aliyun.com;
GRANT describe,select,download ON TABLE table_xxx_done TO USER ****@aliyun.com;

-- If field-level permission checks are enabled in MaxCompute, you may be unable to access fields with high-level permissions when pulling data, which prevents index creation. In this case, you must grant the account the required field-level permissions.
-- Grant permissions on the entire project.
SET LABEL 3 to USER ****@aliyun.com;

-- Grant permissions on a single table.
GRANT LABEL 3 ON TABLE table_xxx(col1, col2) TO ****@aliyun.com;

OpenSearch Vector Search Edition supports the following MaxCompute table field types: STRING, BOOLEAN, DOUBLE, BIGINT, and DATETIME.

Note

For detailed information about table creation statements and parameter configuration, see Table creation statements for a MaxCompute data source.

Configure a MaxCompute data source

Log on to the OpenSearch console. In the upper-left corner, switch to OpenSearch Vector Search Edition. On the Instance Management page, find your instance and click Manage in the Actions column.

In the navigation pane on the left, choose Configuration Center > Data Source Configuration. Click Add Data Source. Select MaxCompute as the data source type and enter information such as Data Source Name, Project, AccessKey ID, AccessKey Secret, Table, Partition Key, and Automatic Reindexing.

After you configure the parameters, click Check. Once the configuration is validated, the OK button is enabled.
After you add a data source, you must configure an index schema and add an index table. For more information, see Add an index table.
After the index table is configured, you must update the configuration and perform reindexing to apply the changes to your online cluster. For more information, see Configuration update.

Note:

Data Source Name: The name of the data source. The name must follow the format: InstanceName_CustomName.
Project, AccessKey ID, AccessKey Secret, Table, and Partition Key: The parameters required to access the MaxCompute data source.
Automatic Reindexing: Specifies whether to enable automatic reindexing. If enabled, the system automatically triggers reindexing for index tables that reference this data source each time a change in the data source is detected.
If you enable automatic reindexing, you must create a done table. For more information, see the Automatic reindexing section below.

Automatic reindexing

Purpose of the done table: When you enable automatic reindexing for a data source, OpenSearch Vector Search Edition monitors the done table for changes to automatically trigger reindexing.

Example: Your MaxCompute table is named mytable and it is partitioned by ds=20220113. After the initial reindexing, a new partition containing full data is generated daily. You need OpenSearch Vector Search Edition to scan for the new partition and automatically trigger reindexing to pull the new data. To achieve this, you must use the automatic reindexing feature with a done table.

Procedure:

When adding the data source, enable automatic reindexing.
In MaxCompute, create a done table. If your data table is named mytable and its partition key is ds, the done table must be named mytable_done with a partition key also named ds. The two tables appear in MaxCompute as follows:
```
odps:sql:xxx> show tables;
InstanceId: xxx  
SQL: .                                                                                              
ALIYUN$****@aliyun.com:mytable          # The table that contains the full data.
ALIYUN$****@aliyun.com:mytable_done     # The done table that controls automatic full data ingestion.
```
The following describes the structure of the done table:

Example data for a done table: The attribute column has the value {"swift_start_timestamp":1603096083}, and the ds partition has the value 20180724.

Use the following statement to create the done table:
```
create table mytable_done (attribute string) partitioned by (ds string);
```
After data generation for the ds=20220114 partition of the mytable table is complete, you must update the done table to trigger reindexing in OpenSearch Vector Search Edition.
```
-- Add the partition.
alter table mytable_done add if not exists partition (ds="20220114");

-- Insert the signal data to trigger automatic full ingestion.
insert into table mytable_done partition (ds="20220114") select '{"swift_start_timestamp":1642003200}';
```
The done table now contains the following content:
```
odps:sql:xxx> select * from mytable_done where ds=20220114 limit 1;
InstanceId: xxx  
SQL: .                                                                                              
+-----------+----+
| attribute | ds |
+-----------+----+
| {"swift_start_timestamp":1642003200} | 20220114 |
+-----------+----+
```
Once the signal data is inserted into the done table, OpenSearch Vector Search Edition detects this signal and automatically triggers reindexing.
Important
- The done table must have at least one partition key, and the name of this key must be identical to that of the data table's partition key. For example, if the data table's partition key is ds, the done table's partition key must also be ds.
- The done table must have only one field of the STRING type, and the field name must be attribute.
- The partition you add to the done table must exist in the data table. For example, if the data table has partitions ds="20220114", ds="20220115", and ds="20220116", the new partition added to the done table must be one of these.
- When you insert data into the done table, the value for the attribute field must be a JSON string, such as {"swift_start_timestamp":1642003200}. This timestamp indicates the start offset for real-time incremental synchronization.

Modify a MaxCompute data source

On the Data Source Configuration page, find the data source to modify and click Modify in the Actions column.
On the Edit Data Source page, you can modify information such as Project, AccessKey ID, AccessKey Secret, Table, and Partition Key.

After making changes, click Check. After the configuration is validated, click OK to save the changes.
After modifying the data source, you must update the configuration and perform reindexing to apply the changes to your online cluster. For more information, see Configuration update.

Note
Set the partition granularity to the hour, for example, yyyymmddhh (such as 2022011314). This allows you to run multiple full reindexing jobs in a single day.

Delete a MaxCompute data source

Go to Data Source Configuration and click Delete.
When you click Delete, the system checks if any index table references the data source.

If the data source is not referenced by any index table, click OK to delete the data source. You must then update the configuration and perform reindexing for the deletion to take effect.
If an index table references the data source, the error message Data source is referenced by an index table is displayed (error code: DataSourceInUse). You must delete the index table or remove the association before you can delete the data source.

Before you can delete the data source, you must first delete the index table that references it. See Delete an index table.

Usage notes

You cannot change the Data Source Name when editing a data source.
MaxCompute does not support external tables. You must create an internal table.
The table specified for a MaxCompute data source must be a partitioned table.
Use MaxCompute tables for full data ingestion. For real-time incremental updates, use an API-based data source.