MaxCompute data source configuration

Set up an AccessKey

OpenSearch Industry Algorithm Edition uses an AccessKey and AccessKey Secret to download data from your MaxCompute tables. Before you configure the data source, you must have an AccessKey pair.

• You can use the AccessKey pair of your Alibaba Cloud account to access tables in any MaxCompute project under that account.

• For enhanced security, we recommend using a RAM user. Follow these steps to configure one.

1. Create a RAM user under your Alibaba Cloud account. For more information, see Create and authorize a RAM user.

2. In MaxCompute, add the RAM user as a member to your project.

In the Manage Members dialog box of your MaxCompute project, search for and select the target RAM user from the Search Members list on the left. Click the right arrow button to move the user to the Selected/To-be-added Members list on the right. Alternatively, you can enter one or more member names, separated by commas, in the Manually Add Member input box at the bottom, and then click OK.

3. After you add the member, go to the MaxCompute DataStudio page and run the list users; command to find the full username. For more information, see Connect using the console (Query Editor).

   OK
   OK
   OK
   OK
   OK
   OK
   OK
   OK
   A xxx
   RAM$o; xxx h_test@test.xxx xxx st_1
   2022-02-10 18:34:33 INFO ===xxx=====================================================
   2022-02-10 18:34:33 INFO Exit code of the Shell command 0
   2022-02-10 18:34:33 INFO -- Invocation of Shell command completed ---

4. Copy the full username and grant the required permissions. In the following commands, replace xxx with the username that you copied.

   -- 1. Grant permissions to list projects and create instances.
   grant CreateInstance,List on project zy_ts_test to user xxx;
   -- 2. Grant permissions to select, describe, and download table data.
   GRANT select,describe,download ON TABLE people_info TO USER xxx;
   -- 3. (Optional) Grant label-based permissions on the ODPS table.
   set label 2 to USER  xxx;
   -- Show the grants for the specified user.
   show grants for xxx;

After you grant the permissions, you can configure the MaxCompute data source in the OpenSearch console.

Configure MaxCompute data source

1. On the application configuration page, select Use Data Source, and then select the MaxCompute data source.

1. In the data source type list, click MaxCompute.

2. Click Connect to Database, enter your MaxCompute information, including the project name, and the AccessKey and AccessKey Secret of the authorized account.

   After you enter the details, click Connect.

3. After connecting, select the data tables to configure. The following example shows how to add three-level tables.

   In the Select Data Source dialog box, click the MaxCompute tab. In the Select Database area, choose the connected database. In the Select Data Table area, select the target tables, such as default.item_table and opensearch.bank_data. Move them to the Selected list, and then click OK.

The system automatically maps the fields from your source tables. You can adjust these mappings as needed. When finished, click Next.

The auto-mapped field configuration table includes the Field name, Primary key, Type, Join with another table, and Actions columns. For each field, you can set a data type (such as TEXT or INT), specify a primary key, configure joins with other tables, and add or remove fields.

4. Configure the index schema. Select an appropriate analyzer for your search requirements. For more information, see Index schema. When finished, click Next.

   In the example, two indexes are configured. The default index includes the country, source_buy_cnt, city, item_type, last_modify_time, channel, title, and subscribe_cnt fields, and it uses the Chinese - General analyzer. The id index includes the custom_pk_id field and uses the keyword analyzer.

5. Configure the data source. This includes configuring field mappings, selecting partition information, and choosing a data synchronization control mechanism.

   On the Data Source configuration page, click + Add a data source to add the MaxCompute data source.

   5.1. Configure field mappings: Click the edit icon in the Actions column. OpenSearch Industry Algorithm Edition provides several data source plug-ins for MaxCompute. To use a plug-in, click the plus sign (+) in the Content Conversion column when you configure the field mapping. The plug-in transforms the source field's content before synchronizing it to OpenSearch. If a plug-in fails due to a configuration error or connection issue, the source field is synchronized without transformation.

   5.2. Select partition information: Based on your MaxCompute data structure, you can specify which partitions to import. You can use a regular expression to define the partitions. For example, you can combine this with the scheduled reindexing feature to import data from a new partition each day.

   Regular expression: The equals sign (=), comma (,), semicolon (;), and double vertical bars (||) are reserved characters. To automatically import the full data from the previous day's partition, use an expression like ds=%Y%m%d || -1 days.

The following examples demonstrate how to use partition conditions for MaxCompute.

• 1. To specify multiple partition filter rules, separate them with a semicolon (;). For example, pt=1;pt=2 matches all partitions where the partition field pt is 1 or 2.

• 2. A filter rule can specify values for multiple partition fields, separated by commas (,). For example, pt1=1,pt2=2,pt3=3 matches partitions that satisfy all three conditions simultaneously. Note that functions like %Y%m%d || -1 days are supported for single-level partitions but not for multi-level partition filters.

Example: Assume a MaxCompute table has a pt partition that contains a ds sub-partition, as shown in the preceding figure.

• Multiple partitions: pt=1;pt=2 synchronizes all data from partitions pt=1 and pt=2.

• Multiple partition fields: pt=1,ds=1 synchronizes data from the ds=1 sub-partition within the pt=1 partition.

• Unsupported syntax: Expressions such as pt=1,ds=%Y%m%d || -1 days or pt=1;pt=%Y%m%d || -1 days are not currently supported.

• 3. The value of a partition field can be a wildcard character (*), which matches any value. In this case, you can omit the field from the filter rule.

• 4. The value of a partition field can be a regular expression. For example, pt=[0-9]* matches all partitions where the value of pt is a number.

• 5. The value of a partition field can use time-based matching. The format is partition_field=formatted_time_value||time_interval_expression. For example, ds=%Y%m%d || -1 days specifies that the partition field is ds, the format is 20150510, and data from the previous day is accessed.

• 5.1. The formatted time supports the standard date format specifiers listed in the following table.

• 5.2. The time interval expression supports the format +/- n unit, where unit can be week(s), day(s), hour(s), minute(s), second(s), or microsecond(s). A plus sign (+) indicates a time n units after the task is created, while a minus sign (-) indicates a time n units before.

• 5.3. By default, the system replaces time parameters in all filter rules with a +0 days offset. Therefore, be aware that field values used for filtering cannot contain these date format specifiers as literal strings. For example, if a task is created on a Wednesday, a filter pt=%abc matches partitions where the pt value is Wedbc, not the literal string %abc.

Available date format specifiers:

%d: Day of the month (01-31)  
%H: Hour in 24-hour format (00-23)     
%m: Month (01-12)  
%M: Minute (00-59)   
%S: Second (00-61)   
%y: Year as a two-digit number (00-99)  
%Y: Year as a four-digit number

5.3. Configure the data synchronization control mechanism:

Select Use DONE file to control when the system starts a full synchronization, which helps ensure data integrity. Before starting a full synchronization from MaxCompute, the system checks for the current day's DONE file. If the file is not found, the system waits. The default timeout is one hour.

• Download the odpscmd client from the official MaxCompute website. The package name is odps_clt_release_64.tar.gz.

• You must have the CreateResource permission on the target project.

• After installation, run the following command to upload the DONE file. Name the file using the format $prefix_%Y-%m-%d, where $prefix is a prefix that defaults to the table name, and %Y-%m-%d is the date of the reindex task. The minimum interval for scheduled tasks is one day.

  odpscmd -u accessid -p accesskey --project=<prj_name>-e "add file <done file>;"

• Instructions for using the MaxCompute client (odpscmd): For more information, see Connect using the local client (odpscmd).

• The DONE file must be in JSON format and contain a timestamp in milliseconds for the full data set. The system retains incremental data for a maximum of three days, so this timestamp cannot be more than three days in the past.

• This timestamp specifies the point in time from which to start applying incremental data. If you do not configure it, the system appends incremental data starting from the reindex task's start time.

• For example, consider a full data set generated at 09:00. MaxCompute finishes processing it at 10:00, and a scheduled reindex in OpenSearch is set for 10:30. To ensure data integrity, the DONE file must specify the 09:00 timestamp (in milliseconds). After the full data is synchronized, the system appends all incremental data generated after 09:00. If you omit this timestamp, the system appends incremental data starting from the reindex time (10:30), causing data generated between 09:00 and 10:30 to be lost. This setting is critical for preventing data loss. The timestamp is not required if you do not use incremental data.

• The data time setting functions on a similar principle to the DONE file's timestamp. It is used to track the starting point for incremental data pushed via API calls during a reindex.

{
"timestamp":"1234567890000"
}

Priority of the DONE file and data time:

1. The data time setting for a MaxCompute data source is required and takes priority over the DONE file.

2. If you create only a single application version, you must specify the data time. A DONE file cannot be used alone in this scenario.

3. To use scheduled reindexing, you must configure both the data time and a DONE file. The first version uses the data time. For all subsequent, automatically created versions, the DONE file takes priority.

OpenSearch Industry Algorithm Edition supports three Alibaba Cloud data sources: RDS, MaxCompute (formerly ODPS), and PolarDB. RDS and PolarDB support multi-table join configurations, allowing you to link a primary table and external tables within an application. With a MaxCompute data source, you are limited to configuring one database per application. OpenSearch Industry Algorithm Edition does not support self-managed databases or other external data sources.

To separate testing and production environments, create independent OpenSearch instances and configure them with different data sources to ensure environment isolation.

Usage notes