A/B Test
The A/B test feature lets you validate algorithm configurations on a small portion of traffic before applying them to all online traffic, preventing untested changes from negatively impacting your services. You can create tests for query analysis, rough sort, fine sort configurations.
Usage
1. Basic workflow for configuring an A/B test
We recommend enabling click data collection before you configure an A/B test to obtain more comprehensive metrics. If this is your first time using the feature, follow the steps below to configure and launch a test:
1. Start the test creation process:
In the OpenSearch console, navigate to Feature Extensions---->A/B Test in the left-side navigation pane. Click Create to start creating an A/B test.
2. Create a test group:
A test group allows you to run tests on query analysis, rough sort, fine sort, and category prediction. The test group name is for display purposes only and can be modified. The name cannot exceed 30 characters. The Test Application field displays the current application name. After you enter the information, click Next to proceed to the test configuration.
3. Create a test:
After a test group is created, click "Create Test" to add specific tests to the group. You can create a maximum of 20 tests in each test group, with a maximum of 10 tests running online simultaneously. By default, the page includes a baseline test named Online Performance (Baseline) with ID 0 that receives 100% of the traffic.
3.1 Test name: Define a custom name for the test, which cannot exceed 30 characters. After you enter the test name, the configuration area for Sort policy types and policies appears on the right. 3.2 Add a configuration:
You can configure tests for query analysis, rough sort, and fine sort by selecting existing rules in the configuration dialog box.
When you select a query analysis type and policy:
-
If you select "Custom", the available options are all the query analysis rules that you have created for the current application.
-
Selecting "Use Default Online Configuration" applies the existing online logic and excludes this configuration item from the test.
When you select a sort policy type and policy:
The same logic applies to sort policy types and policies. Because sort policies in the "Configuring" state can still be edited, which would affect online A/B test results, you can only select policies in the "Published" state.
3.3 Test traffic: The minimum traffic allocation for a test is 1%. For a single scenario, the total traffic allocated to all online tests within the same test group must be less than or equal to 100%.
4. Complete the test group creation:
After configuring the test, click "Next", and then click "Complete" to return to the A/B test homepage. A message indicating that the test group was created successfully is displayed, along with recommendations for the next steps. The newly configured test group is in the "Pending" state.
5. Start testing:
After the test group is created, find the group and click "Start Testing" in the Actions column. The status of the test group changes to "Testing".
6. Activate the A/B test:
After enabling A/B testing in the console and configuring a test, specify the abtest parameter in a query to apply the test to online queries. abtest has two parts: scene_tag and flow_divider. On the console Search Test page, enter the abtest value in the Parameters area to apply it.
Example request URL:
/v3/openapi/apps/160029126/search?query=query=default:'Shenzhen'&&config=start:0,hit:10,format:fulljson&abtest=scene_tag:test_1,flow_divider:123456
-
scene_tag: The name of the test group. Set this parameter to the name of a test group you created in the console. Traffic is then routed to the tests within that group.
-
flow_divider:Required. The system hashes this value to distribute query traffic among tests according to the configured traffic allocation. We recommend using a unique user ID. If unavailable, use a device ID or IP address.
Note:
-
When accessing OpenSearch through an SDK, if scene_tag and flow_divider contain no spaces or punctuation, call the corresponding operation without encoding these values. If they contain punctuation, encode them first. For details, see Practical example.
-
When accessing OpenSearch through the API, URL-encode scene_tag and flow_divider. The final abtest parameter format is abtest=urlencode(scene_tag:urlencode(\$scene),flow_divider:urlencode(\$value)), where urlencode is a URL-encoding function.
-
For more information, see the FAQ about A/B testing document.
2. Manage test groups and tests
Manage test groups
The A/B test homepage displays a list of test groups with the Test group name, Status, Creation time, Last modified time, and Actions columns. You can perform the following operations on test groups:
1. Start testing:
You can start test groups that are in the "Pending" or "Stopped" state.
2. Stop testing:
You can stop test groups that are in the "Testing" state.
3. Delete a test group:
You can delete any test group from the A/B test homepage.
Manage tests
After creating test groups and tests, the test group details page displays the test list. On the A/B test home page, in the test group's Actions column, click Details to open the details page. The test list displays test names, query analysis types and policies, sort strategy types and policies, traffic allocations, and actions. You can perform the following operations on tests:
-
Edit a test: You can modify the test name, configuration, and traffic allocation.
-
Delete a test: When a test is deleted, its configuration information is removed, and the test is no longer effective online.
-
Whitelist.
To assign a particular flow_divider to a specific test and view that test's search results, OpenSearch provides a whitelist. On the whitelist configuration page, enter a flow_divider value, click Add and Save.
3. A/B test group details
Behavioral data status
On the test group details page, the behavioral data status can be one of the following:
-
Not Activated: No behavioral data is being uploaded for the current application. (Click here for details)
-
Activated with No Data: Behavioral data collection is enabled for the current application, but no data has been received.
-
Data Abnormal: The quality check found that the current behavioral data is unreliable due to a high number of issues.
Test group status
On the test group details page, the test group status can be one of the following:
-
Pending: The test group is ready to start. This status is used whether the group has never been run or has been previously stopped.
-
Testing: Indicates that the test group has been started in the console. The number of days elapsed since the test group was started is displayed.
-
Stopped: Indicates that the test group is stopped. The accumulated test duration is the total number of days the test group was actively running, excluding periods when it was stopped.
Data statistics
Click Details in the A/B test group list, or click Statistical Reports > A/B Test Reports in the application menu to open the test group details page. On the Data Statistics page, view the A/B test report. Console data is available on day T+1: a test run today can be viewed tomorrow. The updated console combines core metric comparisons and detailed metric tables. Select a Metric Type from the drop-down list to view the corresponding metrics. Core metrics are displayed as line charts instead of raw values for a clearer view. By default, the latest day's data is displayed. Select multiple tests from the test drop-down list to compare them. Core metrics currently include search PV, search UV, no-result rate, search PV per user, impressions, query count, and queries per user.
-
Note: The A/B test report for a given day is available after 8:00 AM the following day. Even if you stop a test during the day, you can view data from before it stopped the next day.
Practical example
An e-commerce product uses OpenSearch for two types of search queries:
-
Type 1: Search traffic from end-users searching for products by keyword. The query format is:
query=config=format:fulljson&&query=default:'baby formula'&&sort=price
-
Type 2: Traffic from calls made by other internal services. The query format is:
query=config=format:fulljson&&query=cat_id:'1'|'2'|'3'&&sort=timestamp
For the first type of traffic, you want to split traffic based on end-user member IDs to compare the effectiveness of different sort expressions, category prediction models, or query analysis rules. The user configures the test as follows:
1.Create a test group and tests using the A/B Test feature in the console.When creating the test group, set its name to user_search.
2.Set the A/B test parameters in the query.Because the console test group is named user_search, set scene_tag:user_search,flow_divider:xxxx for the first type of query, where xxxx is the end user's member ID.
2.1 By using an SDK (The following example uses the Java SDK. The PHP SDK usage is similar.):
OpenSearch opensearch = new OpenSearch(accesskey, secret, host);
OpenSearchClient serviceClient = new OpenSearchClient(opensearch);
SearcherClient searcherClient = new SearcherClient(serviceClient);
searchParams = new SearchParams();
searchParams.setQueryString("default:'baby formula'");
searchParams.setFormat("json");
searchParams.addSort("price", "-");
searchParams.setAbtest(new Abtest().setSceneTag("user_search").setFlowDivider("Alex"));
aliyun-sdk-opensearch-3.4.1 (Java), opensearch-sdk-php-3.3.0 (PHP).
2.2 By using the API
i. query=config=format:fulljson&&query=default:'baby%20formula'&&sort=-price&abtest=scene_tag:user_search,flow_divider:Alex
Note: The values of the abtest subparameters scene_tag and flow_divider are both URL-encoded here.
ii. URL-encode the value of each parameter in the request (i.e., query, sort, abtest):
query=config%3dformat%3afulljson&&query%3ddefault%3a%27%e5%ae%9d%e5%ae%9d%e5%a5%b6%e7%b2%8
3. After completing these configurations, you can implement the A/B test for the first traffic scenario.
Business operations report
Interface
To open the A/B test statistics report, navigate to [Feature Extensions > A/B Test] and click "Report Statistics".
Alternatively, navigate directly to "Report Statistics > A/B test report".
The report page analyzes data across five dimensions: core metrics, traffic metrics, behavioral metrics, conversion metrics, and user analysis metrics. You can filter the data by date range and test.
Note: For descriptions of the A/B test report metrics, click here.