Programmatic Experiments
0. Overview
Programmatic experiments are A/B tests conducted through code. They are widely used for frontend optimization, policy optimization, and backend algorithm optimization. This topic describes how to create and manage programmatic experiments.
1. Feature path
You can access the product by clicking the A/B testing button on the home page or by clicking A/B testing in the application list.
The experiment list page appears, which displays experiments from the last month by default.
This product supports the creation of programmatic experiments. On the New Programmatic Experiment page, you can create an experiment by completing four steps: Enter basic information, Design targeting policy, Configure experiment groups, and Add experiment metrics.
2. Create experiment metrics
Before creating an experiment, first create a metric group under Metric Management.
(1) Enter basic information
Metric group type: Required. You can select event metrics, retention metrics, or funnel metrics.
Metric group name: Required. The name cannot exceed 100 characters and must be unique within the application.
Metric group description: Optional. The description cannot exceed 1000 characters.
(2) After you select the metric configuration, click New to create the group.
Basic metric configuration:
Event metrics: Supports queries for metrics such as total count, total users, audience count, conversion rate, in-group audience count, and sum by.
Retention metrics: Specify the initial behavior and subsequent behavior.
Funnel metrics: Specify the funnel steps and conversion cycle.
3. Create an experiment
In Experiment Management > Experiment List, click New Experiment to create a new experiment.
(1) Enter basic information
Experiment name: Required. The name cannot exceed 100 characters and must be unique within the application.
Experiment description: Optional. The description cannot exceed 1000 characters.
Experiment type: Required. Select client or server-side.
Experiment duration: Required. The default is 30 days. You can change it to a maximum of 365 days.
Owner: Required. The default is the current user.
(2) Design targeting policy
After you enter the basic information, click Next to design the targeting policy for the experiment.
Mutually exclusive: The default value is No. If traffic is not mutually exclusive, the experiment is an orthogonal experiment. If traffic is mutually exclusive, the experiment is a mutually exclusive experiment.
Traffic splitting entity: You can select entity ID, user ID, or device ID.
Traffic layer: For orthogonal experiments, you do not need to select a traffic layer. For mutually exclusive experiments, you must select a traffic layer. Traffic for experiment groups in the same traffic layer is mutually exclusive. You can select the traffic layer based on the traffic splitting entity.
Experiment traffic: The default value is 100%. For orthogonal experiments, you can adjust the value from 1% to 100%. For mutually exclusive experiments, the upper limit is the remaining traffic of the current traffic layer. You can use the traffic calculator to calculate the minimum required sample size for each group and the minimum required total sample size.
Select metric: Select a metric that can be calculated.
Metric mean: This field is automatically populated by calculating data from the previous 7 days.
Metric target value: Enter a value that is higher than the mean.
Statistical power (1-β): The default value is 80%.
Confidence level (1-α): The default value is 95%.
Global filter: You can set user audience rules. If you set a rule, only users who meet the rule are included in the experiment. For example, if you set the traffic to 50% and set an audience rule, the actual users included are those who match the audience rule after the 50% traffic split. The available settings are the same as the global filter and user segmentation for the current application analysis.
User segmentation: You can set the audience for the experiment. The experiment runs on data from the selected audience.
Consistent experience: This feature is enabled by default. When enabled, the version a user sees does not change with adjustments to traffic or audience conditions during the experiment. This ensures that a user stays in the same experiment group throughout the experiment. The user does not leave the experiment unless it is paused or stopped. If this feature is not enabled, the user's experiment group is determined in real-time for each session.
Traffic estimation: Click Start Estimation to calculate the total number of entity IDs covered during the experiment period based on the current configuration. This provides a reference for configuring the experiment.
After you complete the settings, click Next to configure the experiment groups.
(3) Configure experiment groups
In this step, you can configure the experiment group parameters.
Test user hit condition: This feature is disabled by default. When disabled, test users are added to a group by default without considering audience conditions. When enabled, test users must meet the audience conditions to be added to a group.
Experiment parameter settings:
A programmatic experiment requires at least one experiment parameter.
In the same experiment, after you add a parameter key, the corresponding key is added to each experiment group by default.
For programmatic experiments, the SDK can retrieve the value of the parameter for the current experiment hit through an interface.
Parameters support four data types: number, string, boolean, and JSON.
Parameter keys must be unique within the same experiment. If a parameter key has the same name as one in a different experiment, you can choose to skip the uniqueness check.
Control group/Experiment group settings:
Type: An experiment can have control groups and experiment groups. An experiment must have exactly one control group and can have multiple experiment groups. By default, one control group and one experiment group are created. You can click Add Experiment Group to add a new one.
Experiment group name: The default name is "Experiment Group" followed by an auto-incrementing number.
Test users: Users who are added to the group by default. You can manually enter the corresponding ID. The ID type must match the traffic splitting entity. You can also add users by scanning a QR code. Use a test device to scan the code to automatically retrieve the device's entity ID and add it.
Description: You can add a description for the experiment group.
Parameter settings: Each experiment group in the same experiment has the corresponding parameter key added by default. You can set different parameter values.
Operations: You can collapse or expand the experiment group information. You can also view sample code, delete an experiment group, or add an experiment group.
-
View sample code: Generates sample code based on the current experiment parameter configuration.
Delete experiment group: Deletes the current experiment group.
Add experiment group: Adds a new experiment group.
Traffic allocation: By default, traffic is allocated evenly. When even allocation is disabled, you can manually adjust the traffic allocation for each group. The total must be 100%.
(4) Add experiment metrics and complete experiment creation
In this step, you can configure the experiment metrics.
Select the experiment metrics to follow: If no experiment metrics are available, see "2. Create experiment metrics" to create them.
Core experiment metric configuration: Note that only retention rate and conversion rate metrics can be core metrics.
Experiment confidence level: The probability that a real performance difference exists between the experiment group and the control group. You can adjust it between 50% and 100%.
After you create the A/B test, you must integrate the A/B Test SDK. You can publish the experiment only after debugging is complete.
4. Experiment management
4.1 Experiment list
The experiment list has three tabs: All Experiments, My Favorites, and Drafts.
All Experiments: A list of all experiments in the current application, except for those with a Draft status.
This tab displays the experiment name, status (Testing, Running, Paused, or Finished), set traffic, number of users in groups, runtime, owner, and an Operations column (Start/Stop, Delete, and Show Details).
If you click Finish for a configured experiment, the experiment is created with a default status of Testing.
My Favorites: A list of experiments in the current application that the current user has added to their favorites.
This tab displays the experiment name, status (Testing, Running, Paused, or Finished), set traffic, number of users in groups, runtime, owner, and an Operations column (Start/Stop, Delete, and Show Details).
Drafts: A list of experiments created by the current user in the current application with a Draft status.
This tab displays the display name, progress, owner, and an Operations column (Delete and Edit).
4.2 Start an experiment
In the Operations column or on the details page, click Start to start the experiment. You can choose to publish it immediately or at a specified time. After the experiment starts, its status changes to Running.
4.3 View experiment details
In the experiment list, click Show Details in the Operations column for an experiment to go to its details page. On the details page, you can view the specific configuration information. In the upper-right corner of the details page, you can also add the experiment to or remove it from your favorites, or stop, start, pause, or edit it.
Favorites icon: Add the current experiment to or remove it from your favorites.
Stop: Stops the current experiment. The change takes effect immediately. No more regular users are included in the experiment. This operation only affects test users.
Start: Starts the current experiment. You can choose to publish it immediately or at a specified time. After the experiment starts, its status changes to Running.
Pause: Pauses the current experiment. The change takes effect immediately. No more regular users are included in the experiment. This operation only affects test users. During the pause, the experiment collects no data, which affects the overall data analysis. Segmented data analysis is recommended.
Edit: Edit the basic configuration and targeting policy of the current experiment.
4.4 View the experiment report
In the experiment list, click Show Details in the Operations column for an experiment, and then click Experiment Report to view the metric data for the experiment. You can add extra filter rules as needed. If no rules are set, the experiment data is viewed according to the rules in the experiment details.
Global filter: Set additional filter conditions.
User segmentation: You can set the audience for the experiment. The experiment runs on data from the selected audience.
5. Traffic layer management
You can manage traffic layers in A/B testing > Traffic Overview. Each traffic splitting entity has a default orthogonal experiment layer. Non-mutually exclusive experiments belong to this traffic layer by default.
5.1 Create a traffic layer
Click the New Traffic Layer button to go to the traffic layer configuration page.
Name: Required. The name of the traffic layer. The name cannot exceed 100 characters.
Description: Optional. A description of the traffic layer. The description cannot exceed 1000 characters.
Experiment type: Required. You can select client or server-side.
Traffic splitting entity: Required. You can select entity ID, user ID, or device ID.
Visibility scope: Optional. You can select fully public or partially public.
5.2 Traffic layer list
This list displays the traffic layer name, experiment type, number of experiments, traffic splitting entity, available traffic, description, and operations. The following operations are supported:
Details: View the details of the load balancing layer.
Modify: Edit the details of the load balancing layer.
Delete: Delete the traffic layer.