Configure sort policies

Search engines have high requirements for retrieval performance. To meet these requirements, the system provides a two-stage sorting process: basic sort and business sort, also known as rough sort and fine sort. Basic sort is a preliminary selection that quickly identifies high-quality documents from the retrieval results. The top N results are then scored in detail by fine sort, and the best results are returned to the user. Therefore, basic sort has a large impact on performance, whereas business sort has a large impact on the final sorting effect. For this reason, basic sort should be as simple and effective as possible and extract only the key factors used in business sort. Both basic sort and business sort are currently configured by using sort expressions.

Sort expressions (Ranking Formula) allow you to customize how the search results of an application are sorted. You specify an expression in a query request to sort the results. Sort expressions support basic operations (arithmetic, relational, logical, bitwise, and conditional operations), mathematical functions, sort features (features), and more. OpenSearch provides expression templates for several typical applications, such as forums and news. You can select an expression template that suits the characteristics of your data and modify it to create your own expression.

Before you perform relevance sorting (business sort), you need to understand the system sort policy: After documents that meet the conditions are found by using clauses such as the query clause, the sorting stage begins. For more information, see sort clause. If no sort clause is specified, or if RANK is explicitly specified in the sort clause, the documents enter the relevance scoring stage.

How you design the expressions for basic sort and business sort depends mainly on the requirements of your actual search scenarios. The best practices for features include an article named "Relevance in practice", which describes in detail how to think about and design sort factors in several typical scenarios. You can refer to it.

Basic operations

Operation	Operator	Description
Unary operation	-	The negative sign, which returns the negative of the value of an expression. Examples: -1 and -max(width).
Arithmetic operation	+, -, *, /	Example: width / 10
Relational operation	==,!= ,>, <, >=, <=	Example: width>=400
Logical operation	and ,or,!	Examples: width>=400 and height >= 300, !(a > 1 and b < 2)
Bitwise operation	&, |,^	Example: 3 & (price ^ pubtime) + (price | pubtime)
Conditional operation	if(cond, thenValue, elseValue)	If the value of cond is not 0, the value of the if expression is thenValue. Otherwise, the value is elseValue. For example, the value of if(2, 3, 5) is 3, and the value of if(0, 3, 5) is 5. (Note: String field types, such as LITERAL and TEXT, are not supported. The value range is the value range of int32.)
in operation	i in (value1, value2, …, valuen)	If the value of i appears in the set (value1, value2, …, valuen), the value of the expression is 1. Otherwise, the value is 0. For example, the value of 2 in (2, 4, 6) is 1, and the value of 3 in (2, 4, 6) is 0.

Mathematical functions

Function	Description
max(a, b)	Returns the larger value of a and b.
min(a, b)	Returns the smaller value of a and b.
ln(a)	Returns the natural logarithm of a.
log2(a)	Returns the base-2 logarithm of a.
log10(a)	Returns the base-10 logarithm of a.
sin(a)	The sine function.
cos(a)	The cosine function.
tan(a)	The tangent function.
asin(a)	The arcsine function.
acos(a)	The arccosine function.
atan(a)	The arctangent function.
ceil(a)	Rounds a up to the nearest integer. For example, ceil(4.2) returns 5.
floor(a)	Rounds a down to the nearest integer. For example, floor(4.6) returns 4.
sqrt(a)	Returns the square root of a. For example, sqrt(4) returns 2.
pow(a,b)	Returns a raised to the power of b. For example, pow(2, 3) returns 8.
now()	Returns the current time, calculated from the Epoch (00:00:00 UTC, January 1, 1970). Unit: seconds.
random()	Returns a random value in the range of [0, 1].

Built-in feature functions

OpenSearch provides a variety of built-in feature functions, such as LBS functions, text functions, and timeliness functions. You can use these functions in sort expressions and combine them to achieve powerful relevance sorting.

Cava plug-ins

Cava is an efficient programming language developed by the OpenSearch engine team based on LLVM. Its syntax is similar to that of Java, and its performance is comparable to that of C++. Cava is an object-oriented programming language that supports just-in-time (JIT) compilation and various security checks to make programs more robust. You can use Cava and the Cava libraries provided by OpenSearch to customize your own sort plug-ins in OpenSearch. Compared with the expressions supported by OpenSearch, sort plug-ins implemented in Cava provide the following benefits:

• Stronger customization capabilities: Cava provides richer syntax features than expressions, such as for loops, function definitions, and class definitions. You can use them to implement your own business requirements.

• Easier maintenance: Sort plug-ins implemented in Cava are more readable and easier to maintain than expressions.

• Easier adoption: The syntax of Cava is similar to that of Java. Developers who are familiar with Java can easily use Cava for development at a low learning cost.

Tip: Cava plug-ins can be configured only for exclusive applications.

Procedure demonstration

The following example uses the configuration of a text relevance sort function to demonstrate how to configure basic sort and business sort:

1. Create a basic sort policy. Go to the OpenSearch console. Choose Sort Configuration > Policy Management and click Create. Enter a policy name, set Scope to Basic Sort, set Type to Expression, and then click Next.

Select static_bm25 as the scoring feature and set the weight to 10. (Note: A weight of 10 means that the score is multiplied by 10 during calculation.) You can also select a search field and set a weight for it. The field must be an attribute field, and only numeric fields are supported, such as INT, DOUBLE, and FLOAT fields. The score calculated as the field value multiplied by the weight is also added to the sort score.

For example, select the sale_price field under Search Field and set the weight to 0.08.

After the configuration is complete, you are returned to the Policy Management page.

2. Create a business sort policy. Go to the OpenSearch console. Choose Sort Configuration > Policy Management and click Create. Enter a policy name, set Scope to Business Sort, set Type to Expression, and then click Next.

Select text_relevance from the built-in functions, enter the name of a field configured in the index to be queried within the parentheses, and then click Finish. In the sort configuration step, select a field from the Field drop-down list, enter the sort expression text_relevance(brand) in the editor, and then click Finish. After the configuration is complete, you are returned to the sort configuration page.

3. View the sorting effect. On the search test page, configure the basic sort and business sort parameters and enable the display of sort details: On the Search Test page, set the first_rank_name parameter to the name of the basic sort policy that you created, such as test_1, set the second_rank_name parameter to the name of the business sort policy that you created, such as test_2, and turn on Show Sort Details to view the score details of documents. View the scoring results of each function.

SDK configuration examples

Java SDK example:

// Specify the rough sort and fine sort expressions. The default expressions are used here.
Rank rank =new Rank();
rank.setFirstRankName("default");// The name of the basic sort policy
rank.setSecondRankName("default");// The name of the business sort policy
rank.setReRankSize(5);// The number of documents that participate in fine sort

PHP SDK example:

// Specify the rough sort expression
$params->setFirstRankName('default');
// Specify the fine sort expression
$params->setSecondRankName('default');

Note:

• If you set default basic sort and business sort policies in the console and also specify basic sort and business sort policies in your code, the query operation uses the basic sort and business sort policies configured in the code when the program runs.

• View sort details in code:

  Method: Add the format:fulljson parameter to the config clause.

  In the returned results, sortExprValues indicates the document score.

  "sortExprValues": [
      "10000.0399786383"
  ],
  "property": {
  }

  sortExprValues is an array that contains the values of the sort fields in the sort clause. Example:

  sort=-price;-RANK

  In this case, sortExprValues is [price, document score].

  If sort is not specified, the value is the document score by default.