Improved SWING similarity calculation algorithm

Improved SWING algorithm

Improvement 1: Limit common neighbors

The original SWING algorithm is unreliable when only a few users have interacted with an item pair. With insufficient data, the resulting similarity score can have significant errors.

Consider the following extreme example:

As shown in the following figure, there are three videos: A, B, and C. There are ten loyal Real Madrid fans (X1 to X10) and one music enthusiast (Y) who only listens to music. The Real Madrid fans watch content related to their team, including both the popular video C and the less frequent video A. We can assume they have all watched 200 videos about Real Madrid. One fan, X1, is recommended a niche music video B but dislikes its dark classical style and still prefers the rousing Real Madrid anthem A. The music enthusiast Y listens to various kinds of music, including both A and B. In this scenario, because only two users have watched both videos A and B, the SWING algorithm produces the scores(A,B)=0.33 > s(A,C)=0.22. However, it is clear that recommending video C is a better choice than recommending video B when a user is watching video A.

Solution: This problem arises from the small number of users common to both items. We address this by processing the results based on this number. We enhance the SWING formula by adding an indicator function, Id(.), for noise reduction. Additionally, we introduce behavioral weights to down-weight overly popular users or items. The improved formula is as follows:

Improvement 2: Support scene-specific i2i

Scene-specific item-to-item (i2i) uses the SWING algorithm to learn from the co-occurrence of both global clicks and scene-specific clicks. This approach focuses on predicting user clicks within a particular scene:

The following figure illustrates global i2i and scene-specific i2i. Unlike global i2i, scene-specific i2i ignores users who have no interaction within the target scene and only considers user clicks within the scene.

Improved SWING deployment

1. Download the algorithm package: swing-1.0.jar.

If the download fails, copy the link and paste it into your browser's address bar.

2. In your MaxCompute project, add the JAR resource.

   In the Create Resource dialog box, select JAR for Resource Type, select Upload as ODPS Resource, upload the local file swing-1.0.jar, and click Create.

   Note: You only need to deploy the package once per project.

Input and output format

The input table, which can be a partitioned table, must contain at least two columns:

• user_id: The user ID or session ID (recommended). The type can be BIGINT or STRING. The code does not check the specific data type.

• item_list: A STRING containing a semicolon-separated (;) list of clicked items. Each item in the list is a comma-separated string containing multiple fields.

Each item in the list consists of several comma-separated fields, such as item_id,norm,timestamp,scene, where item_id must be the first field.

The norm field represents the recent popularity of an item, such as the number of users who have clicked it. This value represents the magnitude of the item and penalizes extremely popular items and mitigates the "Harry Potter effect". If you are unsure how to calculate norm or if the "Harry Potter effect" is not significant in your data, you can leave this field empty.

The "Harry Potter effect" is a phenomenon where a very popular and widely known item, such as the Harry Potter book series, dominates recommendation lists, making it difficult for other items to gain similar attention.

Note: The norm value is related only to the current item and represents its global popularity, independent of the current user. For the same item, the norm value must be consistent across all records in the training dataset.

The timestamp must follow the %Y%m%d%H%M%S format (for example, 20190805223205). If timestamps are not required, you can use the same timestamp for all items. The item_list should be ordered chronologically by click time, from oldest to newest.

The scene field is optional. It specifies the scene where the user's action occurred and supports scene-specific i2i.

user_id	item_list
12031602	558448406561,137,20190805223205;585456515773,39397,20190806170331;10200442969,81,20190807223820
3954442742	658448406561,137,20190805223206;485456515773,39397,20190806170335

Note: Do not include duplicate item_id values in the item_list for a single record. We recommend keeping only one <user, item> pair per day. You can create a virtual session ID by processing the user_id in the input table, for example, using concat(user_id, date).

The output table, which supports partition columns, has the following format:

• item_id: The ID of the trigger item (BIGINT).

• similar_items: A list of similar items.

The similar_items are in the format item_id1,score1,coccur1,ori_score1;item_id2,score2,coccur2,ori_score2;.... Here, ori_score1 is the original similarity score, score1 is the score after max normalization, and coccur1 is the co-occurrence count.

Note: The output table must be created in advance. Ensure the column types are correct. You can customize the column names.

Example result:

item_id	similar_items
1084315	7876717,0.000047,2,0.003601;6929557,0.000250,2,0.019373;1084342,0.000780,4,0.060325;1089552,0.000963,4,0.074516;1083467,0.008233,5,0.637016;66042,0.012925,6,1.000000
1090195	1090172,0.015136,1,1.000000

Reference command

In DataWorks, create a new ODPS MR node and use the following command to submit the job. Using an ODPS SQL node may cause an error.

jar [<GENERIC_OPTIONS>] <MAIN_CLASS> [ARGS];
        -conf <configuration_file>         Specify an application configuration file
        -resources <resource_name_list>    file\table resources used in mapper or reducer, seperate by comma
        -classpath <local_file_list>       classpaths used to run mainClass
        -D<name>=<value>                  Property value pair, which will be used to run mainClass
ARGS: <in_table/input_partition> <out_table/output_partition>

Example:

Command for public cloud (out-of-band) users:

##@resource_reference{"swing-1.0.jar"}
jar -resources swing-1.0.jar 
  -classpath swing-1.0.jar 
  -DtopN=150
  -Dmax.user.behavior.count=500
  -Dcommon.user.number.threshold=0
  -Dmax.user.per.item=600
  -Ddebug.info.print.number=10
  -Dalpha1=5
  -Dalpha2=1
  -Dbeta=0.3
  -Dodps.stage.mapper.split.size=1
  com.alibaba.algo.PaiSwing
  swing_click_input_table/ds=${bizdate}
  swing_output/ds=${bizdate}
;

Note: The complete code includes the comment in the first line. Use -D= to specify a parameter value. Do not add a space after -D.

Command for in-band users:

jar -resources swing-1.0.jar 
  -classpath http://schedule@{env}inside.cheetah.alibaba-inc.com/scheduler/res?id=XXXXX
  -DtopN=150
  -Dmax.user.behavior.count=500
  -Dcommon.user.number.threshold=0
  -Dmax.user.per.item=600
  -Ddebug.info.print.number=10
  -Dalpha1=5
  -Dalpha2=1
  -Dbeta=0.3
  -Dodps.stage.mapper.split.size=1
  com.alibaba.algo.PaiSwing
  swing_click_input_table/ds=${bizdate}
  swing_output/ds=${bizdate}
;

To get the in-band classpath:

In DataWorks, right-click the SWING resource package and click "Historical Versions" to get the file path, which starts with http.

Parameters

Parameter	Description	Type and default
common.user.number.threshold	The threshold for the number of users who have interacted with both items. This acts as a filter. If this value is set too high, too few results may be returned. You should tune this parameter based on your use case.	Integer. Default: 0.
max.user.per.item	The maximum number of user click sequences to use for calculating the k-nearest neighbors of each item.	Integer. Default: 700.
max.user.behavior.count	The maximum length of a user's behavior sequence. If a sequence exceeds this length, it is truncated to keep only the most recent interactions.	Integer. Default: 600.
debug.info.print.number	The number of records for which to print debug information.	Integer. Default: 10.
alpha1	A SWING algorithm parameter. See formula [1].	Integer. Default: 5.
beta	A SWING algorithm parameter. See formula [1].	Real number. Default: 0.3.
alpha2	A SWING algorithm parameter. See formula [1].	Integer. Default: 1.
user.column.name	The name of the column that contains the user ID or session ID.	String. Default: "user_id".
item.list.column.name	The name of the column that contains the list of {item_id, norm, ...}.	String. Default: "item_list".
topN	The number of k-nearest neighbors to retain for each trigger item.	Integer. Default: 200.
odps.stage.mapper.split.size	[Concurrency Control] The amount of data processed by each mapper.	Integer. Unit: MB. Default: 256.
odps.stage.reducer.num	[Concurrency Control] The number of reducers used to calculate item pair similarity.	Integer. Default: 200.
item.delimiter	The delimiter for the item list in the input table.	String. Default: semicolon (;).
item.field.delimiter	The delimiter for the fields of an item in the input table.	String. Default: comma (,).
pos_norm	The zero-based index of the field corresponding to item popularity (norm). In the example, this is 1.	Integer. Default: 1.
pos_time	The zero-based index of the field corresponding to the timestamp. In the example, this is 2.	Integer. Default: 2.
pos_scene	The zero-based index of the field corresponding to the scene name.	Integer. Default: 3.
target.scene.name	The name of the target scene for scene-specific i2i modeling.	If not specified, global i2i is used by default.
max.time.span	The maximum time interval in days between two clicks for the items to be considered neighbors.	Integer. Default: 1.
do_supplement_by_adamic_adar	Specifies whether to use the Adamic/Adar algorithm to supplement the results if the number of similar items found is less than topN.	Boolean. Default: true.

FAQ

1. java.lang.ClassCastException: com.aliyun.odps.io.LongWritable cannot be cast to com.aliyun.odps.io.Text

FAILED: ODPS-0123131:User defined function exception - Traceback:
java.lang.ClassCastException: com.aliyun.odps.io.LongWritable cannot be cast to com.aliyun.odps.io.Text
 at com.aliyun.odps.udf.impl.batch.TextBinary.put（TextBinary.java:55）
 at com.aliyun.odps.udf.impl.batch.BaseWritableSerde.put（BaseWritableSerde.java:20）
 at com.aliyun.odps.udf.impl.batch.BatchUDTFCollector.collect（BatchUDTFCollector.java:54）
 at com.aliyun.odps.udf.UDTF.forward（UDTF.java:164）
 at com.aliyun.odps.mapred.bridge.LotTaskUDTF.collect（LotTaskUDTF.java:62）
 at com.aliyun.odps.mapred.bridge.LotReducerUDTF$ReduceContextImpl.write（LotReducerUDTF.java:167）
 at com.aliyun.odps.mapred.bridge.LotReducerUDTF$ReduceContextImpl.write（LotReducerUDTF.java:162）
 at com.aliyun.odps.mapred.bridge.LotReducerUDTF$ReduceContextImpl.write（LotReducerUDTF.java:151）
 at com.alibaba.algo.Paiswing$swingI2IReducer.reduce（Paiswing.java:346）
 at sun.reflect.NativeMethodAccessorImpl.invoke0（Native Method）
 at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:62）
 at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
 at java.lang.reflect.Method.invoke（Method.java:497）
 at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runReducer（MapReduceUtils.java:160）
 at com.aliyun.odps.mapred.bridge.LotReducerUDTF.run（LotReducerUDTF.java:330）
 at com.aliyun.odps.udf.impl.batch.BatchStandaloneUDTFEvaluator.run（BatchStandaloneUDTFEvaluator.java:53）

The item_id column in the output table must have the type BIGINT, not STRING. Check the schema information in your CREATE TABLE statement.

References

For the basic SWING algorithm, see SWING Algorithm Tool.