召回配置

如何配置

PAI-Rec引擎已经内置了多个召回模板，包括协同过滤（UserCollaborativeFilterRecall），向量召回（HologresVectorRecall），U2I 召回（UserCustomRecall）等等，并且支持 mysql、hologres、OTS(Tablestore)等多个数据源。

召回公共配置一览

每种召回配置，都会用到公共配置中的一部分，在此统一解释，单独的召回配置中则不再赘述。

配置示例：

"RecallConfs" :[    
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
  	"RecallAlgo":"",
    "ItemType":"",
    "CacheAdapter":"",
    "CacheConfig":"",
    "CachePrefix":"",
    "CacheTime":0
  }
]

协同过滤（UserCollaborativeFilterRecall）

协同过滤需要有两张表，一张u2i表，根据user_id获取item列表，一张i2i表，获取相似的item，这两张表的 schema是固定格式的。

u2i 表

i2i 表

协同过滤不同召回源的配置都是类似的，都在UserCollaborativeDaoConf中进行配置

hologres

配置示例：

"RecallConfs" :[    
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
    "UserCollaborativeDaoConf": {
      "AdapterType": "hologres",
      "HologresName": "holo-info",
      "User2ItemTable": "u2i_table",
      "Item2ItemTable": "i2i_table",
      "Normalization" : "on"
    }
  }
]

UserCollaborativeDaoConfig：

ots(tablestore)

"RecallConfs" :[
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
    "UserCollaborativeDaoConf": {
      "AdapterType": "tablestore",
      "TableStoreName": "tablestore_info",
      "User2ItemTable": "u2i_table",
      "Item2ItemTable": "i2i_table",
      "Normalization" : "on"
    }
  }
]

redis

Redis进行collaborative_filter流程比较特殊，单独说明。 实际上也是分两步：

• 根据RedisPrefix + uid构造key ， 查询U2I列表， 是个string, 支持格式： item_id1,item_id2,item_id3..... 或者 item_id1:score1,item_id2:score,item_id3:score2......

• 然后查询I2I列表， 根据上步查询到的多个item_id，使用MGET进行查询。 I2I的数据也是string , 格式如下：item_id1:score1,item_id2:score2,item_id3:score3......

配置示例：

"RecallConfs" :[
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
    "UserCollaborativeDaoConf": {
      "AdapterType": "redis",
      "RedisName": "redis_info",
      "RedisPrefix": "cr_",
      "Normalization" : "on"
    }
  }
]

实时 U2I2I(RealTimeU2IRecall)

获取数据的思路和协同过滤中的是一样的，只不过U2I的数据获取是通过user历史行为表实时计算的。

user历史行为表是根据实时日志来同步更新的，这样的召回是实时的召回。

user历史行为表

i2i 表

hologres

行为表建表语句：

BEGIN;
CREATE TABLE "sv_rec"."user_behavior_seq" (
  "user_id" text NOT NULL,
  "item_id" text NOT NULL,
  "event" text NOT NULL,
  "play_time" float8  NULL,
  "timestamp" int8 NOT NULL
);
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'orientation', 'column');
call set_table_property('table_name', 'distribution_key', '"user_id"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'clustering_key', '"user_id:asc","timestamp:desc"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'bitmap_columns', '"user_id","event"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'dictionary_encoding_columns', '"user_id:auto","item_id:auto","event"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'time_to_live_in_seconds', '2592000');
comment on table "sv_rec"."user_behavior_seq" is '用户实时行为序列';
comment on column "sv_rec"."user_behavior_seq"."user_id" is '用户id';
comment on column "sv_rec"."user_behavior_seq"."item_id" is 'item id';
comment on column "sv_rec"."user_behavior_seq"."event" is '事件类型';
comment on column "sv_rec"."user_behavior_seq"."play_time" is '阅读时长,播放时长';
comment on column "sv_rec"."user_behavior_seq"."timestamp" is '时间戳,单位秒';
COMMIT;

配置示例：

"RecallConfs" :[    
    {
      "Name": "RealTimeEtrecRecall",
      "RecallType": "RealTimeU2IRecall",
      "RecallCount": 200,
      "RealTimeUser2ItemDaoConf": {
        "UserTriggerDaoConf": {
          "AdapterType": "hologres",
          "HologresName": "holo_info",
          "HologresTableName": "user_behavior_table",
          "WhereClause": "event='xxx'",
          "Limit": 200,
          "EventPlayTime": "playback:5000;playvslide:5000",
          "EventWeight": "playback:1;playvslide:2",
          "WeightExpression": "(-0.2)*((currentTime-eventTime)/3600/24)",
          "WeightMode": "sum",
          "NoUsePlayTimeField": false
        },
        "Item2ItemTable": "i2i_table"
      }
    }
]

RealTimeUser2ItemDaoConf：

向量召回（HologresVectorRecall）

目前向量召回只能使用hologres数据源，向量数据都存在hologres表中

配置示例：

"RecallConfs" :[    
 {
           "Name": "vector_recall",
           "RecallType": "HologresVectorRecall",
           "RecallCount": 100,
           "VectorDaoConf" :{
               "AdapterType": "hologres",
               "HologresName": "holo_info",
               "HologresTableName": "user_embedding_table",
               "KeyField": "user_id",
               "EmbeddingField" :"emb"
           },
           "HologresVectorConf" :{
               "VectorTable" :"item_embedding_table",
               "VectorEmbeddingField" :"emb",
               "VectorKeyField" :"item_id"
           }
        }
]

VectorDaoConf：

HologresVectorConf：

VectorDaoConf里记录的是user向量信息，表的定义如下

BEGIN;
CREATE TABLE "public"."graphsage_user_embedding" (
 "user_id" text NOT NULL,
 "emb" float4[] NOT NULL,
 "dt" text,
PRIMARY KEY ("user_id")
);
CALL SET_TABLE_PROPERTY('"public"."graphsage_user_embedding"', 'orientation', 'row');
CALL SET_TABLE_PROPERTY('"public"."graphsage_user_embedding"', 'clustering_key', '"user_id:asc"');
CALL SET_TABLE_PROPERTY('"public"."graphsage_user_embedding"', 'time_to_live_in_seconds', '3153600000');
comment on column "public"."graphsage_user_embedding"."user_id" is '用户ID';
comment on column "public"."graphsage_user_embedding"."emb" is '用户特征向量';
comment on column "public"."graphsage_user_embedding"."dt" is '日期 yyyyMMdd';
COMMIT;

HologresVectorConf记录的是item向量， 表定义如下

BEGIN;
CREATE TABLE "public"."graphsage_item_embedding" (
 "item_id" text NOT NULL,
 "emb" float4[] NOT NULL,
PRIMARY KEY ("item_id")
);
CALL SET_TABLE_PROPERTY('"public"."graphsage_item_embedding"', 'orientation', 'column');
CALL SET_TABLE_PROPERTY('"public"."graphsage_item_embedding"', 'bitmap_columns', '"item_id"');
CALL SET_TABLE_PROPERTY('"public"."graphsage_item_embedding"', 'time_to_live_in_seconds', '3153600000');
comment on column "public"."graphsage_item_embedding"."item_id" is '物品ID';
comment on column "public"."graphsage_item_embedding"."emb" is '物品特征向量';
COMMIT;

实时向量召回（OnlineHologresVectorRecall）

实时向量召回和向量召回的实现方式是一致的，也是基于hologres的表数据，不同的是，user向量不是从表里获取的，而是实时通过模型获取到的，然后去item向量表查数据。实现思路基本上分为三步：

• 获取user相关特征，user特征可以从数据表里查询

• 调用向量模型，获取user向量。在我们的支持中，模型是部署在EAS上的

• 和向量召回一样，通过item向量表进行查询

配置示例：

"RecallConfs" :[    
  {
      "Name": "online_vector_recall",
      "RecallType": "OnlineHologresVectorRecall",
      "RecallCount": 500,
      "UserFeatureConfs": [
        {
          "FeatureDaoConf": {
            "AdapterType": "hologres",
            "HologresName": "holo_info",
            "FeatureKey": "user:uid",
            "UserFeatureKeyName": "userid",
            "HologresTableName": "user_all_feature_table",
            "UserSelectFields": "*",
            "FeatureStore": "user"
          },
          "Features": []
        }
      ],
      "RecallAlgo": "sv_v2_mind",
      "HologresVectorConf": {
        "HologresName": "holo_info",
        "VectorTable": "item_embedding_table",
        "VectorEmbeddingField": "item_emb",
        "VectorKeyField": "item_id"
      }
    }
]

UserFeatureConfs：

HologresVectorConf：

模型sv_v2_mind在AlgoConfs里定义如下

{
"AlgoConfs": [
  {
      "Name": "sv_v2_mind",
      "Type": "EAS",
      "EasConf": {
        "Processor": "EasyRec",
        "Timeout": 100,
        "ResponseFuncName": "easyrecUserEmbResponseFunc",
        "Url": "http://xxx.vpc.cn-beijing.pai-eas.aliyuncs.com/api/predict/sv_v2_mind",
        "Auth": "xxx"
      }
  }
]
}

和排序模型的配置是一样的，不同点是ResponseFuncName需要固定为easyrecUserEmbResponseFunc

U2I召回

根据user id来找到对应的item列表。这里的表定义是约定好的

u2i 表

hologres

配置示例：

"RecallConfs" :[    
{
           "Name": "user2item_recall",
           "RecallType": "UserCustomRecall",
           "RecallCount": 500,
           "DaoConf" :{
               "AdapterType": "hologres",
               "HologresName": "holo_info",
               "HologresTableName": "user_item_table"
           }
}
]

OTS(Tablestore)

配置示例：

"RecallConfs" :[    
{
           "Name": "user2item_recall",
           "RecallType": "UserCustomRecall",
           "RecallCount": 500,
           "DaoConf" :{
               "AdapterType": "tablestore",
               "TableStoreName": "ots_info",
               "TableStoreTableName": "user_item_table"
           }
}
]

Redis

配置示例：

"RecallConfs" :[    
{
           "Name": "user2item_recall",
           "RecallType": "UserCustomRecall",
           "RecallCount": 500,
           "DaoConf" :{
               "AdapterType": "redis",
               "RedisName": "redis_info", 
               "RedisPrefix": ""
           }
}
]

图召回（GraphRecall）

图召回也属于U2I召回的一种。通过GraphCompute图数据库进行召回。GraphCompute文档参考概览。

配置示例：

"RecallConfs" :[    
{
  "Name": "graph_recall",
  "RecallType": "GraphRecall",
  "RecallCount": 500,
  "GraphConf": {
    "GraphName": "graph_test",
    "ItemId": "item_id",
    "QueryString": "g(\"test\").V(\"$1\").hasLabel(\"user\").outE().inV()",
    "Params": ["user.uid"]
  }
}
]

GraphConf：

用户分组热门召回（UserGroupHotRecall）

分组热门召回的表的格式也是约定好的

group_hot_table

按照用户特征和context信息（例如地域、机型等）组装trigger_id

• 按照顺序将特征用下划线（_）拼接为trigger_id

• 特征值为空对应 "NULL"

• 包含 Boundaries 字段的特征需要进行离散化（左开，右闭区间），比如年龄 [20, 30, 40, 50] --> trigger对应 <=20, 20-30, 30-40，40-50, >50

用户年龄23，对应"20-30"

用户年龄空，对应"NULL"

用户年龄60，对应">50"

用户年龄19，对应"<=20"

hologres表示例：

此处使用性别、年龄、机型三个特征

配置示例：

"RecallConfs" :[    
{
          "Name":"user_group_hot_recall",
          "RecallType": "UserGroupHotRecall",
          "RecallCount" :500,
          "Triggers": [
            {
              "TriggerKey": "gender"
            },
             {
              "TriggerKey": "age",
              "Boundaries": [20,30,40,50]
            },
            {
              "TriggerKey": "os"
            }
          ],
          "DaoConf":{
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "group_hotness_table"
          }
}
]

全局热门召回（UserGlobalHotRecall）

全局热门召回的表schema和分组召回的表schema是相同的，只是全局热门召回表中只有一条数据，而且 trigger_id = -1

hologres

配置示例：

"RecallConfs" :[    
{
          "Name":"UserGlobalHotRecall",
          "RecallType": "UserGlobalHotRecall",
          "RecallCount" :500,
          "DaoConf":{
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "global_hotness_table"
          }
  }
]

OTS(Tablestore)

配置示例：

"RecallConfs" :[    
{
          "Name":"UserGlobalHotRecall",
          "RecallType": "UserGlobalHotRecall",
          "RecallCount" :500,
          "DaoConf":{
                "AdapterType": "tablestore",
                "TableStoreName": "ots_info",
                "TableStoreTableName": "global_hotness_recall"
          }
  }
]

冷启动召回（ColdStartRecall）

查询item表，根据条件或者时间进行过滤，查询出符合规则的候选集

hologres

"RecallConfs" :[    
    {
      "Name": "AllLiveItemRecall",
      "RecallType": "ColdStartRecall",
      "RecallCount": 3000,
      "ColdStartDaoConf": {
        "AdapterType": "hologres",
        "HologresName": "holo_info",
        "HologresTableName": "item_status_table",
        "WhereClause": "islist_status=1",
        "PrimaryKey": "\"item_id\"",
        "TimeInterval": 0
      }
    }
]

ColdStartDaoConf：

如何使用

召回的使用位置对应配置总览中的SceneConfs，SceneConfs是一个Map[string]object结构，可以分场景的使用召回，配置如下

"SceneConfs": {
  "scene_name": {
    "default": {
      "RecallNames": [
        "collaborative_filter"
      ]
    }
  }
}

• scene_name需要替换为自己的场景名。

• default为目录，这里保持默认即可。

• RecallNames是一个[]string，值为召回配置中的自定义名称。