Personalized recommendation (PAI-Rec)

Overview

PAI-Rec is an Alibaba Cloud platform for developing deeply customizable, end-to-end recommendation systems. The platform covers multiple dimensions, including offline processing, online services, real-time data streams, and engineering architecture, and incorporates functional modules such as retrieval, ranking, filtering, and re-ranking. Built on Alibaba Cloud's Apsara big data architecture, PAI-Rec allows developers to choose components and customize recommendation pipeline code to match their tech stack and development practices. The platform also provides tools for data diagnostics, recommendation result debugging, and engine release management, and accelerates iteration through A/B testing and an experiment reporting platform.

By analyzing impression logs, developers can customize feature engineering algorithms, engine configurations, and experiment report metrics to quickly build and optimize recommendation systems. The white-box development model enhances transparency and flexibility. For teams new to recommendation algorithms, we recommend starting with custom algorithm models from Alibaba's algorithm team. This helps deploy the system quickly and lets developers master model training and performance evaluation. For further optimization, you can request a business consultation to collaborate with Alibaba Cloud engineers.

PAI-Rec also supports cold start, traffic throttling, and online learning to meet diverse business requirements.

Product advantages

PAI-Rec offers the following advantages for building a recommendation system:

• Highly transparent white-box design

  Extensive source code helps you understand the details of recommendation algorithms and allows you to flexibly customize code for specific business needs.

  Source code is provided for data feature engineering and sample processing, retrieval and ranking model invocation scripts, EasyRec retrieval and ranking models, and the PAI-Rec engine business logic.

• Convenient customization process for recommendation algorithms

  Generate scripts and configuration files for retrieval and ranking by configuring a user table, item table, and behavior table, simplifying the deployment of customized recommendations.

• Comprehensive engine and experiment management system

  A complete management backend for engines and experiments lets you manage retrieval and ranking components and update engine parameters.

• Fine-grained metric monitoring and reporting

  A metrics and reports backend lets you define custom metrics and track experiment performance daily or hourly for precise control and timely feedback on recommendation effectiveness.

• Offline and online feature consistency assurance

  Specialized tools compare offline and online features for consistency, preventing experimental deviations caused by data discrepancies.

• Intelligent data diagnostics and analysis

  Intelligent diagnostic tools help developers quickly understand their data and select appropriate features and time windows for feature engineering.

• Intuitive toolset for observing recommendation results

  Diagnostic tools help you visualize recommendation results and retrieval data.

• Powerful feature management capabilities

  Integrates with feature platform management tools for better feature management and improved experimental efficiency.

• Comprehensive technical service and support

  Technical services to help you quickly get started with the solution.

Dependent cloud products

PAI-Rec uses PAI-EasyRec to train retrieval and ranking models and the Go-based PAI-Rec engine to build the recommendation system. You can use DataWorks or PAI-Designer to edit and schedule code for feature engineering, sample generation, and model training. The system uses BE, GraphCompute, or Hologres for storing user features and enabling item-to-item (i2i) and vector retrieval, and PAI-EAS for elastic, scalable scoring. The dependencies are as follows:

• Platform for AI (PAI) is a machine learning and deep learning engineering platform for developers and enterprises. It provides end-to-end AI development services, including data annotation, model building, training, deployment, and inference optimization.

• The EasyRec framework includes state-of-the-art deep learning models and supports multiple TensorFlow versions (1.12 ≤ version ≤ 2.4, PAI-TF). It covers the end-to-end needs of a recommendation pipeline, including retrieval, pre-ranking, ranking, re-ranking, multi-objective optimization, and cold start. Developers can use the EasyRec framework to accelerate iteration across the entire recommendation pipeline.

• DataWorks and MaxCompute are two integrated cloud-native big data services. For tasks in a recommendation system such as feature processing, sample generation, profile management, model scheduling, and data updates, they provide easy-to-use development tools and a stable data environment.

  Note

  PAI-Rec currently supports only DataWorks and MaxCompute. If your business requires other big data services, you may need to modify the corresponding engine code. Please discuss your plan with an architect in advance.

• Hologres is a one-stop, real-time data warehouse engine developed by Alibaba Cloud that supports real-time writing, updating, processing, and analysis of massive data volumes. It supports standard SQL (compatible with the PostgreSQL protocol, its syntax, and most of its functions) and petabyte-scale multidimensional analysis (OLAP) and ad hoc analysis. Hologres provides high-concurrency, low-latency online data serving, fine-grained workload isolation, and enterprise-grade security. It integrates with MaxCompute, Flink, and DataWorks to offer an integrated solution for both offline and online data warehousing.

  You can use Hologres to store real-time user behavior sequences, user features, and recommendation retrieval data, and to use its vector retrieval capabilities.

• Graph Compute is a high-performance, distributed graph computing product developed by Alibaba Cloud that provides a one-stop graph technology service for developers at the trillion-level scale. It supports the storage, querying, and computation of complex graph relationship data and integrates with graph algorithms and models. It is widely used in search, recommendation, advertising, real-time risk control, knowledge graphs, and social networks.

Billing

For more information, see PAI-Rec for Personalized Recommendation Billing.