The content community industry

The core of a community is its users, who join primarily to consume content. Content can be text, images, audio, and video in the broad sense, or a way to find solutions to problems through the community. High-quality content increases user engagement, brings traffic and time spent to the platform, drives user growth and retention, and creates more business opportunities and revenue.

Search is the most effective way to directly access content in a community. Every community constantly works to solve the following search performance problems:

• How can we accurately understand user search intent and return the most relevant results?

• How can we use differentiated and personalized content distribution to improve the user search experience and strengthen a sense of community and user loyalty?

• How can we enable interaction and connection when retrieving content across different realms, vertical categories, and channels within the community?

• How can we better integrate and develop non-commercial and commercial content?

With these questions in mind, this topic analyzes the search characteristics and challenges of the content community industry, and introduces the solution and best practices of Alibaba Cloud Intelligent OpenSearch Content Community Enhanced Edition.

Search business requirements for the content industry

» More exposure opportunities: A low no-result rate. » Better search quality: High relevance and higher-quality sorting. » Richer business features: Adjust search results based on business attributes. » More comprehensive supporting features: Features such as spelling correction, top searches, hints, and drop-down suggestions. » Lower ownership cost: Lower development, resource, and O&M costs compared with self-managed search engines. » Easier development and use: A short time to market and reduced difficulty in search engine development and result tuning.

» Strong user search intent: The main search aggregates content from multiple channels and requires more precise search relevance.

For example, a forum community has product lines that cover multiple platforms, such as web pages, apps, and miniapps, and has multiple business channels. As the business grows, the traffic to the home page main search (hereinafter referred to as the "main search") keeps increasing. The integration of various paid and traffic acquisition services makes search traffic operations more important. The business demands on the main search keep growing, requiring it to aggregate content from multiple channels, while search result relevance must be more precise, incorporating more commercial factors on top of text relevance. A mature search engine involves systems such as offline modules, online modules, a query understanding service, and an algorithm platform. These systems require extensive development, algorithm tuning, and continuous, complex O&M work. With limited human resources, a self-managed search system struggles to meet business needs.

Common search scenarios

• Search for content such as blog posts, Q&A pairs, and experience sharing

• Discovery of premium content and popular posts

• Traffic acquisition for paid resources

• Filtering and screening by tag categorization

(Image from Alibaba Cloud Developer Community)

• Popular activities and topic interactions

• PGC and UGC

• Search guidance such as top searches, hints, and drop-down suggestions

• Personalization and timeliness

(Image from Alibaba Cloud Developer Community)

OpenSearch Content Community Enhanced Edition

Solution architecture

Feature overview

Based on the latest Alibaba algorithm capabilities, the Content Industry Enhanced Edition is tailored to the pain points and needs of different vertical content search scenarios. It provides intelligent semantic understanding, vector recall, and sorting algorithms dedicated to the content industry, providing dual assurance for search performance and result accuracy in the content industry. It also effectively resolves industry challenges caused by extremely large vocabulary data, such as high search latency, high resource consumption, and a high no-result rate. For the content industry, OpenSearch also provides vector models to enable vector recall and multi-way search, improving query accuracy, and offers a multimodal search solution.

1. Feature differences

Feature list	General Edition	Content Industry Enhanced Edition
One-stop configuration	After you create an application, you must manually create and configure query analysis, sorting policies, and a drop-down suggestion model.	Based on common search scenarios in the content industry, you can select the capabilities and features you need. Application structure templates and index structure templates are also provided to enable one-click configuration and lower the barrier to entry for new users.
Query analysis	Provides capabilities such as synonym expansion, stop word omission, spelling correction, term weight analysis, and category prediction for general industries.	Provides an enhanced analyzer and query analysis features for the content industry. Tailored to content search scenarios and industry challenges, it builds indexes and identifies user query intent more accurately, delivering better results than the General Edition.
Policy-based sorting	After you create an application, you must manually configure and debug the corresponding sorting policies based on your business scenarios.	Building on the application structure templates and index structure templates, it provides sorting expressions commonly used in the content industry, meeting most content industry sorting needs without additional configuration.
Feature iteration	Periodically updates system default dictionaries for the analyzer, query analysis, and more.	Continuously iterates and updates based on changes in content industry terms and products, optimizing the existing tokenization and query analysis capabilities to provide more timely service assurance.

2. Query analysis effect comparison

Compared with the General Edition, the Industry Enhanced Edition provides deeper optimization of query analysis. It not only updates common bad cases on top of the General Edition, but also enriches the existing vocabulary by drawing on the strengths of various sources for the content industry.

• Tokenization: (separated by spaces)

query	General Edition	Industry Enhanced Edition
to decompress	for understanding decompression	for decompression understanding
arguments and parameters	actual participate formal parameter	argument and parameter
struct overloading	structure body weight loading	struct overloading
googlechromeframe	googlechromeframe	google chrome frame

• Spelling correction

  :

query	General Edition	Industry Enhanced Edition
Taobao can only vision	Taobao can only vision	Taobao intelligent vision
mybatics code generation	mybatics code generation	mybatis code generation
computer netwrok	computer netwrok	computer network
WeiHang mini program	micro mini program	WeChat mini program
deep learn west	deep learn west	deep learning

Provides high-quality vector recall models tailored to the vertical data distribution of the content industry, ensuring the recall effectiveness for long-tail queries, queries containing typos, and queries that rely on synonym rewriting for recall

• Vector recall

query	download data for US gmted2010
Vector recall top 1	download URL for gmt43-related code and materials
Vector recall top 2	gmt0054-2010.pdf
Vector recall top 3	gmted2010 US download URL

query	3D game graphics processing
Vector recall top 1	3d game animation processing basics
Vector recall top 2	basics of 3d game animation
Vector recall top 3	animation game processing

query	disable N card
Vector recall top 1	disabling and starting the network card
Vector recall top 2	disable network card
Vector recall top 3	disabling and enabling the network card

Personalized search based on sequence behavior modeling

For example, when a user searches for "interview" and "Java" consecutively, the results differ from those of a user who searches only for "Java". This enables personalized recall to meet the search needs of different users and improve the user search experience.

DeepRanking deep sorting model

The model can have up to hundreds of billions of parameters to ensure better search results, with low training and usage costs that are far below the costs of in-house development in terms of manpower, machines, and R&D support.

The deep recall model integrates the NLP capabilities of Alibaba Cloud DAMO Academy to improve search results and reduce the no-result rate

Based on user and data characteristics and Alibaba's long-term technical expertise, the model structure is deeply customized to achieve a unique Make for you deep model structure.

Onboarding process for the Enhanced Edition

Easy to onboard and quick to get started, with one-click access to industry templates. You can customize feature selection based on your own business needs, and meet the needs of non-technical staff for business intervention tuning and digital operations.

Table structure design

For details, see Application structure ---> Create a multi-table join.

Data ingestion

OpenSearch supports ingesting data into search through data sources, and also supports importing data without a data source by uploading files through the API, SDK, or console. Details are as follows:

1. RDS data source configuration

2. MaxCompute (formerly ODPS) data source configuration

3. PolarDB data source configuration

4. Data import through API/SDK

Content community industry template configuration

Feature selection. Take the "IT" vertical industry as an example. You can select the template features as needed (all selected by default).

The template features include query analysis (IT term weight, IT synonym package, text vectorization, and more), sorting policies (multi-way search, text relevance, and vector relevance), and drop-down suggestions.

Search testing

1. To use "IT

   vector index

   " search, you must first configure the query analysis text vectorization feature and add the corresponding IT

   vector index

2. , then test it in search testing.

Effect customization service

If you need deep recall, sorting effect tuning, or personalized search, the OpenSearch team of experts provides an effect customization service. Contact technical support or your sales representative.

Case study

As a Chinese IT content community, a certain technology community is dedicated to providing Chinese software developers with full lifecycle services such as knowledge dissemination, online learning, and career development, and it offers multiple products.

Since it began using the Alibaba Cloud OpenSearch product, over the course of a year it has gradually onboarded multiple platforms on both PC and mobile, covering channel search for segmented businesses such as home page search, blogs, downloads, and Q&A. Through OpenSearch, it provides high-quality search services to users of its products, and drives more business conversions by optimizing search capabilities, increasing overall business revenue.

• Compared with the self-managed service built on open source, the CTR increased by more than 80%.

• Later, algorithm experts continuously tuned the models for the customer through deep customization. Clicks per exposed user increased by 16.7%, and Item-CTR increased by 11.8%, with results still improving.