Database solution for a government public opinion analysis system-Alibaba Gov Cloud-阿里云帮助中心

Information spreads rapidly on the Internet. Irrational and negative comments can incite negative public sentiment and even erode public trust in the government. An efficient, network-wide public opinion analysis system helps you monitor public opinion in real time, identify negative information early, and take corrective action. This helps prevent potentially significant losses. The requirements for network-wide monitoring, real-time analysis, and data storage for this type of system create significant challenges for the database.

Background information

A network-wide public opinion analysis system must store tens of billions of web pages. It must also crawl and store new web pages and extract their metadata in real time. After the metadata is extracted, further data mining and analysis are performed. This analysis includes the following:

Public opinion impact diagnosis: Predict the scale of communication and diffusion trends to determine whether an issue will become a major public opinion event.
Propagation path analysis: Analyze the key paths of public opinion propagation.
Sentiment analysis: Analyze whether news or comments are positive or negative. After sentiment classification, statistical aggregation is performed.
Alert settings: Set thresholds for the volume of public discussion. When a threshold is reached, the system pushes notifications to relevant parties. This ensures you do not miss the critical window of opportunity to respond.

Based on the preceding description, the data link of a network-wide public opinion analysis system typically involves the following stages:

Full crawl:
A public opinion system first requires a crawler engine. This engine collects messages from various sources, such as major portals, shopping websites, community forums, Weibo, and WeChat Moments. The massive amount of collected web page data, which can be on the scale of tens of billions of records, must be stored in real time.
Incremental deduplication:
For newly crawled pages, the system checks the URL to determine if the page has been fetched before. This check avoids unnecessary duplicate crawling.
Public opinion analysis:
After web pages are collected, their content is extracted. This process involves removing unnecessary tags and extracting the title, summary, body text, and comments. The extracted content is then placed in a storage system for easy querying.
Search and presentation:
The public opinion analysis results must be pushed to a computing platform. This platform performs statistical analysis and generates reports. It can also provide features such as public opinion search.

Public opinion is time-sensitive. This means the system must process new content efficiently, with sub-second retrieval and storage. The data link and business requirements for monitoring public opinion indicate that the database selection for this system is very demanding.

Database selection requirements

The database for a network-wide public opinion analysis system requires the following capabilities:

Massive concurrency:
The database must support massive data storage at the terabyte (TB) or petabyte (PB) level. It must also support high-concurrency access, from hundreds of thousands to tens of millions of Transactions Per Second (TPS), with low access latency.
Elastic scaling:
The volume of data collection changes dynamically when you adjust the subscribed web page sources. In addition, the number of pages crawled per day has clear peaks and troughs. Therefore, the database must support elastic scaling to scale out and in as needed.
Flexible table schema:
A flexible table property structure is required. The properties that you need to track for regular web pages can be very different from those for social media platforms. A flexible schema makes it easier to extend the system.
Tiered storage:
The database must provide options to automatically expire old data or move it to tiered storage. Public opinion data often focuses on recent hot spots. Therefore, older data is accessed less frequently.
Incremental channel:
An efficient incremental channel is necessary. It must periodically export new data to a computing platform. Incremental public opinion data must be synchronized in real time with the computing platform for analysis. The analysis results are then written back to the storage engine. If the database engine supports incremental data, the system architecture can be greatly simplified.

Alibaba Cloud Tablestore is a serverless, NoSQL, multi-model database developed by Alibaba Cloud for storing massive amounts of structured data. It is widely used in scenarios such as social media, the Internet of Things (IoT), artificial intelligence, metadata, and big data. Tablestore provides a Wide Column model that is compatible with HBase and an innovative Timeline model for messages. It delivers petabyte-level storage, tens of millions of TPS, and millisecond-level latency. It is well-suited for network-wide public opinion analysis systems in government scenarios and perfectly meets the database challenges of this business scenario.

Alibaba Cloud e-government cloud public opinion analysis system architecture

The Alibaba Cloud e-government cloud solution uses Tablestore and combines it with other computing, search, and big data products. The following diagram shows a recommended architecture for a public opinion monitoring system in government scenarios:

Architecture description:

Elastic Compute Service (ECS):
The crawler engine can be deployed on Alibaba Cloud ECS. The number of required ECS instances depends on the crawling volume. You can temporarily scale out resources to crawl web pages during daily peak hours. After the original web pages are crawled, their URLs and content are written to the storage system. To avoid duplicate crawling, the crawler engine must deduplicate URLs against a list before crawling.
Tablestore:
Tablestore supports:
- Low-latency random access queries to determine whether a URL already exists. If a URL exists, it does not need to be crawled again.
- Pushing incremental content to computing platforms.
  Previous architectures often required dual writes at the application layer. This means that original web page data, incremental data, and public opinion analysis results had to be written to both the data store and the computing platform. This type of architecture requires you to maintain two sets of write logic. Each dual write from a data source needs to be aware of the downstream system, or you must use a message service to decouple the systems by performing dual writes of messages.
  Traditional databases such as MySQL support subscribing to incremental logs, such as binlogs. If a distributed storage product can support high access volumes and large storage capacities while also providing incremental subscriptions, the system architecture can be greatly simplified.
Realtime Compute for Apache Flink (Blink) and MaxCompute:
After web page data is collected and stored, you can use Blink and MaxCompute to extract metadata from the incremental data. When a new page needs to be processed, a trigger is sent to Blink to extract the web page metadata. The extracted results are stored in Tablestore and simultaneously pushed to MaxCompute for public opinion analysis, such as sentiment analysis and text clustering.
OpenSearch or Elasticsearch:
Public opinion analysis results, such as report data and user sentiment statistics, are written to Tablestore and a search engine (OpenSearch or Elasticsearch). Some reports and threshold alerts are pushed to subscribers. The data in the search engine is used by the online public opinion search system.
DataV:
The statistical results from public opinion analysis, such as report data and user sentiment data, can be very complex. To quickly understand their dynamic development, a graphical representation is required. Alibaba Cloud DataV lets you simply and quickly perform business monitoring, risk warning, and public opinion information analysis. It meets the display requirements of various businesses.