PLDA is a PAI component for topic modeling. It runs Latent Dirichlet Allocation (LDA) — an unsupervised algorithm that discovers how words cluster into abstract topics across a document collection. Specify the number of topics K, and LDA infers topic structure without labeled training data.
LDA was developed by David M. Blei, Andrew Y. Ng, and Michael I. Jordan in 2003. It is widely used for text recognition, classification, and similarity calculation in the text mining field.
How it works
LDA treats each document as a mixture of topics, and each topic as a probability distribution over words. Given a corpus, the algorithm infers:
Which topics each document belongs to — P(z/d), the document-topic distribution
Which words characterize each topic — P(w/z), the topic-word distribution
As a concrete example, given documents where the words are _eat_, _sleep_, _play_, _meow_, and _bark_, LDA might produce:
| Topic | eat | sleep | play | meow | bark |
|---|---|---|---|---|---|
| Topic 1 | 0.1 | 0.3 | 0.2 | 0.4 | 0.0 |
| Topic 2 | 0.2 | 0.1 | 0.4 | 0.0 | 0.3 |
Topic 1 (high weight on _meow_ and _sleep_) likely represents cats; Topic 2 (high weight on _play_ and _bark_) likely represents dogs. This is how to interpret the probability tables in the output.
Prerequisites
Before you begin, ensure that you have:
A PAI workspace with Machine Learning Designer access
Input data in sparse matrix format (key-value pairs of word IDs and frequencies)
(Optional) The Convert Row, Column, and Value to KV Pair component, if your data is not already in key-value format
Configure the component
Use one of the following methods to configure PLDA.
Method 1: Machine Learning Designer
On the pipeline canvas, click the PLDA component and configure the parameters in the right panel. Machine Learning Designer is formerly known as Machine Learning Studio.
Fields Setting tab
| Parameter | Description |
|---|---|
| Feature columns | The columns used as input for LDA training |
Parameters Setting tab
| Parameter | Description | Default |
|---|---|---|
| Topics | The number of topics (K). Valid values: 2–500. Start with a value between 10 and 50, then adjust based on your corpus size and the coherence of resulting topics. | — |
| Alpha | Prior Dirichlet parameter for the document-topic distribution P(z/d). Smaller values produce sparser mixtures — each document focuses on fewer topics. Larger values produce more uniform mixtures across topics. | — |
| Beta | Prior Dirichlet parameter for the topic-word distribution P(w/z). Smaller values produce sharper, more distinct topics. Larger values make topics more diffuse. | — |
| Burn-in iterations | Warm-up iterations before sampling begins. Must be smaller than Total iterations. The default value works well in most cases and does not need adjustment. | 100 |
| Total iterations | Total number of Gibbs sampling iterations. Increase this value if the model has not converged. | 150 |
Method 2: PAI commands
Use the SQL Script component to run PLDA with PAI commands. For more information, see SQL Script.
pai -name PLDA
-project algo_public
-DinputTableName=lda_input
-DtopicNum=10
-topicWordTableName=lda_output;The following table lists all available parameters.
| Parameter | Required | Type | Default | Description |
|---|---|---|---|---|
inputTableName | Yes | STRING | — | Name of the input table |
inputTablePartitions | No | STRING | All partitions | Partitions to use for training. Format: partition_name=value. For multi-level partitions: name1=value1/name2=value2. Separate multiple partitions with commas. |
selectedColNames | No | STRING | All columns | Column names to use as input for LDA |
topicNum | Yes | Positive integer | — | Number of topics. Valid values: 2–500 |
kvDelimiter | No | STRING | : | Delimiter between keys and values. Valid values: space, ,, : |
itemDelimiter | No | STRING | space | Delimiter between key-value pairs. Valid values: space, ,, : |
alpha | No | FLOAT | 0.1 | Prior Dirichlet parameter of P(z/d). Valid values: (0, ∞). Smaller values produce sparser topic mixtures; larger values produce more uniform mixtures. |
beta | No | FLOAT | 0.01 | Prior Dirichlet parameter of P(w/z). Valid values: (0, ∞). Smaller values produce sharper, more distinct topics; larger values produce more diffuse topics. |
topicWordTableName | Yes | STRING | — | Output table for topic-word frequency contributions |
pwzTableName | No | STRING | Not generated | Output table for P(w/z) |
pzwTableName | No | STRING | Not generated | Output table for P(z/w) |
pdzTableName | No | STRING | Not generated | Output table for P(d/z) |
pzdTableName | No | STRING | Not generated | Output table for P(z/d) |
pzTableName | No | STRING | Not generated | Output table for P(z) |
burnInIterations | No | Positive integer | 100 | Warm-up iterations before sampling begins. Must be less than totalIterations. The default value works well in most cases. |
totalIterations | No | Positive integer | 150 | Total Gibbs sampling iterations. z = topic, w = word, d = document. |
enableSparse | No | BOOL | true | Input data format: true for key-value pairs, false for word segmentation results |
coreNum | No | Positive integer | -1 (auto) | Number of cores. Must be set together with memSizePerCore. The system determines this automatically based on input data size. |
memSizePerCore | No | Positive integer | -1 (auto) | Memory per core in MB. Valid values: 1024–65536. The system determines this automatically. |
Input and output
Input
Input data must be in sparse matrix format, with one row per document:
Column 1: document ID
Column 2: key-value pairs of word IDs and word frequencies
If your data is not already in this format, use the Convert Row, Column, and Value to KV Pair component to convert it.
Output
PLDA generates the following tables in order. Only the topic-word frequency contribution table (topicWordTableName) is required. The remaining tables are optional and generated only when you specify their output table names.
| Table | Parameter | Description |
|---|---|---|
| Topic-word frequency contribution table | topicWordTableName | Required. Word frequency contributions per topic. |
| P(w/z) table | pwzTableName | Probability of each word given a topic |
| P(z/w) table | pzwTableName | Probability of each topic given a word |
| P(d/z) table | pdzTableName | Probability of each document given a topic |
| P(z/d) table | pzdTableName | Probability of each topic given a document |
| P(z) table | pzTableName | Marginal probability of each topic |
What's next
SQL Script — learn how to run PAI commands using the SQL Script component