The Keyword Extraction component identifies the most significant words in a text document and scores them by relevance. Use this component to surface key topics from large volumes of unstructured text without manually reviewing each document.
The component uses the TextRank algorithm: it builds a word co-occurrence network and scores each word using a method similar to PageRank. Words that appear frequently alongside other high-scoring words receive higher scores. For example, in the phrase "high-performance distributed cache," both "high-performance" and "distributed" co-occur with "cache" and with each other, so TextRank rates them as more significant than generic words like "high" or "method" used in isolation.
The processing pipeline is:
Load source data into an input table.
Tokenize the text into words.
Filter out stop words and punctuation.
Run TextRank to score and extract the top keywords.
Configure the component
The Keyword Extraction component supports two configuration methods. Use the Designer GUI for interactive exploration and validation. Use PAI commands when integrating keyword extraction into an automated SQL Script pipeline.
Method 1: Designer GUI
On the Designer workflow page, add the Keyword Extraction component and configure the following parameters in the right-side panel.
| Parameter group | Parameter | Required | Description |
|---|---|---|---|
| Fields setting | Document ID column | Yes | The column containing document IDs. |
| Fields setting | Word segmentation results | Yes | The column containing tokenized document content. |
| Parameters setting | Number of keywords to output | No | Number of top keywords to return. Default: 5. |
| Parameters setting | Window size | No | Word co-occurrence window size for TextRank. Default: 2. |
| Parameters setting | Damping coefficient | No | TextRank damping coefficient. Default: 0.85. |
| Parameters setting | Maximum iterations | No | Maximum number of TextRank iterations. Default: 100. |
| Parameters setting | Convergence coefficient | No | Residual threshold for convergence. Default: 0.000001. |
| Execution tuning | Number of cores | No | Worker count. Auto-assigned by default. |
| Execution tuning | Memory per core | No | Memory per worker, in MB. Auto-assigned by default. |
Method 2: PAI commands
Use PAI commands to configure and run Keyword Extraction in a SQL Script component.
PAI -name KeywordsExtraction
-DinputTableName=maple_test_keywords_basic_input
-DdocIdCol=docid -DdocContent=word
-DoutputTableName=maple_test_keywords_basic_output
-DtopN=19;| Parameter | Required | Default | Description |
|---|---|---|---|
inputTableName | Yes | — | The input table. |
inputTablePartitions | No | All partitions | Partitions to use. Format: Partition_name=value. For multi-level partitions: name1=value1/name2=value2. Separate multiple partitions with commas. |
outputTableName | Yes | — | The output table. |
docIdCol | Yes | — | The column containing document IDs. Single column only. |
docContent | Yes | — | The column containing tokenized words. Single column only. |
topN | No | 5 | Number of top keywords to return. If the total number of extracted keywords is less than topN, all keywords are returned. |
windowSize | No | 2 | Word co-occurrence window size for TextRank. |
dumpingFactor | No | 0.85 | The damping coefficient for the TextRank algorithm. |
maxIter | No | 100 | Maximum number of iterations for the TextRank algorithm. |
epsilon | No | 0.000001 | Convergence residual threshold for the TextRank algorithm. |
lifecycle | No | — | Lifecycle of the output table. |
coreNum | No | Auto | Worker count. |
memSizePerCore | No | Auto | Memory per worker, in MB. |
Example
This example extracts the top 19 keywords from a technical document about blended-wing-body aircraft.
Step 1: Prepare the input table
The input table requires two columns: docid (document ID) and word (tokenized content). In the word column, separate tokens with spaces and remove stop words (for example, "the" and "a") and all punctuation.
| docid:string | word:string |
|---|---|
| doc0 | blended-wing-body aircraft is future aviation field development a new direction many research institutions have started on blended-wing-body aircraft research and its fully-automatic shape optimization algorithm has become a new research hot-spot existing achievements basis on top of analyze compare common modeling solving platform usage methods and features design write blended-wing-body aircraft shape optimization geometric modeling grid division flow-field solving shape optimization module compare different algorithms between pros and cons implement blended-wing-body aircraft conceptual-design in shape optimization geometric modeling and grid generation module implement based-on transfinite interpolation grid generation algorithm based-on spline curve modeling method flow-field solving module includes finite difference solver finite element solver and panel method solver among them finite difference solver mainly includes based-on finite difference method potential-flow mathematical modeling based-on Cartesian grid variable step-size difference format derivation Cartesian grid generation index algorithm based-on Cartesian grid Neumann boundary-condition expression form derivation implement based-on finite difference solver two-dimensional airfoil aerodynamic parameters calculation example finite element solver mainly includes based-on variational principle potential-flow finite element theory modeling two-dimensional finite element Kutta condition expression derivation based-on least squares velocity solving algorithm design based-on Gmsh two-dimensional with-wake airfoil spatial grid generator development implement based-on finite element solver two-dimensional airfoil aerodynamic parameters calculation example panel method solver mainly includes based-on panel method potential-flow theory modeling automatic wake generation algorithm design based-on panel method three-dimensional blended-wing-body aircraft flow-field solver development based-on Blasius flat-plate solution drag estimation algorithm design solver Fortran language on port Python and Fortran code mixed-compilation based-on OpenMP and CUDA parallel acceleration algorithm design and development implement based-on panel method solver three-dimensional blended-wing-body aircraft aerodynamic parameters calculation example shape optimization module implemented based-on free form deformation grid deformation algorithm genetic-algorithm differential evolution algorithm aircraft surface-area calculation algorithm based-on moment integration aircraft volume calculation algorithm development based-on VTK data visualization format tool |
Step 2: Run the PAI command
PAI -name KeywordsExtraction
-DinputTableName=maple_test_keywords_basic_input
-DdocIdCol=docid -DdocContent=word
-DoutputTableName=maple_test_keywords_basic_output
-DtopN=19;Step 3: Review the output
The output table contains three columns: docid, keywords, and weight. The weight value reflects each keyword's relative importance in the document—higher values indicate more central terms. Use the weight to filter keywords by relevance. For example, to focus on the most significant terms, keep only keywords where weight > 0.02.
| docid | keywords | weight |
|---|---|---|
| doc0 | based-on | 0.041306752223538405 |
| doc0 | algorithm | 0.03089845626854151 |
| doc0 | modeling | 0.021782865850562882 |
| doc0 | grid | 0.020669749212693957 |
| doc0 | solver | 0.020245609506360847 |
| doc0 | aircraft | 0.019850761705313365 |
| doc0 | research | 0.014193732541852615 |
| doc0 | finite element | 0.013831122054200538 |
| doc0 | solving | 0.012924593244133104 |
| doc0 | module | 0.01280216562287212 |
| doc0 | derivation | 0.011907588923852495 |
| doc0 | shape | 0.011505456605632607 |
| doc0 | difference | 0.011477831662367547 |
| doc0 | potential-flow | 0.010969269350293957 |
| doc0 | design | 0.010830986516637251 |
| doc0 | implement | 0.010747536556701583 |
| doc0 | two-dimensional | 0.010695570768457084 |
| doc0 | development | 0.010527342662670088 |
| doc0 | new | 0.010096978306668461 |