What is the Text Summarization component-Platform For AI(PAI)-阿里云帮助中心

Before you begin

Add the Sentence Splitting component upstream in your workflow. It splits each document into one sentence per row, which is the required input format for Text Summarization.

Configure the component

Choose one of the following methods:

Method 1: Use the GUI
Method 2: Use PAI commands

Method 1: Use the GUI

Configure the component on the Designer workflow page.

Tab	Parameter	Description
Fields setting	Column for document ID	The name of the column that contains document IDs.
Fields setting	Sentence column	The column that contains sentences. Only one column can be specified.
Parameters setting	Number of key sentences to output	The number of key sentences to include in the summary. Default: `3`.
	Sentence similarity calculation method	The algorithm used to measure sentence similarity. Options: `lcs_sim`, `levenshtein_sim`, `ssk`, `cosine`.
	Weight of matching string	Applies only when the similarity method is `ssk`. Default: `0.5`.
	Length of substring	Applies only when the similarity method is `ssk` or `cosine`. Default: `2`.
	Damping factor	Controls the probability of random traversal in the TextRank graph. Default: `0.85`.
	Maximum iterations	The maximum number of TextRank iterations. Default: `100`.
	Convergence coefficient	The threshold for declaring convergence. Default: `0.000001`.
Execution tuning	Number of cores	Automatically allocated by the system.
Execution tuning	Memory per core	Automatically allocated by the system.

Weight of matching string is active only when Sentence similarity calculation method is set to ssk. Length of substring is active when the method is set to ssk or cosine. Both parameters are ignored for lcs_sim and levenshtein_sim.

Method 2: Use PAI commands

Use the SQL Script component or an ODPS SQL Node to run PAI commands.

The following example extracts the top 2 key sentences using the default similarity method (lcs_sim):

PAI -name TextSummarization
    -project algo_public
    -DinputTableName="test_input"
    -DoutputTableName="test_output"
    -DdocIdCol="doc_id"
    -DsentenceCol="sentence"
    -DtopN=2
    -Dlifecycle=30;

To use a different similarity method, set similarityType. For example, to use the cosine method:

PAI -name TextSummarization
    -project algo_public
    -DinputTableName="test_input"
    -DoutputTableName="test_output"
    -DdocIdCol="doc_id"
    -DsentenceCol="sentence"
    -DtopN=2
    -DsimilarityType="cosine"
    -Dk=3
    -Dlifecycle=30;

Parameters

Parameter	Required	Description	Default
`inputTableName`	Yes	The input table name.	—
`inputTablePartitions`	No	The partitions in the input table to use. If not specified, all partitions are used.	All partitions
`outputTableName`	Yes	The output table name.	—
`docIdCol`	Yes	The column that contains document IDs.	—
`sentenceCol`	Yes	The sentence column. Only one column can be specified.	—
`topN`	No	The number of key sentences to output.	`3`
`similarityType`	No	The sentence similarity method: `lcs_sim`, `levenshtein_sim`, `ssk`, or `cosine`.	`lcs_sim`
`lambda`	No	The weight of a matching string. Applies only when `similarityType` is `ssk`.	`0.5`
`k`	No	The length of a substring. Applies only when `similarityType` is `ssk` or `cosine`.	`2`
`dampingFactor`	No	The damping factor.	`0.85`
`maxIter`	No	The maximum number of iterations.	`100`
`epsilon`	No	The convergence coefficient.	`0.000001`
`lifecycle`	No	The lifecycle of the output table, in days.	—
`coreNum`	No	The number of cores for computation.	Automatically allocated
`memSizePerCore`	No	The memory per core.	Automatically allocated

lambda applies only when similarityType is ssk. k applies only when similarityType is ssk or cosine.

Example

This example walks through extracting a two-sentence summary from a short document about wildlife regulation.

Step 1: Prepare the input table

Create the input table test_input with the following data. The table has two columns: doc_id (the document ID) and sentence (the full document text, one document per row).

doc_id	sentence
1000897	Since the COVID-19 outbreak, the consumption of wild animals has become a prominent issue. This poses a great risk to public health and has drawn widespread social concern. Public security, forestry, and market regulation departments across the country have launched special campaigns to combat the illegal hunting, trafficking, and consumption of wild animals, achieving notable success. While cracking down on these illegal activities, law enforcement found that a large consumer base, enormous poaching profits, and the difficulty and high cost of identification are key reasons the illegal wildlife trade continues to thrive.

doc_id

sentence

1000897

Since the COVID-19 outbreak, the consumption of wild animals has become a prominent issue. This poses a great risk to public health and has drawn widespread social concern. Public security, forestry, and market regulation departments across the country have launched special campaigns to combat the illegal hunting, trafficking, and consumption of wild animals, achieving notable success. While cracking down on these illegal activities, law enforcement found that a large consumer base, enormous poaching profits, and the difficulty and high cost of identification are key reasons the illegal wildlife trade continues to thrive.

To create the table and upload data, use the MaxCompute client with Tunnel commands. For setup instructions, see Connect using the local client (odpscmd) and Tunnel commands.

Step 2: Split sentences with the Sentence Splitting component

Run the Sentence Splitting component on the sentence column. The output table test_output contains one sentence per row:

doc_id	sentence
1000897	Since the COVID-19 outbreak, the consumption of wild animals has become a prominent issue.
1000897	This poses a great risk to public health and has drawn widespread social concern.
1000897	Public security, forestry, and market regulation departments across the country have launched special campaigns to combat the illegal hunting, trafficking, and consumption of wild animals, achieving notable success.
1000897	While cracking down on these illegal activities, law enforcement found that a large consumer base, enormous poaching profits, and the difficulty and high cost of identification are key reasons the illegal wildlife trade continues to thrive.

Step 3: Run Text Summarization

Run the following PAI command using an SQL Script component or an ODPS SQL Node. This command reads from test_output (the sentence-split table) and writes the summary to test_output1.

PAI -name TextSummarization
    -project algo_public
    -DinputTableName="test_output"
    -DoutputTableName="test_output1"
    -DdocIdCol="doc_id"
    -DsentenceCol="sentence"
    -DtopN=2
    -Dlifecycle=30;

The output table test_output1 has two columns: doc_id and abstract.

doc_id	abstract
1000897	Since the COVID-19 outbreak, the consumption of wild animals has become a prominent issue. Public security, forestry, and market regulation departments across the country have launched special campaigns to combat the illegal hunting, trafficking, and consumption of wild animals, achieving notable success.

References

Sentence Splitting — preprocesses text by splitting a document into one sentence per row, required as the upstream component for Text Summarization.
Designer overview — overview of the PAI Designer workflow page.