The Sentence Splitting component splits document text into individual sentences by punctuation marks. Use it to pre-process text before running text summarization in a Machine Learning Designer pipeline.
Configure the component
Machine Learning Designer (formerly Machine Learning Studio) supports two configuration methods.
Method 1: Configure on the pipeline page
On the Fields Setting tab, set the input columns and delimiters. On the Tuning tab, adjust compute resources if needed.
Fields Setting tab
| Parameter | Description |
|---|---|
| Column of Marked Document IDs | The name of the document ID column in the input table. |
| Marked Document Content Column | The name of the document content column in the input table. |
| Sentence Delimiter Set | The punctuation characters that mark sentence boundaries. Defaults to period (.), exclamation point (!), and question mark (?). |
Tuning tab
| Parameter | Description |
|---|---|
| Cores | The number of cores to use for processing. Defaults to system-determined. |
| Memory Size per Core | The memory allocated to each core. Defaults to system-determined. |
Method 2: Run a PAI command
Submit the PAI command through the SQL Script component. For details on setting up the SQL Script component, see SQL Script.
PAI -name SplitSentences
-project algo_public
-DinputTableName="test_input"
-DoutputTableName="test_output"
-DdocIdCol="doc_id"
-DdocContent="content"
-Dlifecycle=30Parameters
| Parameter | Required | Description | Default |
|---|---|---|---|
inputTableName | Yes | The name of the input table. | — |
inputTablePartitions | No | The partitions from the input table to use for processing. | All partitions |
outputTableName | Yes | The name of the output table. | — |
docIdCol | Yes | The name of the document ID column. | — |
docContent | Yes | The name of the document content column. Accepts a single column only. | — |
delimiter | No | The punctuation characters that mark sentence boundaries. | Period (.), exclamation point (!), and question mark (?) |
lifecycle | No | The lifecycle of the input and output tables. | — |
coreNum | No | The number of cores to use for processing. | System-determined |
memSizePerCore | No | The memory allocated to each core. | System-determined |
Example
The following example shows how the component splits a document into individual sentences.
The input document (doc_id = 1000894) contains two sentences separated by periods. After processing, the output table contains one row per sentence:
| doc_id | sentence |
|---|---|
| 1000894 | In 2008, the Shanghai Stock Exchange published disclosure guidelines on the corporate social responsibility (CSR) of listed companies. Three types of companies were urged to disclose their CSR reports, and other qualified listed companies were encouraged to voluntarily disclose their CSR reports. |
| 1000894 | In 2012, a total of 379 listed companies made up 40% of all listed companies disclosed CSR reports. Among those companies, 305 were mandated to disclose CSR reports and 74 voluntarily disclosed CSR reports. |
The output table contains two columns: doc_id and sentence.