LLM-Sensitive Keywords Filter (DLC)
The LLM-Sensitive Keywords Filter (DLC) component filters out text entries that contain sensitive keywords from your input data. It recognizes more than 12,000 sensitive keywords by default, and you can provide a custom keyword file to define your own set. The component can also return the detected sensitive keywords so that you can identify which keywords triggered the filter.
This component runs on Deep Learning Containers (DLC) and is configured on the pipeline page of Machine Learning Designer.
Prerequisites
Make sure the following requirements are met:
You have a PAI workspace with access to Machine Learning Designer.
Your input data is stored in an Object Storage Service (OSS) bucket.
Your input data file is in JSON Lines (JSONL) format. For details, see Input data format.
(Optional) If you want to use a custom keyword list instead of the default list, prepare a sensitive keywords file. For details, see Sensitive keywords file format.
Input data format
The input data file in OSS must be in JSON Lines (JSONL) format and meet the following requirements:
Each line in the file is a valid JSON object.
The file consists of multiple lines of JSON objects.
The file itself is not a valid JSON object.
For an example of a valid JSON Lines input file, see the sample data file.
Sensitive keywords file format
If you provide a custom sensitive keywords file, it must follow this format:
Sensitive keyword 1
Sensitive keyword 2
Sensitive keyword 3
...Separate each keyword with a line feed (\n). Each line contains one sensitive keyword.
Configure the component
On the pipeline page of Machine Learning Designer, add the LLM-Sensitive Keywords Filter (DLC) component to your pipeline and configure its parameters. The parameters are organized into the following tabs:
Fields Setting-- Define the input field, output directory, and keyword file.
Tuning -- Set parallel processing options, node configuration, and computing resources.
Fields Setting
Parameter | Required | Description | Default value |
Text Field Name | Yes | The name of the field to process in each JSON object. | N/A |
batch_size | No | The batch size for processing data. | 1000 |
OSS Directory for Saving OutputData | No | The OSS directory for storing the output data. If not specified, the default workspace path is used. | Default path of the workspace |
Sensitive Keywords File | No | The path of the sensitive keywords file. If left empty, the default sensitive keywords list is used. For the file format, see Sensitive keywords file format. | Default sensitive keywords list |
Note: The Fields Setting tab also displays an OSS RAM Role Setting status indicator showing whether your workspace has authorized OSS access.
Tuning
Parameter | Required | Description | Default value |
Number of Processes | No | The number of parallel processes. | 8 |
single/multi nodes | No | The node execution mode. Select single node to run on one node, or multi nodes to distribute processing across multiple nodes. | single node |
Select Resource Group
Configure the computing resources for the component:
Parameter | Required | Description | Default value |
Select Resource Group | No | The resource group to use. Select Public Resource Group to use shared resources, or Dedicated resource group to use a dedicated resource group configured for your workspace. | Public Resource Group |
Node Type | No | The type of computing node. Select CPU or GPU, then choose an instance type from the dropdown list. | CPU |
Nodes | No | The number of computing nodes. Configurable only when multi nodes is selected. | 1 |
VPC Settings | No | The VPC for the computing resources. | N/A |
Maximum Running Duration (seconds) | No | The maximum running time in seconds. If this duration is exceeded, the job is terminated. | N/A |