LLM data processing for GitHub code-Platform For AI(PAI)-阿里云帮助中心

Use PAI large language model (LLM) data processing components to clean GitHub code data. This topic walks through a sample workflow that uses open-source RedPajama-Data records to demonstrate sensitive-content masking, text normalization, length and repetition filtering, and deduplication.

The DLC component runs on the distributed Ray framework and supports intelligent aggregation, which reduces unnecessary storage operations and improves resource utilization. For more information, see Group data processing components for large models.

Dataset

The preset template uses a dataset of 5,000 samples extracted from the open-source RedPajama-Data project.

Create and run a workflow

Go to the Visualized Modeling (Designer) page.
1. Log on to the PAI console.
2. In the upper-left corner, select your desired region.
3. In the left-side navigation pane, choose Workspaces and click the name of the workspace that you want to open.
4. In the left-side navigation pane, choose Model Training > Visualized Modeling (Designer).
Create a workflow.
1. On the Preset Templates tab, under Business Domain > LLM, click Create on the LLM Data Processing for GitHub Code - DLC component template card.
2. Configure the workflow parameters or use the default settings, and then click Confirm.
3. In the workflow list, select the created workflow and click Open.

Workflow description:

Key algorithm components in the workflow:

LLM-Sensitive Content Mask (DLC)-1

Masks sensitive information in the content field. For example:
- Replaces email address strings with [EMAIL].
- Replaces phone numbers with [TELEPHONE] or [MOBILEPHONE].
- Replaces ID numbers with [IDNUM].
LLM-Clean Special Content (DLC)-1

Removes URLs from the content field.
LLM-Text Normalizer (DLC)-1

Applies Unicode normalization to the text in the content field.
LLM-Clean Copyright Information (DLC)-1

Removes copyright information from the content field.
LLM-Count Filter (DLC)-1

Removes samples whose content field does not meet the specified ratio of alphabetic and numeric characters, or the ratio of alphabetic characters to text tokens. Because GitHub code data is predominantly alphanumeric, this component effectively filters out low-quality samples.
LLM-Length Filter (DLC)-1

Filters samples based on the length, average line length, and maximum line length of the content field. Average and maximum line lengths are calculated by splitting the content field at each newline character (\n).
LLM-N-Gram Repetition Filter (DLC)-1

Filters samples by character-level and word-level N-gram repetition ratios of the content field. All words are converted to lowercase before word-level calculation. A sliding window of size N generates a sequence of grams, and the repetition ratio is sum of the frequencies of grams that appear more than once / sum of the frequencies of all grams.
LLM-Length Filter (DLC)-2

Filters samples based on the length of the content field.
LLM-Document Deduplicator (DLC)-1

Removes similar samples based on the configured window_size, num_blocks, and hamming_distance values.

Run the workflow.

After the workflow finishes, right-click the LLM-Document Deduplicator (DLC)-1 component and choose View Data > Output Data (OSS) to view the data processed by the preceding components.

In the OSS file list, the output is a JSONL file, such as pai_output_data_flow_tdjpk9ztiewm404h48_node_cftghqhnpnjl8cbs8a.jsonl.

LLM data processing for GitHub code (DLC)

Dataset

Create and run a workflow

Related topics