Training datasets often contain images that are too small (low resolution or corrupt) or too large (inflated file size that slows training). The LVM-Image-Size Filter (DLC) component removes these outliers by filtering image data outside a size range you define, helping you maintain training data quality for image generation models.
How it works
The component calculates the file size of each image and compares it against the minimum and maximum thresholds you set. Images outside the specified range are excluded from the output, which is written to an Object Storage Service (OSS) directory you specify.
On the first run, the component generates a metadata file (meta.jsonl) in the parent directory of your image data path. On subsequent runs, point to this file directly instead of re-scanning the full directory.
Supported computing resources
Configure the component
Add the LVM-Image-Size Filter (DLC) component to your pipeline in Machine Learning Designer, connect an input source, and set the parameters below.
Connect an input source
Connect one of the following to the component's input port:
A Read File Data component that reads the OSS path where your training data is stored
An Image Data OSS Path pointing to the OSS directory or the
meta.jsonlmetadata file from a previous runAny upstream image preprocessing component
Set parameters
Tab | Parameter | Required | Description | Default | |
Field Settings | Image Data OSS Path | No | The OSS directory containing your image data, or the | — | |
OSS Directory for Saving OutputData | No | The OSS directory where filtering results are stored. | — | ||
Output Filename | No | The base name for the output file. |
| ||
Parameter Settings | Minimum Size | Yes | Images smaller than this value are filtered out. Accepted units: KB, MB, GB, TB, PB, EB, ZB, YB. Example: |
| |
Maximum Size | Yes | Images larger than this value are filtered out. Accepted units: KB, MB, GB, TB, PB, EB, ZB, YB. Example: |
| ||
Execution Tuning | Number of processes | No | The number of parallel processes for filtering. |
| |
single/multi nodes | No | Single-node or multi-node mode. Default value: single-node. Multi-node mode supports only the Ray framework. | single-node | ||
Select Resource Group | No | Select a public resource group or a resource quota (if distributed mode is used, you must set resource parameters for the head and worker nodes separately):
| — | ||
Maximum running duration (seconds) | No | The time limit for the component run. The job stops if this duration is exceeded. | — | ||
Review the output
The component writes the following files to the OSS Directory for Saving OutputData:
File | Description | When to use |
| The filtered image list. The file name is set by Output Filename. | Use this as input for downstream training or preprocessing steps. |
| The state file. | Review this file to understand how many images were filtered and why. |
| The parameter configuration used during the run. | Use this to reproduce or audit the filter settings. |