One-hot encoding

How it works

For each column you specify, the component maps every distinct value to a unique index and outputs a binary indicator for each mapping. A column with three distinct values produces three binary columns, with exactly one set to 1.0 per row.

The component supports two modes:

• Training: Reads the input data, builds a mapping table, and writes both the encoded result and the model table. Only the left input node (Input Table Name) is required.

• Prediction: Reads a previously built model table from the right input node (Input Model Table) and applies the existing encoding to new data. Values absent from the mapping table are ignored and not encoded.

Configure the component

PAI -name one_hot
    -project algo_public
    -DinputTable=one_hot_test
    -DbinaryCols=f0,f1,f2
    -DmodelTable=one_hot_model
    -DoutputTable=one_hot_output
    -Dlifecycle=28;

Example

The following example encodes three columns — f0 (BIGINT), f2 (DATETIME), and f4 (BOOLEAN) — while passing f3 (DOUBLE) through as a reserved feature and appending the original f0, f2, and f4 columns to the output.

Step 1: Sample input data

Step 2: PAI command

PAI -project algo_public -name one_hot
    -DinputTable=one_hot
    -DbinaryCols=f0,f2,f4
    -DoutputModelTable=one_hot_model_8
    -DoutputTable=one_hot_in_table_1_output_8
    -DdropLast=false
    -DappendCols=f0,f2,f4
    -DignoreNull=false
    -DoutputTableType=table
    -DreserveCols=f3
    -DcoreNum=4
    -DmemSizePerCore=2048;

Step 3: Output

The model table maps each distinct value to an index. The top row (with col_name fixed to _reserve_) stores the reserved column mapping. All other rows are the encoding mappings.

Scalability test

Test data: 200 million samples, 100,000 distinct enumeration values.

Usage notes

• A single encoding column can have up to tens of millions of distinct values.

• The KV output uses zero-based indexing.

• When predicting with a trained model, dropLast, ignoreNull, and reserveCols are fixed to the values from training and cannot be overridden. Retrain if you need different values.

• When encoding new data using the model, if the data contains discrete values not present in the mapping table of the model, these values are ignored and not encoded. To encode these values, the model mapping table must be retrained.

Console workflow examples:

• Directly use the component for encoding.

• Train a model with the component, then use the model to encode new data.