Columns to KV
The Columns to KV component converts data from multiple table columns into a single column in key-value format. Each column name becomes a key, and the corresponding cell value becomes its value — producing strings like col1:value1,col2:value2. This format simplifies distributed storage and improves query efficiency in big data processing pipelines.
Limitations
Supported computing engines: MaxCompute and Flink.
How it works
The component takes the columns you select and maps each column name to its cell value, separated by the key-value separator. The resulting key-value pairs are joined by the key separator and written to a single output column.
Example transformation
Input:
| row | f0 | f1 |
|---|---|---|
| 1 | 10 | hello |
| 2 | 20 | world |
Output (with default separators : and ,):
| row | kv |
|---|---|
| 1 | f0:10,f1:hello |
| 2 | f0:20,f1:world |
The row column is kept as-is because it is specified in Reserved Column Names. The f0 and f1 columns are converted because they are specified in Selected columns.
Parameters
Input ports
| Port | Data type | Recommended upstream component | Required |
|---|---|---|---|
| Data | None | Read Table, Read CSV File | Yes |
Configuration
Field settings tab
| Parameter | Description |
|---|---|
| Reserved Column Names | Columns to pass through unchanged to the output. By default, all columns are kept. Specify only the columns you want to preserve alongside the new KV column — typically ID or label columns. |
| Selected columns | Columns to convert to key-value format. Each selected column name becomes a key, and its cell value becomes the corresponding value. |
Parameter settings tab
| Parameter | Description |
|---|---|
| KV column name | Name of the new output column that contains the key-value strings. |
| Error handling policy | Behavior when a conversion error occurs. ERROR (default): the task fails and throws an exception. Alternative: skips the conversion and outputs NULL for that row. |
| Key Separator | Delimiter between key-value pairs in the output string. Default: , (comma). For example, two pairs appear as f0:10,f1:hello. |
| Key-value separator | Delimiter between a key and its value. Default: : (colon). For example, column f0 with value 10 becomes f0:10. |
Execution tuning tab
| Parameter | Description |
|---|---|
| Number of workers | Number of parallel workers. Integer from 1 to 9999. Used together with Memory per worker (MB). |
| Memory per worker (MB) | Memory allocated to each worker. Range: 1024–65536. |
Output ports
| Port | Storage location | Recommended downstream component | Model type |
|---|---|---|---|
| Output result | No configuration required | None | None |
Example
The following PyAlink Script example produces the same result as the component configured with f0 and f1 as selected columns, row as the reserved column, and kv as the output column name.
from pyalink.alink import *
def main(sources, sinks, parameter):
data = sources[0]
op = ColumnsToKvBatchOp()\
.setSelectedCols(["f0", "f1"])\
.setReservedCols(["row"])\
.setKvCol("kv")\
.linkFrom(data)
result = op.linkFrom(data)
result.link(sinks[0])
BatchOperator.execute()