Serverless Spark supports all open-source Spark SQL built-in functions and adds five proprietary functions for common data processing tasks. This topic covers the proprietary functions and how to use them.
For open-source built-in functions, see Spark SQL Functions.
Supported versions
The proprietary functions described in this topic require the following minimum engine versions:
| Version family | Minimum version |
|---|---|
| esr-4.x | esr-4.4.0 |
| esr-3.x | esr-3.4.0 |
| esr-2.x | esr-2.8.0 |
Function reference
| Function | Syntax | Description |
|---|---|---|
| PARQUET_SCHEMA | PARQUET_SCHEMA('<parquet_file_path>') | Returns the schema of a Parquet file |
| PARQUET_METADATA | PARQUET_METADATA('<parquet_file_path>') | Returns the metadata of a Parquet file |
| URL_DECODE | URL_DECODE(string) | Decodes a URL-encoded string |
| MAX_PT | MAX_PT('tableName') | Returns the maximum partition value in a partitioned table |
| KEYVALUE | KEYVALUE(str [, split1, split2,] key) | Extracts a value from a key-value string |
PARQUET_SCHEMA
Returns the schema of a Parquet file.
Syntax
PARQUET_SCHEMA('<parquet_file_path>')Arguments
| Argument | Type | Description |
|---|---|---|
parquet_file_path | STRING | The path to the Parquet file, for example oss://bucket/path/file.parquet |
Returns
A STRING containing the Parquet schema in message format.
Example
> SELECT PARQUET_SCHEMA('oss://bucket/path/part1.parquet');
message spark_schema {
required int32 id;
}Related functions
PARQUET_METADATA — returns file metadata including row group details and compression information.
PARQUET_METADATA
Returns the metadata of a Parquet file, including the file footer, row group details, and column compression and codec information.
Syntax
PARQUET_METADATA('<parquet_file_path>')Arguments
| Argument | Type | Description |
|---|---|---|
parquet_file_path | STRING | The path to the Parquet file, for example oss://bucket/path/file.parquet |
Returns
A STRING containing metadata organized by row group and column chunk. Each entry includes the following fields:
| Field | Description |
|---|---|
row_group_id | The row group index |
row_group_num_rows | The number of rows in the row group |
row_group_num_columns | The number of columns in the row group |
row_group_bytes | The total byte size of the row group |
column_id | The column chunk index within the row group |
path_in_schema | The column path in the schema |
num_values | The number of values in the column chunk |
type | The column data type |
stats | Column statistics, including min, max, and null count |
compression | The compression codec used |
encodings | The encodings applied to the column |
compressedSize | The compressed size of the column chunk |
uncompressedSize | The uncompressed size of the column chunk |
Example
> SELECT PARQUET_METADATA('oss://bucket/path/part1.parquet');
=== Row Group ID: 0 ===
row_group_id: 0
row_group_num_rows: 1
row_group_num_columns: 1
row_group_bytes: 27
--- Column Chunk ID: 0 in Row Group 0 ---
column_id: 0
path_in_schema: id
num_values: 1
type: INT32
stats: min: 1, max: 1, num_nulls: 0
compression: SNAPPY
encodings: PLAIN,BIT_PACKED
compressedSize: 29
uncompressedSize: 27Related functions
PARQUET_SCHEMA — returns the schema structure of a Parquet file.
URL_DECODE
Decodes a string from application/x-www-form-urlencoded Multipurpose Internet Mail Extensions (MIME) format to a standard string.
Syntax
URL_DECODE(string)Arguments
| Argument | Type | Description |
|---|---|---|
string | STRING | The URL-encoded string to decode |
Returns
A STRING containing the decoded value.
| Input | Behavior |
|---|---|
| Valid URL-encoded string | Returns the decoded string |
| Invalid string, safe mode off | Raises CANNOT_DECODE_URL |
| Invalid string, safe mode on | Returns NULL |
To return NULL instead of raising an error for invalid input, enable safe decoding mode:
SET spark.sql.emr.tryUrlDecode=true;Examples
Invalid input with safe mode off (default behavior):
> SELECT URL_DECODE('http%3A%2F%2spark.apache.org');
Error: CANNOT_DECODE_URLEnable safe mode, then retry:
> SET spark.sql.emr.tryUrlDecode=true;
> SELECT URL_DECODE('http%3A%2F%2spark.apache.org');
NULLMAX_PT
Returns the maximum value of a first-level partition that contains data in a partitioned table. Partitions are sorted alphabetically, so this function returns the alphabetically largest non-empty partition name. Use it to query the latest partition without hardcoding the partition value.
Syntax
MAX_PT('tableName')Arguments
| Argument | Type | Description |
|---|---|---|
tableName | STRING | The name of the partitioned table. Supports catalog.database.tableName and database.tableName formats. If no catalog or database is specified, the default catalog and database are used. |
Returns
A STRING containing the maximum partition value among all partitions with data.
Example
Table tbl has two partitions, 2025-05-21 and 2025-05-20, both containing data. The following query returns all rows from the 2025-05-21 partition:
> SELECT * FROM tbl WHERE pt = MAX_PT('tbl');The following result is returned: 
KEYVALUE
Splits a string by two separators and returns the value associated with the specified key.
Syntax
KEYVALUE(str [, split1, split2,] key)Arguments
| Argument | Type | Required | Description |
|---|---|---|---|
str | STRING | Yes | The source string to parse |
split1 | STRING | No | The separator used to split key-value pairs. Defaults to ";" |
split2 | STRING | No | The separator used to split keys from values within each pair. Defaults to ":" |
key | INT | Yes | The key whose value to return |
Returns
A STRING containing the value associated with the specified key. If a segment split by split1 contains multiple instances of split2, the result is undefined.
Example
> SELECT KEYVALUE('0:1\;1:2', 1);
2The string 0:1\;1:2 is split by ; into two pairs: 0:1 and 1:2. The function returns the value associated with key 1, which is 2.