Paimon stores function metadata in a catalog so that user-defined functions (UDFs) are persisted across sessions and reusable across compute engines. This topic explains how to create, view, update, and delete functions in a Paimon catalog using Apache Flink SQL and Apache Spark SQL.
How it works
Paimon's function abstraction normalizes UDF support across engines. Each function is registered with a fully qualified name (catalog.database.function) and backed by a JAR file that Paimon loads at runtime.
Function metadata is stored in the catalog, not in the JAR file. Dropping a function removes its catalog entry but leaves the JAR file intact. Make sure you have the required network connectivity and file read and write permissions to access the JAR file.
Key concepts
Persistent function — Stored in the catalog and available across sessions and engines. Requires a three-part name: catalog.database.function_name.
Temporary function (Flink) — Available only in the current session. No catalog entry is created.
Session-level function (Spark) — Created with TEMPORARY. Valid only in the current Spark session. Do not specify a database name for temporary functions.
File Function — The function type supported by Paimon. Backed by an implementation class packaged in a JAR file. Supports modular development by letting you update the JAR independently of the catalog entry.
Use cases
-
Data transformation: Define column-level logic for encryption, decryption, or complex type parsing. Register the function once, then call it consistently across Flink and Spark jobs.
-
Parameterized views: Reference functions in view definitions to support dynamic, parameterized queries without duplicating transformation logic.
Manage functions in Flink
Register UDFs to a Paimon catalog in Flink SQL. Both persistent and temporary functions are supported.
Create a function
Use CREATE FUNCTION to register a UDF by specifying the function identifier, Java class name, and JAR file path.
Syntax:
CREATE [TEMPORARY] FUNCTION
[catalog_name.][db_name.]function_name
AS 'fully_qualified_class_name'
LANGUAGE JAVA
USING JAR 'jar_path';| Parameter | Description |
|---|---|
TEMPORARY |
Creates a session-level function. Omit to create a persistent function. |
catalog_name.db_name.function_name |
Fully qualified function name. Required for persistent functions. |
AS 'fully_qualified_class_name' |
The Java class that implements the function logic. |
LANGUAGE JAVA |
The implementation language. |
USING JAR 'jar_path' |
Path to the JAR file. Supports Object Storage Service (OSS) paths (oss://bucket/path/file.jar). |
Example — create a persistent function:
-- Register parse_str in the default database of paimon_catalog.
-- The implementation class is loaded from OSS at runtime.
CREATE FUNCTION `paimon_catalog`.`default`.parse_str
AS 'com.streaming.flink.udf.StrUdf'
LANGUAGE JAVA
USING JAR 'oss://my_bucket/my_location/udf.jar';Paimon does not validate the JAR path at registration time. Make sure you have the required network connectivity and file read and write permissions. If you use Realtime Compute for Apache Flink, upload the JAR through File Management and use the resulting OSS path to avoid connectivity and permission issues.
Update a function
Use ALTER FUNCTION to point an existing function to a new implementation class. This avoids having to drop and recreate the function.
Syntax:
ALTER FUNCTION [catalog_name.][db_name.]function_name
AS 'new_fully_qualified_class_name'
LANGUAGE JAVA;Example — update the implementation class of `parse_str`:
-- Update parse_str to use StrUdf2 instead of StrUdf.
ALTER FUNCTION `paimon_catalog`.`default`.parse_str
AS 'com.streaming.flink.udf.StrUdf2'
LANGUAGE JAVA;View functions
To view functions in the DLF console:
-
In the left navigation pane, click Data Catalog.
-
Click the catalog name to open its details page.
-
Click the database name to open its details page.
-
Click the Functions tab.
Delete a function
Use DROP FUNCTION to remove a function from the catalog.
Syntax:
DROP FUNCTION [catalog_name.][db_name.]function_name;Example:
-- Remove parse_str from the catalog.
-- The JAR file at the registered path is not deleted.
DROP FUNCTION `paimon_catalog`.`default`.parse_str;Dropping a function removes its metadata from the catalog. The referenced JAR file is not affected.
Manage functions in Spark
Register UDFs to a Paimon catalog in Spark SQL. Both persistent and session-level temporary functions are supported.
Create or replace a function
Use CREATE FUNCTION to register a persistent function, or add TEMPORARY to create a session-level function.
Syntax:
CREATE [OR REPLACE] [TEMPORARY] FUNCTION
[catalog_name.][db_name.]function_name
AS 'fully_qualified_class_name'
USING JAR 'jar_path';| Parameter | Description |
|---|---|
OR REPLACE |
Overwrites the existing function with the same name. |
TEMPORARY |
Creates a session-level function. Do not specify a database name for temporary functions. |
AS 'fully_qualified_class_name' |
The Java class that implements the function logic. |
USING JAR 'jar_path' |
Path to the JAR file. Required so that both the driver and executors can load the function. |
Example — create a persistent function:
-- Register simple_udf in the default database of paimon_catalog.
CREATE FUNCTION `paimon_catalog`.`default`.simple_udf
AS 'com.example.SimpleUdf'
USING JAR 'oss://my_bucket/my_location/udf.jar';Example — create a session-level temporary function:
-- Create a temporary function valid only in the current Spark session.
-- Do not specify a database name for temporary functions.
CREATE OR REPLACE TEMPORARY FUNCTION simple_temp_udf
AS 'com.example.SimpleUdf'
USING JAR 'oss://my_bucket/my_location/udf.jar';Always specify USING JAR in the Spark environment. Without it, the driver and executors cannot load the external implementation class.
View functions
To view functions in the DLF console:
-
In the left navigation pane, click Data Catalog.
-
Click the catalog name to open its details page.
-
Click the database name to open its details page.
-
Click the Functions tab.
Delete a function
Use DROP FUNCTION to remove a persistent function. Add the TEMPORARY keyword to drop a session-level function.
Example — drop a persistent function:
DROP FUNCTION `paimon_catalog`.`default`.simple_udf;