Develop Python UDFs

Use cases

• Field cleansing and standardization: Apply custom logic row by row for tasks like masking phone numbers, normalizing addresses, or converting encodings. This approach is often simpler and more maintainable than using complex SQL expressions.

• Business rule encapsulation: Package business logic, such as risk scoring or tag determination, into reusable functions for multiple SQL jobs.

• Complex parsing: Parse JSON, log strings, or semi-structured text to split a single input row into multiple output rows or columns.

• Calculations with third-party libraries: Import third-party Python packages to perform calculations unavailable in built-in functions.

Feature index

Function types and skeletons

The type you select when creating a draft determines the generated function skeleton:

The following function skeleton is generated for a UDF. The function signature is determined by the parameter signature you provided during creation:

from odps.udf import annotate

@annotate('string -> string')class str_upper(object):
    def __init__(self):
        passdef evaluate(self, a):
        return a

For a UDTF, you must declare multiple output columns in the Parameter signature section by using the Add output column option.

Editor and status indicators

The editor is based on Monaco and provides Python syntax highlighting, bracket pair colorization, and automatic line wrapping. Code changes are saved automatically, so there's no need to save manually.

The editor displays status indicators at the top based on the synchronization state between the draft and MaxCompute:

• Never registered: A message indicates that the current draft has not been registered to ODPS. You can click Register to ODPS to publish it.

• Registered but modified: A yellow warning bar indicates "The code has been modified, and the version running on ODPS is from the previous registration." The live function continues to run the old logic until you register the new version.

Context table

You can bind a context table to your draft by clicking Select table in the top bar. After binding:

• The Data tab displays the table's column names, types, partition identifiers, and comments, and allows you to preview its data.

• The Debug tab automatically populates the table name and uses the first 10 non-partition columns as input columns.

• The validation statement generated in the SQL tab automatically references this table.

Click the × icon next to the table name to unbind it.

Draft list statuses

The draft list in the sidebar uses two indicators to show the status of a draft:

• UDTF or UDAF badge: Appears after the name to indicate a non-standard UDF type. Standard UDFs do not have a badge.

• Yellow dot: Appears after the name to indicate that the code has been modified but not yet re-registered.

You can right-click a draft to Add to AI chat (sends the code as an attachment to the AI), Copy name, or Delete.

Register the function

Click Register to ODPS at the top, or review the resource list in the Register tab below before publishing. The registration process proceeds through the following stages, with real-time progress shown in the status area:

1. Pre-check: Validates the draft's integrity and determines the list of resources to be uploaded.

2. Upload resources: Uploads the code and dependencies as MaxCompute resources. A counter shows the uploaded/total progress.

3. Create function: Executes the CREATE FUNCTION statement.

4. Check (Optional): Runs if Post-registration check is selected. See the section below for details.

5. Persist: Records the registration result, and the "unregistered" indicator in the editor disappears.

The Register tab also displays the function name, language, and a resource manifest (including type, path, size, and sha256) for the current release, which you can review before publishing.

Post-registration check

The Post-registration check option is disabled by default. If you enable it, the client submits a smoke query to MaxCompute to confirm that the function loads correctly at runtime.

This check submits an actual job, which takes longer and incurs compute costs. However, it helps detect issues where a function registers successfully but fails at runtime. We recommend running this check at least once before final deployment.

The check covers two runtimes: it first validates against Python 3.11 (cp311) and then against the server's default, Python 2.7 (cp27). If the latter fails, the function is still registered, but a warning is issued indicating that SQL calls without the cp311 flag may fail.

Handle naming conflicts

If a resource or function with the same name already exists in MaxCompute, the registration will fail with the error: A function with the same name already exists on ODPS. You can then:

• Click Force overwrite to replace the existing resource and function with the current draft.

• Alternatively, cancel the registration and create a new draft with a different function name.

Call the function in SQL

After the function is registered, you can call it in SQL. Because the server's default runtime is cp27, we recommend explicitly declaring the runtime version:

SET odps.sql.python.version=cp311;
SELECT str_upper(name) FROM <project_name>.<table_name> LIMIT 10;

The SQL tab below provides another way to validate your function. It generates a query that references the context table and automatically prepends SET odps.sql.python.version=cp311; to the statement. You can click Run to view the results and the Logview.

Unregister

Click Unregister in the Register tab to remove the function from MaxCompute. The local draft will be preserved.