UDF development (Python 3)

You can use MaxCompute Studio to write a UDF in Python 3. The code must contain the following components:

• Module import: Required.

  At a minimum, you must include from odps.udf import annotate to import the function signature decorator. This lets MaxCompute recognize the function signature you define. If your UDF code needs to reference a file resource or table resource, you must also include from odps.distcache import get_cache_file or from odps.distcache import get_cache_table, respectively.

• Function signature: Required.

  The format is @annotate(<signature>), where signature defines the data types of the function's input parameters and return value. For more information about function signatures, see Function signatures and data types.

• Custom Python class: Required.

  The class organizes your UDF code and defines the variables and methods that implement your business logic. You can also reference a built-in third-party library or reference file and table resources in your code. For more information, see Third-party library or Reference resources.

• The evaluate method: Required.

  This method is part of the custom Python class. The evaluate method defines the input parameters and the return value. A Python class can contain only one evaluate method.

The following code provides a UDF example.

# Import the function signature module.
from odps.udf import annotate
# Define the function signature.
@annotate("bigint,bigint->bigint")
# Define the custom Python class.
class MyPlus(object):
# Implement the evaluate method.
    def evaluate(self, arg0, arg1):
        if None in (arg0, arg1):
            return None
        return arg0 + arg1

The format of a function signature is as follows:

@annotate(<signature>)

The signature is a string that specifies the data types of the input parameters and return value. When you execute the UDF, the data types of the input parameters and return value must be the same as those specified in the function signature. During semantic parsing, the system checks for usage that does not conform to the function signature. If a type mismatch is detected, the system reports an error.

'arg_type_list -> type'

Where:

• arg_type_list: Specifies the data types of the input parameters. You can specify multiple input parameters, separated by commas (,). The supported data types are BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), CHAR, VARCHAR, and complex data types such as ARRAY, MAP, and STRUCT, including nested complex data types.

  arg_type_list also supports an asterisk (*) or an empty string ('').

  • If you set arg_type_list to an asterisk (*), the function accepts any number of input parameters.

  • If you set arg_type_list to an empty string (''), the function has no input parameters.

• type: Specifies the data type of the return value. A UDF returns a single column. The supported data types are: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), and complex data types such as ARRAY, MAP, and STRUCT, including nested complex data types.

The following table provides examples of valid function signatures.

The following code provides an example of how to reference a table resource.

from odps.udf import annotate
from odps.distcache import get_cache_table
@annotate('->string')
class DistCacheTableExample(object):
    def __init__(self):
        self.records = list(get_cache_table('udf_test'))
        self.counter = 0
        self.ln = len(self.records)
    def evaluate(self):
        if self.counter > self.ln - 1:
            return None
        ret = self.records[self.counter]
        self.counter += 1
        return str(ret)

After developing a Python 3 UDF by following the development workflow, you can call it in MaxCompute SQL as follows:

set odps.sql.python.version=cp37;

Python 2 reached its EOL in early 2020. We recommend migrating your projects based on their type:

• New projects: For new MaxCompute projects or projects where you are writing Python UDFs for the first time. We recommend that you use Python 3 to write all Python UDFs.

• Existing projects: For projects with many existing Python 2 UDFs, exercise caution when enabling Python 3. If you plan to gradually migrate all Python 2 UDFs to Python 3, we recommend that you use the following methods:

  • New jobs and new UDFs: Use Python 3 to write UDFs and enable Python 3 at the session level. For more information about how to enable Python 3, see Enable Python 3.

  • Python 2 UDFs: Rewrite Python 2 UDFs to make them compatible with both Python 2 and Python 3. For more information about how to rewrite the UDFs, see Porting Python 2 Code to Python 3.

    Note

    If you need to write public UDFs and authorize multiple MaxCompute projects to use them, we recommend that you ensure the UDFs are compatible with both Python 2 and Python 3.