Python 3 UDAF

You can use MaxCompute Studio to write user-defined aggregate function (UDAF) code in Python 3. The code must contain the following information:

• Import modules: Required.

  You must import at least from odps.udf import annotate and from odps.udf import BaseUDAF. The from odps.udf import annotate statement imports the function signature module, which allows MaxCompute to recognize the function signature defined in the code. from odps.udf import BaseUDAF imports the base class for Python UDAFs. You must implement methods such as iterate, merge, and terminate in the derived class.

  If the UDAF code needs to reference file or table resources, include from odps.distcache import get_cache_file for file resources or from odps.distcache import get_cache_table for table resources.

• Function signature: Required.

  The format is @annotate(<signature>). The signature defines the data types of the function's input parameters and return value. For more information about function signatures, see Function signature and data types.

• Custom Python class (derived class): Required.

  This class is the organizational unit for the UDAF code. It defines the variables and methods that implement your business logic. You can also reference built-in third-party libraries, files, or table resources in your code. For more information, see Third-party libraries or Reference resources.

• Implement Python class methods: Required.

  The Python class implementation includes the following methods. You can implement the methods as needed.

  Method definition	Description
  BaseUDAF.new_buffer()	Returns a buffer for the intermediate value of the aggregate function. The buffer must be a Marshal object, such as a LIST or DICT. The size of the buffer must not increase with the data volume. In extreme cases, the size of the buffer after object serialization must not exceed 2 MB.
  BaseUDAF.iterate(buffer[, args, ...])	Aggregates args into the intermediate value buffer.
  BaseUDAF.merge(buffer, pbuffer)	Merges the intermediate values buffer and pbuffer and stores the result in buffer.
  BaseUDAF.terminate(buffer)	Converts the buffer to a primitive data type of MaxCompute SQL.

The following code provides a UDAF example.

# Import the function signature module and the base class.
from odps.udf import annotate
from odps.udf import BaseUDAF
# Function signature.
@annotate('double->double')
# Custom Python class.
class Average(BaseUDAF):
# Implement the methods of the Python class.
    def new_buffer(self):
        return [0, 0]
    def iterate(self, buffer, number):
        if number is not None:
            buffer[0] += number
            buffer[1] += 1
    def merge(self, buffer, pbuffer):
        buffer[0] += pbuffer[0]
        buffer[1] += pbuffer[1]
    def terminate(self, buffer):
        if buffer[1] == 0:
            return 0.0
        return buffer[0] / buffer[1]

The Python Software Foundation will soon stop maintaining Python 2. We recommend that you migrate your projects based on the project type:

• New projects: This applies to new MaxCompute projects or MaxCompute projects in which you are writing a Python UDAF for the first time. We recommend that you write all Python UDAFs in Python 3.

• Existing projects: This applies to MaxCompute projects that have many existing Python 2 UDAFs. Enable Python 3 with caution. If you plan to gradually migrate all Python 2 UDAFs to Python 3, use the following methods:

  • New jobs and new UDAFs: Write them in Python 3 and enable Python 3 at the session level. For more information about how to enable Python 3, see Enable Python 3.

  • Python 2 UDAFs: Rewrite the Python 2 UDAFs to be compatible with both Python 2 and Python 3. For more information about how to rewrite the code, see Porting Python 2 Code to Python 3.

    Note

    If you want to write a public UDAF and grant permissions to multiple MaxCompute projects to use it, we recommend that you make the UDAF compatible with both Python 2 and Python 3.

@annotate(<signature>)

'arg_type_list -> type'

• arg_type_list: specifies the data types of input parameters. If multiple input parameters are used, specify multiple data types and separate them with commas (,). The following data types are supported: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), CHAR, VARCHAR, complex data types (ARRAY, MAP, and STRUCT), and nested complex data types.

  arg_type_list can be represented by an asterisk (*) or left empty ('').

  • If arg_type_list is represented by an asterisk (*), a random number of input parameters are allowed.

  • If arg_type_list is left empty (''), no input parameters are used.

  For more information about the syntax extension of the @Resolve annotation, see Dynamic parameters of UDAFs and UDTFs.

• type: Specifies the data type of the return value. A UDAF returns only one column. Supported data types include the following: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), complex data types (ARRAY, MAP, STRUCT), and nested complex data types.

After you develop a Python 3 UDAF by following the development flow, you can call it in a MaxCompute SQL statement. The call methods are as follows:

• Use a UDF in a MaxCompute project: The method is similar to that of using built-in functions.

• Use a UDF across projects: Use a UDF of Project B in Project A. The following statement shows an example: select B:udf_in_other_project(arg0, arg1) as res from table_t;. For more information about cross-project sharing, see Access resources across projects using packages.

For more information about the format of a Python UDAF function signature, see Function signature and data types.

• You can use an asterisk (*) in the parameter list to accept input parameters of any number and any type. For example, @annotate('double,*->string') indicates that the first parameter is of the DOUBLE type, followed by a list of parameters of any number and any type. In this case, you must write code to determine the number and types of the input parameters and then perform the corresponding operations on them. This is similar to the printf function in the C language.

  Note

  An asterisk (*) has a different meaning when used in the return value list.

• You can use an asterisk (*) in the return value of a UDTF to indicate any number of STRING-type return values. The number of return values depends on the number of aliases that are set when the function is called. For example, for @annotate("bigint,string->double,*"), the call method is UDTF(x, y) as (a, b, c). In this example, three aliases are set after as: a, b, and c. The editor considers a to be of the DOUBLE type because the type of the first column in the return value is specified in the annotation, and considers b and c to be of the STRING type. Because three return values are specified, when the UDTF calls forward, forward must be an array of length 3. Otherwise, a runtime error occurs.

  Note

  This type of error cannot be reported at compile-time. Therefore, when the UDTF caller sets the number of aliases in the SQL statement, they must follow the rules that are defined by the UDTF. Because the number of return values for an aggregate function is fixed at 1, this feature is not applicable to a UDAF.

from odps.udf import annotate
from odps.udf import BaseUDAF
@annotate('bigint,*->string')
class MultiColSum(BaseUDAF):
    def new_buffer(self):
        return [0]
    def iterate(self, buffer, *args):
        for arg in args:
            buffer[0] += int(arg)
    def merge(self, buffer, pbuffer):
        buffer[0] += pbuffer[0]
    def terminate(self, buffer):
        return str(buffer[0])

A UDAF can have only one return value. In the preceding UDAF example, the return value is the sum of multiple input parameters, which is then aggregated and summed across multiple rows. The following code provides a usage example.

-- Sums multiple input parameters.
SELECT my_multi_col_sum(a,b,c,d,e) from values (1,"2","3","4","5"), (6,"7","8","9","10") t(a,b,c,d,e);
-- The return value is 55.