Python 2 UDAF

• Encoding declaration: Optional.

  The declaration format is #coding:utf-8 or # -*- coding: utf-8 -*-. The two formats are equivalent. If your Python 2 code contains Chinese characters, an error is reported when the program runs. Add an encoding declaration at the beginning of the code.

• Import modules: Required.

  You must import at least from odps.udf import annotate and from odps.udf import BaseUDAF. The from odps.udf import annotate statement imports the function signature module, which allows MaxCompute to recognize the function signature defined in the code. from odps.udf import BaseUDAF imports the base class for Python UDAFs. You must implement methods such as iterate, merge, and terminate in the derived class.

  If the UDAF code needs to reference file or table resources, include from odps.distcache import get_cache_file for file resources or from odps.distcache import get_cache_table for table resources.

• Function signature: Required.

  The format is @annotate(<signature>). signature defines the data types of the input parameters and the return value. For more information about function signatures, see Function signature and data types.

• Custom Python class (derived class): Required.

  This is the main structure of the UDAF code. It defines the variables and methods that implement your business logic. You can also reference built-in third-party libraries or resources such as files and tables in your code. For more information, see Third-party libraries or Referencing resources.

• Implement Python class methods: Required.

  The Python class implementation includes the following methods. You can implement the methods as needed.

  Method definition	Description
  BaseUDAF.new_buffer()	Returns a buffer for the intermediate value of the aggregate function. The buffer must be a Marshal object, such as a LIST or DICT. The size of the buffer must not increase with the data volume. In extreme cases, the size of the buffer after object serialization must not exceed 2 MB.
  BaseUDAF.iterate(buffer[, args, ...])	Aggregates args into the intermediate value buffer.
  BaseUDAF.merge(buffer, pbuffer)	Merges the intermediate values buffer and pbuffer and stores the result in buffer.
  BaseUDAF.terminate(buffer)	Converts the buffer to a primitive data type of MaxCompute SQL.

#coding:utf-8
# Import the function signature module and the base class.
from odps.udf import annotate
from odps.udf import BaseUDAF
# Function signature.
@annotate('double->double')
# Custom Python class.
class Average(BaseUDAF):
# Implement the methods of the Python class.
    def new_buffer(self):
        return [0, 0]
    def iterate(self, buffer, number):
        if number is not None:
            buffer[0] += number
            buffer[1] += 1
    def merge(self, buffer, pbuffer):
        buffer[0] += pbuffer[0]
        buffer[1] += pbuffer[1]
    def terminate(self, buffer):
        if buffer[1] == 0:
            return 0.0
        return buffer[0] / buffer[1]

• Read data from and write data to local files.

• Start subprocesses.

• Start threads.

• Enable socket communication.

• Use other systems to call Python 2 UDFs.

Due to these limits, the code that you upload must be written by using Python standard libraries. If modules or C extension modules in Python standard libraries are involved in the preceding operations, these modules cannot be used. Take note of the following points about modules in Python standard libraries:

• All the modules that are implemented based on Python standard libraries and do not depend on extension modules are available.

• The following C extension modules are available:

  • array and audioop

  • binascii and bisect

  • cmath, _codecs_cn, _codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw, _collections, and cStringIO

  • datetime

  • _functools and future_builtins

  • _heapq and _hashlib

  • itertools

  • _json

  • _locale and _lsprof

  • math, _md5, and _multibytecodec

  • operator

  • _random

  • _sha256, _sha512, _sha, _struct, and strop

  • time

  • unicodedata

  • _weakref

  • cPickle

• When you run UDF code in a sandbox environment, the maximum size of data that can be written to the standard output (sys.stdout) or standard error output (sys.stderr) is 20 KB. If the size exceeds 20 KB, extra characters are ignored.

Third-party libraries, such as NumPy, are installed in the Python 2 environment of MaxCompute as supplements to standard libraries.

@annotate(<signature>)

'arg_type_list -> type'

• arg_type_list: Represents the data types of the input parameters. Multiple input parameters can be specified, separated by commas (,). Supported data types are BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), CHAR, VARCHAR, complex data types (ARRAY, MAP, STRUCT), and nested complex data types.

  arg_type_list also supports an asterisk (*) or an empty string ('').

  • If arg_type_list is an asterisk (*), it indicates that the function accepts any number of input parameters.

  • If arg_type_list is an empty string (''), it indicates that the function has no input parameters.

  For more information about the extended syntax of the Resolve annotation, see Dynamic parameters for UDAFs and UDTFs.

• type: Represents the data type of the return value. A UDAF returns only one column. Supported data types include BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), complex data types (ARRAY, MAP, STRUCT), and nested complex data types.

• Use a UDF in a MaxCompute project: The method is similar to that of using built-in functions. You can use a user-defined function in the same way that you use a built-in function.

• Use a UDF across projects: Use a UDF of Project B in Project A. The following statement shows an example: select B:udf_in_other_project(arg0, arg1) as res from table_t;. For more information about cross-project sharing, see Cross-project resource access based on packages.