Third-party packages in PyODPS DataFrame custom functions-MaxCompute(MaxCompute)-阿里云帮助中心

You can upload and use third-party packages such as pandas, scipy, and scikit-learn in PyODPS DataFrame custom functions.

PyODPS DataFrame provides a pandas-like API to operate on MaxCompute data. You can also run it locally with pandas and databases.

Beyond pandas-like methods such as map and apply, PyODPS DataFrame provides a MapReduce API to extend the pandas syntax for large-scale data processing.

PyODPS custom functions are serialized for execution on MaxCompute. The default MaxCompute Python environment includes only numpy, but you can also use libraries with C extensions, such as pandas, scipy, and scikit-learn, in your custom functions.

Note

PyODPS 0.7.4 or later is required.

Upload third-party packages

Note

You only need to upload third-party packages once. If the packages are already available as MaxCompute resources, you can skip this step.

Most mainstream Python packages are available as wheel (.whl) files with pre-built binaries for various platforms. The first step is to find packages compatible with the MaxCompute runtime.

To run custom functions on MaxCompute, you must include all dependency packages. The following table lists the dependencies for each package.

Package name	Dependencies
pandas	numpy, python-dateutil, pytz, six
scipy	numpy
scikit-learn	numpy, scipy

Note

The numpy package is pre-installed. Therefore, you only need to upload python-dateutil, pytz, pandas, scipy, sklearn, and six to use pandas, scipy, and scikit-learn.

Go to python-dateutils to find and download python-dateutil-2.6.0.zip.

Rename the file to python-dateutil.zip and upload it as a resource using the MaxCompute Console.

add archive python-dateutil.zip;

Note

Follow the same procedure to upload pytz and six. Find, download, and upload pytz-2017.2.zip and six-1.11.0.tar.gz.

For packages with C extensions, such as pandas, you must find a wheel file with cp27-cp27m-manylinux1_x86_64 in its filename to ensure compatibility with MaxCompute. Find and download pandas-0.20.2-cp27-cp27m-manylinux1_x86_64.whl, change its extension to .zip, and then upload it by running the add archive pandas.zip; command in the MaxCompute Console.

Follow the same procedure for the other packages. The following table lists the required resources.

Package name	Filename	Uploaded resource name
python-dateutil	python-dateutil-2.6.0.zip	python-dateutil.zip
pytz	pytz-2017.2.zip	pytz.zip
six	six-1.11.0.tar.gz	six.tar.gz
pandas	pandas-0.20.2-cp27-cp27m-manylinux1_x86_64.zip	pandas.zip
scipy	scipy-0.19.0-cp27-cp27m-manylinux1_x86_64.zip	scipy.zip
scikit-learn	scikit_learn-0.18.1-cp27-cp27m-manylinux1_x86_64.zip	sklearn.zip

Note

Alternatively, you can use the PyODPS resource upload API to upload the resources, which is also a one-time operation.

Code verification

Write a simple function that uses all the libraries. It is a best practice to import these third-party packages inside the function.

def test(x):
    from sklearn import datasets, svm
    from scipy import misc
    import numpy as np

    iris = datasets.load_iris()
    assert iris.data.shape == (150, 4)
    assert np.array_equal(np.unique(iris.target),  [0, 1, 2])

    clf = svm.LinearSVC()
    clf.fit(iris.data, iris.target)
    pred = clf.predict([[5.0, 3.6, 1.3, 0.25]])
    assert pred[0] == 0

    assert misc.face().shape is not None

    return x

Note

The preceding code is for demonstration purposes only and serves to verify all packages mentioned in this topic.

After defining the function, write a simple map operation.
Note
Ensure that isolation is enabled at runtime. If it is not enabled at the project level, you can enable it for the current session by setting a global option.
```
from odps import options

options.sql.settings = {'odps.isolation.session.enable': True}
```
You can also enable isolation for a specific execution in the execute method.

Similarly, you can specify the required packages globally with options.df.libraries, or specify them at execution time. You must specify all packages, including dependency packages.

Call the defined function.

hints = {
    'odps.isolation.session.enable': True
}
libraries = ['python-dateutil.zip', 'pytz.zip', 'six.tar.gz', 'pandas.zip', 'scipy.zip', 'sklearn.zip']

iris = o.get_table('pyodps_iris').to_df()

print iris[:1].sepal_length.map(test).execute(hints=hints, libraries=libraries)

Summary

If the third-party packages and their dependencies are already uploaded, you can specify them directly in your code. Otherwise, upload the packages first as described in this topic.

PyODPS resources

For more information, see the PyODPS User Guide.
For related code samples, see aliyun-odps-python-sdk on GitHub.