FAQ about SDK for Python

Warning: XXX not installed

Install the missing component identified in the error message:

pip install <component-name>

Project not found

This error has two common causes:

• Wrong endpoint — the endpoint must point to the target MaxCompute project. See Endpoints for the correct values.

• Incorrect parameter order in the MaxCompute entry object — check that all parameters are in the right positions. See Migrate PyODPS nodes from DataWorks to an on-premises environment for the correct initialization syntax.

Syntax error

PyODPS does not support Python 2.5 or earlier. Use one of the following versions:

• Python 2.6

• Python 2.7.6 or later

• Python 3.3 or later

Permission denied on macOS

Run the install command with sudo:

sudo pip install pyodps

Operation not permitted on macOS

This error is caused by System Integrity Protection (SIP). To disable it:

1. Restart your Mac and hold Command + R during boot to enter Recovery Mode.

2. Open Terminal from the Utilities menu and run:

   csrutil disable
   reboot

For background on this macOS restriction, see Operation Not Permitted when on root - El Capitan (rootless disabled).

No module named ODPS

This error usually means PyODPS cannot be found by the current Python interpreter. Check the following:

• Name conflict — if your working directory contains a file named odps.py or a folder named odps, it shadows the installed package. Rename the conflicting file or folder. To remove a previously installed conflicting package, run sudo pip uninstall odps.

• Multiple Python versions — if both Python 2 and Python 3 are installed, the interpreter running your script may not have PyODPS installed. Keep only one Python version, or install PyODPS under each version you use.

• PyODPS not installed — install it under the active Python version. See Install PyODPS.

Cannot import name ODPS

There is a file named odps.py in your working directory that is shadowing the installed package. Rename or remove it before importing.

Cannot import module odps

This error is typically caused by a dependency issue. Join the PyODPS technical support DingTalk group and contact the group administrator for assistance.

ImportError in IPython or Jupyter Notebook

Add the following line at the top of your code:

from odps import errors

If that import itself fails, the Jupyter component is missing. Install it with:

sudo pip install -U jupyter

What does the size field return?

o.get_table('table_name').size returns the physical storage size of the table on disk.

How do I configure a Tunnel endpoint?

Set options.tunnel.endpoint before making any Tunnel calls. For all available options, see the PyODPS options reference.

How do I use third-party packages that contain CPython?

Generate a wheel package that includes CPython, then upload it. See Create a crcmod that can be used in MaxCompute for a step-by-step example.

Is there a size limit for PyODPS DataFrames?

PyODPS has no limit on MaxCompute table size. When you create a DataFrame from a local Pandas DataFrame, the limit is your local machine's available memory.

How do I use max_pt in a DataFrame?

Use the odps.df.func module to call MaxCompute built-in functions:

from odps.df import func
df = o.get_table('your_table').to_df()
df[df.ds == func.max_pt('your_project.your_table')]  # ds is a partition column

What's the difference between open_writer and write_table?

Each write_table() call creates a new file on the server. Creating too many files degrades query performance and can cause memory issues.

To avoid this, write multiple datasets in a single call or pass a Generator object to write_table():

# Before: multiple write_table() calls create many server files
write_table(table, record_batch_1)
write_table(table, record_batch_2)

# After: pass a generator to write in one call
def record_generator():
    yield from record_batch_1
    yield from record_batch_2

write_table(table, record_generator())

See Write data to a table for more examples.

open_writer() writes to a Block by default, which avoids this file-proliferation problem.

Why does my DataWorks PyODPS node return fewer records than local execution?

Instance Tunnel is disabled by default in DataWorks. Without it, instance.open_reader uses the legacy Result interface, which is capped at 10,000 records.

To retrieve all records, enable Instance Tunnel and remove the record limit: After enabling Instance Tunnel, use reader.count to get the total record count.

options.tunnel.use_instance_tunnel = True
options.tunnel.limit_instance_tunnel = False

How do I get the actual row count of a DataFrame?

DataFrame operations in PyODPS are lazy — they only run when explicitly triggered. To get a count immediately:

iris = DataFrame(o.get_table('pyodps_iris'))
iris.count().execute()

For aggregation methods, see Aggregation. For how lazy execution works, see Execution.

sourceIP is not in the white list

The MaxCompute project has IP whitelist protection. Contact the project owner to add your IP address to the whitelist. See Manage IP address whitelists.

options.sql.settings changes are not taking effect

The parameter names differ between the MaxCompute client and PyODPS:

Change your PyODPS code to use odps.stage.mapper.split.size:

from odps import options
options.sql.settings = {'odps.stage.mapper.split.size': 32}

IndexError when calling DataFrame.head

This error means the list index is out of bounds — either the DataFrame has no rows at that index, or the index exceeds the number of rows. Check that the DataFrame contains data before calling head.

ODPSError when uploading a Pandas DataFrame

The error ODPSError: ODPS entrance should be provided means no global MaxCompute entry object is set. Fix it with one of these approaches:

• Use the Room mechanism (%enter) in DataWorks, which sets a global entry automatically.

• Call to_global() on your MaxCompute entry object.

• Pass the entry object explicitly: DataFrame(pd_df).persist('your_table', odps=odps).

lifecycle is not specified in mandatory mode

The target project requires a lifecycle for every table. Set it before writing:

from odps import options
options.lifecycle = 7  # Number of days; must be an integer

Perhaps the datastream from server is crushed

This error is caused by dirty data. Verify that the number of columns in your source data matches the target table schema.

Project is protected

The project's security policy blocks direct table reads. Depending on how much data you need:

To access all data:

• Ask the project owner to add an exception rule for your use case.

• Use DataWorks or another data masking tool to export the data to an unprotected project, then read it from there.

To access a sample:

• Run a query: o.execute_sql('select * from <table_name>').open_reader()

• Convert to DataFrame: o.get_table('<table_name>').to_df()

ConnectionError timed out

The default PyODPS connection timeout is 5 seconds. If your network or proxy is slow, increase the timeout at the start of your script:

from odps import options
options.connect_timeout = 30

Alternatively, add exception handling to catch and retry on timeout.

If the error occurs only in DataWorks, sandbox restrictions may be blocking network access. Switch to an exclusive resource group to run the task outside the sandbox.

get_sql_task_cost is not defined

The function name is wrong. Replace get_sql_task_cost with execute_sql_cost.

Chinese characters are encoded in logs

This issue only affects Python 2.

Format your print statement to pass the string through % interpolation:

print("Chinese text %s" % ('abc'))

DATETIME field is returned as STRING

When options.tunnel.use_instance_tunnel = False, PyODPS uses the legacy Result interface, which returns data in CSV format. In CSV, all values — including DATETIME — are treated as strings.

Enable Instance Tunnel to get the correct types:

options.tunnel.use_instance_tunnel = True

How do I use Python language features with PyODPS DataFrames?

PyODPS DataFrames support standard Python functions and control flow, which can replace complex SQL.

Define reusable functions:

def euclidean_distance(from_x, from_y, to_x, to_y):
    return ((from_x - to_x) ** 2 + (from_y - to_y) ** 2).sqrt()

def manhattan_distance(center_x, center_y, x, y):
    return (from_x - to_x).abs() + (from_y - to_y).abs()

Call them directly on DataFrame columns:

In [42]: df
     from_x    from_y      to_x      to_y
0  0.393094  0.427736  0.463035  0.105007
1  0.629571  0.364047  0.972390  0.081533
2  0.460626  0.530383  0.443177  0.706774
3  0.647776  0.192169  0.244621  0.447979
4  0.846044  0.153819  0.873813  0.257627
5  0.702269  0.363977  0.440960  0.639756
6  0.596976  0.978124  0.669283  0.936233
7  0.376831  0.461660  0.707208  0.216863
8  0.632239  0.519418  0.881574  0.972641
9  0.071466  0.294414  0.012949  0.368514

In [43]: euclidean_distance(df.from_x, df.from_y, df.to_x, df.to_y).rename('distance')
   distance
0  0.330221
1  0.444229
2  0.177253
3  0.477465
4  0.107458
5  0.379916
6  0.083565
7  0.411187
8  0.517280
9  0.094420

In [44]: manhattan_distance(df.from_x, df.from_y, df.to_x, df.to_y).rename('distance')
   distance
0  0.392670
1  0.625334
2  0.193841
3  0.658966
4  0.131577
5  0.537088
6  0.114198
7  0.575175
8  0.702558
9  0.132617

Use loops to combine multiple tables:

Instead of writing 30 UNION ALL statements in SQL, use Python's reduce:

table_names = ['table1', ..., 'tableN']
dfs = [o.get_table(tn).to_df() for tn in table_names]
reduce(lambda x, y: x.union(y), dfs)

# Equivalent to:
df = dfs[0]
for other_df in dfs[1:]:
    df = df.union(other_df)

How do I debug PyODPS programs locally?

Create a DataFrame from a small Pandas sample during development, then switch to MaxCompute for production by changing a single flag:

df = o.get_table('movielens_ratings').to_df()
DEBUG = True
if DEBUG:
    df = df[:100].to_pandas(wrap=True)

When DEBUG = True, all subsequent operations run locally on the first 100 rows using Pandas. Set it to False to run the full dataset on MaxCompute.

For a richer local debugging experience, use MaxCompute Studio.

Nested loop execution is slow

If you put DataFrame construction (df = XXX) inside a loop, PyODPS creates a new DataFrame object on every iteration, which compounds execution time.

Instead, collect results into a Python dict during the loop, then convert to a DataFrame after the loop finishes.

How do I prevent data from being downloaded locally?

See Use a PyODPS node to download data to a local directory for processing or to process data online.

When should I download PyODPS data locally?

Download data locally only when the volume is small enough to fit in memory. For larger datasets, keep the processing on MaxCompute.

For row-level transformations — such as expanding a JSON string into key-value pairs — use PyODPS DataFrame's apply method to run the operation in parallel on MaxCompute:

In [12]: df
               json
0  {"a": 1, "b": 2}
1  {"c": 4, "b": 3}

In [14]: from odps.df import output

In [16]: @output(['k', 'v'], ['string', 'int'])
    ...: def h(row):
    ...:     import json
    ...:     for k, v in json.loads(row.json).items():
    ...:         yield k, v
    ...:

In [21]: df.apply(h, axis=1)
   k  v
0  a  1
1  b  2
2  c  4
3  b  3

How do I read more than 10,000 records with open_reader?

open_reader is capped at 10,000 records by default.

Save the SQL result as a table first, then read the table:

create table your_result_table as select ...;

Then read it with table.open_reader.

Why use built-in operators instead of UDFs?

Built-in operators run significantly faster than User-Defined Functions (UDFs). On a dataset with millions of rows, a UDF on a single column can increase execution time from 7 seconds to 27 seconds. The gap grows larger with bigger datasets or more complex operations.

Partition column values are empty in DataFrame().schema.partitions

PyODPS DataFrames do not distinguish between partition columns and regular columns — partition columns are treated as regular columns. To filter rows by partition, query the column directly:

df = o.get_table().to_df()
print(df[df.ds == ''].execute())

For more on reading partition data, see Tables.

How do I perform a Cartesian product with PyODPS DataFrames?

See PyODPS DataFrame handling Cartesian product.

How do I segment Chinese text with Jieba in a PyODPS node?

See Use a PyODPS node to segment Chinese text based on Jieba.

How do I download all data from a table?

By default, PyODPS does not limit data reads from an Instance. However, the amount of data that can be downloaded for a protected project by using Tunnel commands is limited. If you do not specify options.tunnel.limit_instance_tunnel, the limit is automatically enabled based on the MaxCompute project configuration — in most cases, a maximum of 10,000 records can be downloaded at a time.

To read all data, enable Instance Tunnel and disable the record limit:

options.tunnel.use_instance_tunnel = True
options.tunnel.limit_instance_tunnel = False  # Remove the record cap

with instance.open_reader() as reader:
    # All records are accessible through Instance Tunnel

Can I compute the percentage of null values with execute_sql or a DataFrame?

Both work, but use a DataFrame for aggregate operations — aggregate performance is higher with DataFrames.

How do I enable new data types in PyODPS?

For a single query with execute_sql:

Pass the type flag as a hint with script submit mode:

o.execute_sql('set odps.sql.type.system.odps2=true;query_sql', hints={"odps.sql.submit.mode": "script"})

For a single DataFrame operation (persist, execute, to_pandas):

Pass the flag in the hints parameter:

from odps.df import DataFrame
users = DataFrame(o.get_table('odps2_test'))
users.persist('copy_test', hints={'odps.sql.type.system.odps2': 'true'})

For all DataFrame operations globally:

options.sql.use_odps2_extension = True

ValueError when using PyODPS

Fix this with either approach:

• Upgrade the SDK to v0.8.4 or later.

• Set the maximum Decimal precision in your code:

  from odps.types import Decimal
  Decimal._max_precision = 38

SQL execution through PyODPS is slow

Slow SQL is usually a network, server, or data-volume issue — not a PyODPS issue. Work through these steps:

1. Check network and server latency

Verify whether your proxy server or network link is adding delay, and whether the MaxCompute task queue is backlogged.

2. Separate task submission from data reading

Use run_sql to submit the task asynchronously, wait for it to finish, then read the results separately. This isolates where the delay occurs:

Before:

with o.execute_sql('select * from your_table').open_reader() as reader:
    for row in reader:
        print(row)

After (split for diagnosis):

inst = o.run_sql('select * from your_table')
inst.wait_for_success()
with inst.open_reader() as reader:
    for row in reader:
        print(row)

3. Check for missing Logview in DataWorks

PyODPS versions below 0.11.6 may submit execute_sql or run_sql tasks correctly but fail to generate a Logview link. Upgrade to 0.11.6 or later if this happens.

4. Enable debug logging to pinpoint the delay

Add debug logging to record timestamps for each request and response:

import datetime
import logging
from odps import ODPS

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
o = ODPS(...)  # Fill in your credentials, or omit if the environment provides a MaxCompute entry
print("Check time:", datetime.datetime.now())
inst = o.run_sql("select * from your_table")

Sample output:

Check time: 2025-01-24 15:34:21.531330
2025-01-24 15:34:21,532 - odps.rest - DEBUG - Start request.
2025-01-24 15:34:21,532 - odps.rest - DEBUG - POST: http://service.<region>.maxcompute.aliyun.com/api/projects/<project>/instances
2025-01-24 15:34:21,532 - odps.rest - DEBUG - data: b'<?xml version="1.0" encoding="utf-8"?>\n<Instance>\n  <Job>\n    <Priority>9</Priority>\n    <Tasks>\n      <SQL>\n        ....
2025-01-24 15:34:21,532 - odps.rest - DEBUG - headers: {'Content-Type': 'application/xml'}
2025-01-24 15:34:21,533 - odps.rest - DEBUG - request url + params /api/projects/<project>/instances?curr_project=<project>
2025-01-24 15:34:21,533 - odps.accounts - DEBUG - headers before signing: {'Content-Type': 'application/xml', 'User-Agent': 'pyodps/0.12.2 CPython/3.7.12', 'Content-Length': '736'}
2025-01-24 15:34:21,533 - odps.accounts - DEBUG - headers to sign: OrderedDict([('content-md5', ''), ('content-type', 'application/xml'), ('date', 'Fri, 24 Jan 2025 07:34:21 GMT')])
2025-01-24 15:34:21,533 - odps.accounts - DEBUG - canonical string: POST

application/xml
Fri, 24 Jan 2025 07:34:21 GMT
/projects/maxframe_ci_cd/instances?curr_project=maxframe_ci_cd
2025-01-24 15:34:21,533 - odps.accounts - DEBUG - headers after signing: {'Content-Type': 'application/xml', 'User-Agent': 'pyodps/0.12.2 CPython/3.7.12', 'Content-Length': '736', ....
2025-01-24 15:34:21,533 - urllib3.connectionpool - DEBUG - Resetting dropped connection: service.<region>.maxcompute.aliyun.com
2025-01-24 15:34:22,027 - urllib3.connectionpool - DEBUG - http://service.<region>.maxcompute.aliyun.com:80 "POST /api/projects/<project>/instances?curr_project=<project> HTTP/1.1" 201 0
2025-01-24 15:34:22,027 - odps.rest - DEBUG - response.status_code 201
2025-01-24 15:34:22,027 - odps.rest - DEBUG - response.headers:
{'Server': '<Server>', 'Date': 'Fri, 24 Jan 2025 07:34:22 GMT', 'Content-Type': 'text/plain;charset=utf-8', 'Content-Length': '0', 'Connection': 'close', 'Location': ....
2025-01-24 15:34:22,027 - odps.rest - DEBUG - response.content: b''

The log shows when the code initiated the task (15:34:21.531), sent the request (15:34:21.533), and received the response (15:34:22.027), making it straightforward to identify which stage is slow.