Dataphin PyODPS-Dataphin(Dataphin)-阿里云帮助中心

Dataphin PyODPS is the Python software development kit (SDK) for Dataphin. It is based on MaxCompute PyODPS and lets you securely access data stored in MaxCompute within your Dataphin project.

Background information

Dataphin PyODPS depends on MaxCompute's PyODPS.

Prerequisites

Before you use Dataphin PyODPS, you must set the compute engine for your Dataphin instance to MaxCompute. For more information, see Set the compute engine for a Dataphin instance.

Limits

Dataphin PyODPS is supported only for Dataphin instances that use MaxCompute as the compute engine.

Obtain permissions for data objects

Request permissions for logical tables. For more information, see Request, renew, and return table permissions.
Request permissions for physical tables. For more information, see Request, renew, and return table permissions.
Upload resources to the project that contains the Python node, and then submit or publish the resources to the production environment. For more information, see Upload and reference resources.
Create functions in the project that contains the Python node, and then submit or publish the functions to the production environment. For more information, see Python UDF or Java UDF best practices.

Usage notes

The Dataphin PyODPS package is built into Dataphin and can be called directly. Note the following:

Currently, Dataphin supports the following PyODPS objects: project, table, SQL, resource, and function. Direct calls to other PyODPS objects for MaxCompute are not supported.

When you call PyODPS in Dataphin, command execution is similar to calling PyODPS directly in MaxCompute. The following table describes the details.

PyODPS	Consistent	Notes
SQL	Inconsistent	When you use PyODPS in Dataphin to access a table in a project, you must prefix the table name with the project or business segment information. The following list describes the details: The format of an SQL statement to access a physical table is `${English_name_of_Dataphin_project}.physical_table_name`. For example, if a Dataphin project is named ProjectA, Dataphin automatically generates the project variable ${ProjectA}. If ProjectA contains a physical table named TableA, you can reference the table using `${ProjectA}.TableA`. The dynamic variable ${ProjectA} is replaced with ProjectA_dev by default when executed in the development environment, and with ProjectA when executed in the production environment. You can also directly specify the development project (ProjectA_dev) or the production project (ProjectA). The format of an SQL statement to access a logical table is `${English_name_of_Dataphin_business_segment}.logical_table_name`. For example, if a Dataphin business segment is named LD_BizUnitA, Dataphin automatically generates the business segment variable ${LD_BizUnitA}. If the business segment LD_BizUnitA contains a logical table named DimTableA, you can reference the table using `${LD_BizUnitA}.DimTableA`. This variable is replaced with LD_BizUnitA_dev by default when executed in the development environment, and with LD_BizUnitA when executed in the production environment. You can also directly specify the development project (LD_BizUnitA_dev) or the production project (LD_BizUnitA).
Table	Consistent	You cannot access data objects stored in MaxCompute across different Dataphin projects. You can access only data objects stored in MaxCompute within the Dataphin project where the current Python node is located. For example, if the current Python node is in a project named ProjectB, running `odps.get_project('Test_Project')` returns the following error. `Exception: Dataphin odps client auth check failed, not allow to operate project Test_Project`
Project
Resource
Function

If a permission-related error occurs during an operation, obtain the required permissions for the data object. For more information, see Obtain permissions for data objects.

Usage example

from dataphin import odps
odps.execute_sql("select * from ${[business_segment]}.[logical_table_name]")