Python SDK examples for managing MaxCompute resources in common scenarios-MaxCompute(MaxCompute)-阿里云帮助中心

Overview

MaxCompute resources are primarily used in user-defined functions (UDFs) and MapReduce. The o variable in the following examples refers to the MaxCompute entry object.

Basic resource operations

PyODPS provides five methods for basic resource operations:

Method	Description
`list_resources`	Queries all resources in a project.
`exist_resource`	Checks whether a resource exists.
`create_resource`	Creates a resource.
`open_resource`	Reads a resource.
`delete_resource`	Deletes a resource. You can also call the `drop` method on a Resource object to delete it.

Resource types

PyODPS supports two types of resources:

Type	Subtypes	Description
File resource	`FILE`, `PY`, `JAR`, `ARCHIVE`	Uploaded files such as scripts, libraries, and archives.
Table resource	--	References to MaxCompute tables.

Note

When you upload a PY file in DataWorks, you must set the resource type to file. For more information, see Python UDF documentation.

Manage file resources

Create a file resource

Specify a resource name, file type, and a file-like object or string as the file_obj parameter, then call the create_resource method.

# Create a file resource from a file-like object.
# Use binary mode ('rb') for files such as compressed packages.
resource = o.create_resource('test_file_resource', 'file', file_obj=open('/to/path/file', 'rb'))

# Create a file resource from a string.
resource = o.create_resource('test_py_resource', 'py', file_obj='import this')

Read and modify a file resource

You can open a file resource by using either of the following methods:

Call the open method on a file resource object.
Call the open_resource method at the MaxCompute entry point.

The returned object is a file-like object. Opening modes work similarly to the built-in Python open function.

Method 1: Open by calling open on the resource object

with resource.open('r') as fp:  # Open in read mode
    content = fp.read()          # Read all content
    fp.seek(0)                   # Return to the beginning
    lines = fp.readlines()       # Read multiple lines
    fp.write('Hello World')      # Error: cannot write in read mode

Method 2: Open by calling open_resource at the entry point

with o.open_resource('test_file_resource', mode='r+') as fp:  # Open in read/write mode
    fp.read()
    fp.tell()                                # Get the current position
    fp.seek(10)
    fp.truncate()                            # Truncate to the specified length
    fp.writelines(['Hello\n', 'World\n'])    # Write multiple lines
    fp.write('Hello World')
    fp.flush()                               # Manually submit the update to MaxCompute

File opening modes

PyODPS supports the following text and binary opening modes:

Text modes

Mode	Access	Behavior
`r`	Read-only	Opens the file for reading. Writing is not allowed.
`w`	Write-only	Opens the file for writing. Reading is not allowed. Clears the file content first.
`a`	Append	Adds data to the end of the file.
`r+`	Read/write	Opens the file for both reading and writing.
`w+`	Read/write	Same as `r+`, but clears the file content first.
`a+`	Read/write (append)	Same as `r+`, but data can only be written to the end of the file.

Binary modes

For some file resources such as compressed files, use binary opening modes:

Mode	Access	Behavior
`rb`	Read-only	Binary read mode.
`r+b`	Read/write	Binary read/write mode.

Manage table resources

Create a table resource

o.create_resource('test_table_resource', 'table', table_name='my_table', partition='pt=test')

Update a table resource

table_resource = o.get_resource('test_table_resource')
table_resource.update(partition='pt=test2', project_name='my_project2')

Obtain table and partition information

table_resource = o.get_resource('test_table_resource')

# Get the table object and its name
table = table_resource.table
print(table.name)

# Get the partition object and its specification
partition = table_resource.partition
print(partition.spec)

Read and write data

table_resource = o.get_resource('test_table_resource')

# Write data to a table resource
with table_resource.open_writer() as writer:
    writer.write([0, 'aaaa'])
    writer.write([1, 'bbbbb'])

# Read data from a table resource
with table_resource.open_reader() as reader:
    for rec in reader:
        print(rec)

Complete resource lifecycle example

The following example demonstrates a typical workflow for managing a file resource, from creation through reading and modification to deletion.

# Step 1: Create a file resource
resource = o.create_resource('test_file_resource', 'file', file_obj='initial content')

# Step 2: Check whether the resource exists
print(o.exist_resource('test_file_resource'))

# Step 3: Read and modify the resource
with resource.open('r+') as fp:
    content = fp.read()
    fp.seek(0)
    fp.write('updated content')
    fp.flush()

# Step 4: List all resources in the project
for res in o.list_resources():
    print(res.name)

# Step 5: Delete the resource
resource.drop()