You can install the MaxFrame SDK on your local machine and use it to interact with MaxCompute for distributed data processing.
Prerequisites
Before you begin, make sure the following requirements are met:
-
Python 3.7 or 3.11 is installed on your local system. Other versions may cause errors.
-
pip is installed on your local system. If not, visit the Python official website for installation instructions.
-
A MaxCompute project is created. For more information, see Create a MaxCompute project.
Install MaxFrame
-
In a command-line window (such as Windows CMD), run the following pip command to install the MaxFrame client:
pip install --upgrade maxframe -
Run the following command to verify the installation. If no output or error appears, the installation succeeded.
python -c "import maxframe.dataframe as md"If an error occurs, verify your default Python version and switch to Python 3.7 or 3.11. For example:
# Check the system default version python --version # Switch the system default version to Python 3.7, where $path/python3.7 is the installed Python path $path/python3.7 -m pip install setuptools>=3.0
Use MaxFrame
-
Create a local file with the
.pyextension (for example,test.py). Copy the following sample code intotest.pyand save the file.import os import maxframe.dataframe as md from odps import ODPS from maxframe import new_session # Create a MaxCompute entry point o = ODPS( # Set the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable to your AccessKey ID, # and ALIBABA_CLOUD_ACCESS_KEY_SECRET to your AccessKey secret. # Do not use AccessKey ID and AccessKey secret strings directly. os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'), os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'), project='your-default-project', endpoint='your-end-point', ) table = o.create_table("test_source_table", "a string, b bigint", if_not_exists=True) with table.open_writer() as writer: writer.write([ ["value1", 0], ["value2", 1], ]) # Create a MaxFrame session session = new_session(o) df = md.read_odps_table("test_source_table",index_col="b") df["a"] = "prefix_" + df["a"] # Print dataframe data print(df.execute().fetch()) # Write MaxFrame DataFrame data to a MaxCompute table md.to_odps_table(df, "test_prefix_source_table").execute() # Destroy the MaxFrame session session.destroy()Parameter descriptions:
-
ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET: Set these environment variables to the AccessKey ID and AccessKey secret that have the required permissions for your target MaxCompute project. You can obtain your AccessKey ID from the AccessKey management page.
-
your-default-project: The name of your MaxCompute project. Log on to the MaxCompute console, and in the left navigation pane, choose Workspace > Projects to view it.
-
your-end-point: The endpoint of the region where your MaxCompute project resides. For more information, see Endpoint.
-
-
Navigate to the directory containing
test.pyand run it. For example:python test.py
Result validation
After running the Python code, query the sink table data. If the result matches expectations, MaxFrame is correctly installed and connected to the MaxCompute cluster.
-
Running the Python code returns the following result:
b a 0 prefix_value1 1 prefix_value2 -
In your target MaxCompute project, run the following SQL to query data in the
test_prefix_source_tabletable.SELECT * FROM test_prefix_source_table;The returned result is:
+------------+------------+ | b | a | +------------+------------+ | 0 | prefix_value1 | | 1 | prefix_value2 | +------------+------------+