Use MaxFrame in a local environment

更新时间:
复制 MD 格式

You can install the MaxFrame SDK on your local machine and use it to interact with MaxCompute for distributed data processing.

Prerequisites

Before you begin, make sure the following requirements are met:

  • Python 3.7 or 3.11 is installed on your local system. Other versions may cause errors.

  • pip is installed on your local system. If not, visit the Python official website for installation instructions.

  • A MaxCompute project is created. For more information, see Create a MaxCompute project.

Install MaxFrame

  1. In a command-line window (such as Windows CMD), run the following pip command to install the MaxFrame client:

    pip install --upgrade maxframe
  2. Run the following command to verify the installation. If no output or error appears, the installation succeeded.

    python -c "import maxframe.dataframe as md"

    If an error occurs, verify your default Python version and switch to Python 3.7 or 3.11. For example:

    # Check the system default version
    python --version 
    
    # Switch the system default version to Python 3.7, where $path/python3.7 is the installed Python path
    $path/python3.7 -m pip install setuptools>=3.0

Use MaxFrame

  1. Create a local file with the .py extension (for example, test.py). Copy the following sample code into test.py and save the file.

    import os
    import maxframe.dataframe as md
    from odps import ODPS
    from maxframe import new_session
    
    # Create a MaxCompute entry point
    o = ODPS(
        # Set the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable to your AccessKey ID,
        # and ALIBABA_CLOUD_ACCESS_KEY_SECRET to your AccessKey secret.
        # Do not use AccessKey ID and AccessKey secret strings directly.
        os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'),
        os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'),
        project='your-default-project',
        endpoint='your-end-point',
    )
    table = o.create_table("test_source_table", "a string, b bigint", if_not_exists=True)
    with table.open_writer() as writer:
        writer.write([
            ["value1", 0],
            ["value2", 1],
        ])
    
    # Create a MaxFrame session
    session = new_session(o)
    
    df = md.read_odps_table("test_source_table",index_col="b")
    df["a"] = "prefix_" + df["a"]
    
    # Print dataframe data
    print(df.execute().fetch())
    
    # Write MaxFrame DataFrame data to a MaxCompute table
    md.to_odps_table(df, "test_prefix_source_table").execute()
    
    # Destroy the MaxFrame session
    session.destroy()

    Parameter descriptions:

    • ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET: Set these environment variables to the AccessKey ID and AccessKey secret that have the required permissions for your target MaxCompute project. You can obtain your AccessKey ID from the AccessKey management page.

    • your-default-project: The name of your MaxCompute project. Log on to the MaxCompute console, and in the left navigation pane, choose Workspace > Projects to view it.

    • your-end-point: The endpoint of the region where your MaxCompute project resides. For more information, see Endpoint.

  2. Navigate to the directory containing test.py and run it. For example:

    python test.py

Result validation

After running the Python code, query the sink table data. If the result matches expectations, MaxFrame is correctly installed and connected to the MaxCompute cluster.

  1. Running the Python code returns the following result:

    b           a
    0  prefix_value1
    1  prefix_value2
  2. In your target MaxCompute project, run the following SQL to query data in the test_prefix_source_table table.

    SELECT * FROM test_prefix_source_table;

    The returned result is:

    +------------+------------+
    | b          | a          |
    +------------+------------+
    | 0          | prefix_value1 |
    | 1          | prefix_value2 |
    +------------+------------+