MaxFrame local debug mode

Background

Traditionally, debugging MaxFrame UDF functions like apply() and apply_chunk() required submitting code to a remote cluster. This prevented setting breakpoints or stepping through code locally. Each modification required resubmission, and developers often had to maintain separate codebases for local and production environments.

MaxFrame local debug mode addresses these challenges by running UDF functions directly in your local Python environment. You can set IDE breakpoints, work entirely offline, and use a single codebase for both local debugging and production runs.

Use cases

Features

Compared to traditional remote debugging, local debug mode offers the following advantages:

• Zero-configuration debugging

  Set debug=True or debug="local". No additional tools or services are required.

  session = new_session(o, debug=True)

• Full offline capability

  Works without network connectivity or remote cluster resources.

• Native IDE support

  • Supports popular IDEs like PyCharm and VSCode, as well as DataWorks Notebook.

  • Retains full debugging capabilities, including setting breakpoints, watching variables, and single-step execution.

  • The debugging experience is identical to native Python development.

• Flexible data sources

  Supports various data sources, including in-memory data, local files, and MaxCompute tables.

  Data source type	Access method	Use case
  In-memory data	md.DataFrame(pd.DataFrame())	Quick logic validation
  MaxCompute table	md.read_odps_table()	Testing with real data
  Local files	Native Pandas data interfaces such as pd.read_csv()	Offline development

• Seamless transition to production

  Your debug code is identical to the production code. Remove debug=True or debug="local" to deploy directly to production.

  # Debug environment
  session = new_session(o, debug=True)
  
  # Production environment
  session = new_session(o)

Quick start

1. Prerequisites

   pip install --upgrade maxframe # MaxFrame SDK v2.5.0 or later is required.

2. Basic example

   from odps import ODPS
   from maxframe import new_session
   import maxframe.dataframe as md
   import pandas as pd
   
   # Initialize an ODPS object.
   o = ODPS(
       access_id='your_access_id',
       secret_access_key='your_secret_key',
       project='your_project',
       endpoint='your_endpoint'
   )
   
   # Enable local debug mode.
   session = new_session(o, debug=True)
   
   # Prepare the data.
   df = md.DataFrame(pd.DataFrame({
       "sales": [5000, 8000, 12000, 3000],
       "region": ["A", "B", "C", "D"]
   }))
   
   def calculate_commission(row):
       sales = row['sales']
       if sales > 10000: # You can set a breakpoint here.
           rate = 0.15
           print(rate)
       elif sales > 5000: # You can set a breakpoint here.
           rate = 0.10
           print(rate)
       else:
           rate = 0.05
       return sales * rate
   
   # Execute and fetch the result.
   result = df.apply(calculate_commission, axis=1).execute().fetch()

Considerations

• Performance differences: Local debug mode is intended for development and validation. Performance does not represent the production environment.

• Data volume limits: Use small datasets for debugging.

• Dependency consistency: Make sure the dependency versions in your local Python environment match those in the production environment.

• Sensitive data: When you debug with a MaxCompute table, be mindful of data permissions and mask sensitive data as needed.