UI automation for mobile use

Tap gestures

Tap the screen at specific coordinates:

session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

# Tap at the specified coordinates.
result = session.mobile.tap(x=500, y=300)
if result.success:
    print("Tap successful")  # Output: Tap successful
else:
    print(f"Tap failed: {result.error_message}")

agent_bay.delete(session)

session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

# Swipe up (from bottom to top).
result = session.mobile.swipe(
    start_x=100,
    start_y=500,
    end_x=100,
    end_y=200,
    duration_ms=300
)
if result.success:
    print("Swipe up successful")  # Output: Swipe up successful

# Swipe left (from right to left).
result = session.mobile.swipe(
    start_x=500,
    start_y=300,
    end_x=100,
    end_y=300,
    duration_ms=300
)
if result.success:
    print("Swipe left successful")  # Output: Swipe left successful

agent_bay.delete(session)

Input text

Enter text into the active input field:

session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

result = session.mobile.input_text("Hello AgentBay!")
if result.success:
    print("Text input successful")  # Output: Text input successful

agent_bay.delete(session)

Send key events

Send Android key events using KeyCode constants:

from agentbay.mobile.mobile import KeyCode

session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

# Press the HOME key.
result = session.mobile.send_key(KeyCode.HOME)
if result.success:
    print("HOME key pressed")  # Output: HOME key pressed

# KeyCode values: HOME=3, BACK=4, VOLUME_UP=24, VOLUME_DOWN=25, POWER=26, MENU=82
print(f"HOME keycode value: {KeyCode.HOME}")  # Output: HOME keycode value: 3

agent_bay.delete(session)

Available KeyCode constants:

Note: All key events are sent directly to the Android system.

Retrieve all UI elements

Retrieve all UI elements in the current screen hierarchy:

session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

result = session.mobile.get_all_ui_elements(timeout_ms=2000)
if result.success:
    print(f"Found {len(result.elements)} UI elements")  # Output: Found 2172 UI elements
    for element in result.elements:
        # The element structure varies. Check the element data.
        print(f"Element: {element}")
        # Example output: Element data contains UI hierarchy information
else:
    print(f"Failed: {result.error_message}")

agent_bay.delete(session)

Parameter:

• timeout_ms: Timeout in milliseconds for UI element retrieval. Default: 2000.

Capture a screenshot

Capture the current cloud phone screen:

session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

result = session.mobile.screenshot()
if result.success:
    screenshot_url = result.data
    print(f"Screenshot URL: {screenshot_url}")
    # Output: Screenshot URL: https://***.***.aliyuncs.com/***/screenshot_1234567890.png?***
else:
    print(f"Screenshot failed: {result.error_message}")

agent_bay.delete(session)

Use a cloud phone image

Cloud phone UI automation requires a mobile OS image such as mobile_latest:

# Correct - Use a cloud phone OS image.
session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

# Incorrect - Cannot be used for cloud phone operations.
session_params = CreateSessionParams(image_id="windows_latest")
session = agent_bay.create(session_params).session

Handle screenshot URLs

The screenshot method returns an OSS URL, not image data:

result = session.mobile.screenshot()
if result.success:
    screenshot_url = result.data
    print(f"Screenshot available at: {screenshot_url}")
    # Output: Screenshot available at: https://***.***.aliyuncs.com/***/screenshot_1234567890.png?***
    # Use the URL to download or display the screenshot.
else:
    print(f"Screenshot failed: {result.error_message}")
    

App navigation

from agentbay import AgentBay
from agentbay.session_params import CreateSessionParams

agent_bay = AgentBay()
session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

try:
    # Tap the application icon.
    tap_result = session.mobile.tap(x=200, y=400)
    print(f"App tap result: {tap_result.success}")  # Output: App tap result: True
    
    # Wait for the application to load.
    import time
    time.sleep(2)
    
    # Swipe to navigate.
    swipe_result = session.mobile.swipe(
        start_x=400,
        start_y=600,
        end_x=100,
        end_y=600,
        duration_ms=300
    )
    print(f"Navigation swipe result: {swipe_result.success}")  # Output: Navigation swipe result: True
    
    # Tap the button.
    button_result = session.mobile.tap(x=300, y=800)
    print(f"Button tap result: {button_result.success}")  # Output: Button tap result: True
    
finally:
    agent_bay.delete(session)

Form filling

from agentbay import AgentBay
from agentbay.session_params import CreateSessionParams

agent_bay = AgentBay()
session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

try:
    # Tap the username field.
    username_tap = session.mobile.tap(x=300, y=400)
    print(f"Username field focused: {username_tap.success}")  # Output: Username field focused: True
    
    # Enter the username.
    username_input = session.mobile.input_text("john_doe")
    print(f"Username entered: {username_input.success}")  # Output: Username entered: True
    
    # Tap the password field.
    password_tap = session.mobile.tap(x=300, y=500)
    print(f"Password field focused: {password_tap.success}")  # Output: Password field focused: True
    
    # Enter the password.
    password_input = session.mobile.input_text("secure_password")
    print(f"Password entered: {password_input.success}")  # Output: Password entered: True
    
    # Tap the logon button.
    login_tap = session.mobile.tap(x=300, y=650)
    print(f"Login button pressed: {login_tap.success}")  # Output: Login button pressed: True
    
finally:
    agent_bay.delete(session)

UI element discovery

from agentbay import AgentBay
from agentbay.session_params import CreateSessionParams

agent_bay = AgentBay()
session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

try:
    # Get all clickable elements.
    result = session.mobile.get_clickable_ui_elements(timeout_ms=3000)
    
    if result.success:
        print(f"Found {len(result.elements)} clickable elements")  # Output: Found 3 clickable elements
        
        # Analyze the elements to find the target.
        for i, element in enumerate(result.elements):
            print(f"Element {i+1}: {element}")  
            # Example output:
            # Element 1: UI element with interaction capabilities
            # Element 2: UI element with interaction capabilities  
            # Element 3: UI element with interaction capabilities
    
    # Take a screenshot for verification.
    screenshot = session.mobile.screenshot()
    if screenshot.success:
        screenshot_url = screenshot.data
        print(f"Screenshot URL: {screenshot_url}")
        # Output: Screenshot URL: https://***.***.aliyuncs.com/***/screenshot_1234567890.png?***
    
finally:
    agent_bay.delete(session)

Content scrolling

from agentbay import AgentBay
from agentbay.session_params import CreateSessionParams

agent_bay = AgentBay()
session_params = CreateSessionParams(image_id="mobile_latest")
session = agent_bay.create(session_params).session

try:
    # Scroll down multiple times.
    for i in range(3):
        scroll_result = session.mobile.swipe(
            start_x=300,
            start_y=800,
            end_x=300,
            end_y=200,
            duration_ms=400
        )
        print(f"Scroll down {i+1}: {scroll_result.success}")  # Output: Scroll down 1: True, etc.
        
        # Pause briefly between scrolls.
        import time
        time.sleep(1)
    
    # Scroll up.
    up_result = session.mobile.swipe(
        start_x=300,
        start_y=200,
        end_x=300,
        end_y=800,
        duration_ms=400
    )
    print(f"Scroll up result: {up_result.success}")  # Output: Scroll up result: True
    
finally:
    agent_bay.delete(session)

FAQ

1. "Tool not found" error

   • Ensure you are using a cloud phone OS image, such as image_id="mobile_latest".

   • Verify that the session was created successfully.

   • Verify that the API key and endpoint are configured correctly.

2. Hardware keypress operations

   • Key events are sent directly to the Android system.

   • Check result.success to verify the key was sent.

   • Error handling example:

     result = session.mobile.send_key(KeyCode.HOME)
     if not result.success:
         print(f"Key press failed: {result.error_message}")
     

3. UI element detection returns an empty result

   • Increase the timeout_ms value.

   • Capture a screenshot to check the current UI state.

   • Ensure the target screen has finished loading.

4. Screenshot URLs

   • The screenshot returns an OSS URL, not image data.

   • result.data contains the download URL.

   • Use the URL to download the screenshot if needed.

5. Swipe gesture does not work as expected

   • Verify the coordinates are within screen boundaries.

   • Adjust duration_ms for different gesture speeds.

   • Ensure the start and end coordinates form a meaningful swipe path.