Desktop application automation

Method summary

Coordinate system

All operations use pixel coordinates with origin (0, 0) at the top-left corner. X increases rightward, Y increases downward. Use get_screen_size() to get screen dimensions and get_cursor_position() for current cursor location.

Result objects

Computer Use methods return one of two result types:

BoolResult (returned by mouse clicks, movement, scrolling, text input, and key operations):

OperationResult (returned by screenshot, get_cursor_position, and get_screen_size):

click_mouse

Clicks at specified screen coordinates.

Signature

session.computer.click_mouse(x, y, button=MouseButton.LEFT)

Parameters

MouseButton enum values

Import: from agentbay import MouseButton

Returns: A BoolResult object. Check result.success to confirm the click was registered.

Example

from agentbay import MouseButton

# Left-click (default button)
result = session.computer.click_mouse(x=500, y=300)
if result.success:
    print("Left-click successful")
# Output: Left-click successful

# Right-click
result = session.computer.click_mouse(x=500, y=300, button=MouseButton.RIGHT)
if result.success:
    print("Right-click successful")
# Output: Right-click successful

# Middle-click
result = session.computer.click_mouse(x=500, y=300, button=MouseButton.MIDDLE)
if result.success:
    print("Middle-click successful")
# Output: Middle-click successful

# Double left-click
result = session.computer.click_mouse(x=500, y=300, button=MouseButton.DOUBLE_LEFT)
if result.success:
    print("Double left-click successful")
# Output: Double left-click successful

move_mouse

Moves the cursor to specified coordinates without clicking.

Signature

session.computer.move_mouse(x, y)

Parameters

Returns: A BoolResult object.

Example

result = session.computer.move_mouse(x=600, y=400)
if result.success:
    print("Mouse move successful")
# Output: Mouse move successful

drag_mouse

Drags from one point to another while holding the specified button.

Signature

session.computer.drag_mouse(from_x, from_y, to_x, to_y, button=MouseButton.LEFT)

Parameters

Supported button values for drag: MouseButton.LEFT, MouseButton.RIGHT, MouseButton.MIDDLE.

MouseButton.DOUBLE_LEFT is not supported for drag operations.

Returns: A BoolResult object.

Example

from agentbay import MouseButton

result = session.computer.drag_mouse(
    from_x=100,
    from_y=100,
    to_x=200,
    to_y=200,
    button=MouseButton.LEFT
)
if result.success:
    print("Drag operation successful")
# Output: Drag operation successful

scroll

Scrolls the mouse wheel at a specified screen location.

Signature

session.computer.scroll(x, y, direction=ScrollDirection.UP, amount=1)

Parameters

ScrollDirection enum values

Import: from agentbay import ScrollDirection

Returns: A BoolResult object.

Example

from agentbay import ScrollDirection

# Scroll up
result = session.computer.scroll(x=500, y=500, direction=ScrollDirection.UP, amount=3)
if result.success:
    print("Scroll up successful")
# Output: Scroll up successful

# Scroll down
result = session.computer.scroll(x=500, y=500, direction=ScrollDirection.DOWN, amount=5)
if result.success:
    print("Scroll down successful")
# Output: Scroll down successful

get_cursor_position

Returns the current cursor position.

Signature

session.computer.get_cursor_position()

Parameters: None.

Returns: An OperationResult object. When result.success is True, result.data contains a JSON string with x and y fields.

Example

import json

result = session.computer.get_cursor_position()
if result.success:
    cursor_data = json.loads(result.data)
    print(f"Cursor position: x={cursor_data['x']}, y={cursor_data['y']}")
# Output: Cursor position: x=512, y=384

input_text

Types text at the current cursor position.

Signature

session.computer.input_text(text)

Parameters

Returns: A BoolResult object.

Example

result = session.computer.input_text("Hello AgentBay!")
if result.success:
    print("Text input successful")
# Output: Text input successful

press_keys

Sends one or more keys simultaneously, including modifier keys. Use for shortcuts like Ctrl+C or Alt+Tab.

Signature

session.computer.press_keys(keys, hold=False)

Parameters

Returns: A BoolResult object.

Example

# Press Ctrl+A to select all
result = session.computer.press_keys(keys=["Ctrl", "a"])
if result.success:
    print("Key press successful")
# Output: Key press successful

# Press Ctrl+C to copy
result = session.computer.press_keys(keys=["Ctrl", "c"])
if result.success:
    print("Copy command sent")
# Output: Copy command sent

release_keys

Releases keys previously held with press_keys(hold=True). Always release held keys to prevent interference with subsequent operations.

Signature

session.computer.release_keys(keys)

Parameters

Returns: A BoolResult object.

Example

# Hold down the Ctrl key
session.computer.press_keys(keys=["Ctrl"], hold=True)

# ... Perform other operations ...

# Release the Ctrl key
result = session.computer.release_keys(keys=["Ctrl"])
if result.success:
    print("Key release successful")
# Output: Key release successful

screenshot

Captures the screen and returns a download URL. The image is saved to Object Storage Service (OSS).

Signature

session.computer.screenshot()

Parameters: None.

Returns: An OperationResult object. When result.success is True, result.data contains the download URL for the screenshot image (not the raw image data).

Example

result = session.computer.screenshot()
if result.success:
    screenshot_url = result.data
    print(f"Screenshot URL: {screenshot_url}")
# Output: Screenshot URL: https://***.***.aliyuncs.com/***/screenshot_1234567890.png?***

get_screen_size

Returns screen dimensions and the display scale factor of the cloud computer.

Signature

session.computer.get_screen_size()

Parameters: None.

Returns: An OperationResult object. When result.success is True, result.data contains a JSON string with the following fields:

Example

import json

result = session.computer.get_screen_size()
if result.success:
    screen_data = json.loads(result.data)
    print(f"Screen width: {screen_data['width']}")
    print(f"Screen height: {screen_data['height']}")
    print(f"DPI scaling factor: {screen_data['dpiScalingFactor']}")
# Output: Screen width: 1024
# Output: Screen height: 768
# Output: DPI scaling factor: 1.0

"Tool not found" error

Symptom: A "Tool not found" error occurs when calling a Computer Use method.

Cause: The session is not running on a supported cloud computer image.

Solution: Set image_id to a supported value (windows_latest or linux_latest) when creating a session.

# Correct - use a supported image ID
session_params = CreateSessionParams(image_id="windows_latest")

Screenshot returns a URL, not image data

Symptom: result.data from screenshot() contains a URL string instead of raw image bytes.

Cause: Expected behavior. The screenshot() method stores images in OSS and returns a download URL.

Solution: Download the image from the returned URL using an HTTP client or browser.

Held keys not released

Symptom: Subsequent keyboard operations behave unexpectedly after using press_keys(hold=True).

Cause: Keys held with press_keys(hold=True) remain active until explicitly released.

Solution: Pair every press_keys(hold=True) call with a corresponding release_keys() call.

# Hold a key
session.computer.press_keys(keys=["Shift"], hold=True)

# Perform operations that need Shift held...

# Always release afterward
session.computer.release_keys(keys=["Shift"])

Coordinates out of bounds

Symptom: A mouse operation produces unexpected results or no visible effect.

Cause: Target coordinates may exceed screen bounds. The SDK does not validate coordinates client-side — they are sent directly to the cloud computer.

Solution: Call get_screen_size() to determine valid bounds, then ensure x and y stay within (0, 0) to (width, height).