Control macOS GUI apps visually — take screenshots, click, scroll, type. Use when the user asks to interact with any Mac desktop application's graphical interface.
The screenshot command captures a window, uses Apple Vision OCR to detect all text elements, draws numbered annotations on the image, and returns both:
Annotated image at /tmp/mac_use.png — numbered green boxes around each detected text
Element list in JSON — [{num: 1, text: "Submit", at: [500, 200]}, {num: 2, text: "Cancel", at: [600, 200]}, ...] where at is the center point [x, y] on the 1000x1000 canvas (origin at top-left)
You receive both by calling Bash (gets JSON with element list) and then Read on /tmp/mac_use.png (gets the visual). Always do both so you can cross-reference the numbers with what you see.
Quick Reference
bash
# List all visible windows
python3 {baseDir}/scripts/mac_use.py list
# Screenshot + annotate (returns image + numbered element list)
python3 {baseDir}/scripts/mac_use.py screenshot <app> [--id N]
# Click element by number (primary click method)
python3 {baseDir}/scripts/mac_use.py clicknum <N>
# Click at canvas coordinates (fallback for unlabeled icons)
python3 {baseDir}/scripts/mac_use.py click --app <app> [--id N] <x> <y>
# Scroll inside a window
python3 {baseDir}/scripts/mac_use.py scroll --app <app> [--id N] <direction> <amount>
# Type text (uses clipboard paste — supports all languages)
python3 {baseDir}/scripts/mac_use.py type [--app <app>] "text here"
# Press key or combo
python3 {baseDir}/scripts/mac_use.py key [--app <app>] <combo>
Workflow
Open the target app with open -a "App Name" (optionally with a URL or file path)
Capture a window, detect text elements via OCR, annotate with numbered markers, and return the element list. The target window is automatically raised to the top before capture, so overlapping windows are handled.
<app>: fuzzy, case-insensitive match (e.g. "chrome" matches "Google Chrome")
--id N: target a specific window ID (required when multiple windows of the same app exist)
Returns JSON with:
file: path to annotated screenshot (/tmp/mac_use.png)
id, app, title, scale: window metadata
elements: array of {num, text, at} — the numbered clickable text elements, where at is [x, y] center coordinates on the 1000x1000 canvas (origin at top-left)
If multiple windows match, returns a list of windows instead — pick one and retry with --id
The image is 1000x1000 pixels with green bounding boxes and blue number badges
Element map is saved to /tmp/mac_use_elements.json for clicknum
clicknum
Click on a numbered element from the last screenshot. This is the primary click method.
Common keys: return, tab, escape, space, delete, backspace, up, down, left, right
Modifiers: cmd, ctrl, alt/opt, shift
Important Rules
Always screenshot before your first interaction with an app
Always screenshot after an action to verify the result
Always Read the screenshot image after running the screenshot command — you need both the element list AND the visual
Prefer clicknum over click — only use direct coordinates for unlabeled icons
Click before typing — ensure the correct input field has focus first
Multiple windows: if you get multiple_windows error, use list to see all windows, then pass --id
Popup windows (like WeChat mini-program panels) are separate windows with their own IDs — use list to find them and --id to target them
Wait after opening apps: use sleep 2-3 after open -a before taking a screenshot
Activate the app before screenshot/click: prepend osascript -e 'tell application "AppName" to activate' && sleep 1 when the target app may be behind other windows
Do not type passwords or secrets via this tool
Coordinate System (for fallback click only)
Screenshots are rendered onto a 1000x1000 canvas:
Origin (0, 0) is at the top-left corner
x increases left to right (0 = left edge, 1000 = right edge)
y increases top to bottom (0 = top edge, 1000 = bottom edge)
The app window is scaled to fit (aspect ratio preserved), centered, with dark gray padding
Example: Order food on Meituan in WeChat
bash
# 1. Open WeChat
open -a "WeChat"
sleep 3
# 2. Screenshot WeChat — find the mini program window
python3 {baseDir}/scripts/mac_use.py list
# → find the mini program window ID
# 3. Screenshot the mini program (annotated + element list)
python3 {baseDir}/scripts/mac_use.py screenshot 微信 --id 41266
# → returns: {"file": "/tmp/mac_use.png", "elements": [{num: 1, text: "搜索", at: [500, 200]}, ...]}
# → Read /tmp/mac_use.png to see annotated image
# 4. Click "搜索" (element #1)
python3 {baseDir}/scripts/mac_use.py clicknum 1
# 5. Type search query
python3 {baseDir}/scripts/mac_use.py type --app 微信 "炸鸡"
# 6. Press Enter
python3 {baseDir}/scripts/mac_use.py key --app 微信 return
sleep 2
# 7. Screenshot to see results
python3 {baseDir}/scripts/mac_use.py screenshot 微信 --id 41266
# → Read /tmp/mac_use.png, pick a restaurant by number
# 8. Click on a restaurant (e.g. element #5)
python3 {baseDir}/scripts/mac_use.py clicknum 5
