PeekabooV3 open claw的桌面自动化skill

Use Peekaboo for macOS desktop GUI automation, screen understanding, and MCP-backed local app control. Trigger when the user wants to inspect or control native macOS apps, Finder, Notes, Xcode, Terminal, system dialogs, menu bar items, Dock, Spaces, or other non-browser UI; when they ask to click, type, scroll, drag, switch apps, handle popups, or automate a desktop workflow on Mac; or when they explicitly mention Peekaboo. If Peekaboo is missing, detect that first and proactively install it for the user, then continue. Prefer this over browser automation for native macOS interfaces.

Peekaboo macOS automation

Use this skill to drive native macOS UI with Peekaboo.

Core rules

• Treat Peekaboo as an external local automation backend, not as a built-in tool.
• Prefer peekaboo CLI via exec.
• Detect installation before doing anything else.
• If Peekaboo is not installed, proactively install it for the user, then verify installation before continuing.
• Check permissions before first real automation step.
• Prefer semantic actions over coordinate clicks whenever possible.
• Use browser automation instead for web-page DOM tasks.

Detection and installation

Run these checks in order:

command -v peekaboo
peekaboo --version

If peekaboo is missing, install it proactively:

brew install steipete/tap/peekaboo

After installation, verify again:

command -v peekaboo && peekaboo --version

If Homebrew is missing, check for it first:

command -v brew

If brew is also missing, ask one concise question before installing Homebrew or using another path.

Permission check

Before GUI actions, check permissions:

peekaboo permissions status

If Screen Recording or Accessibility is missing, tell the user exactly what is missing and pause until granted. Do not pretend automation can succeed without them.

For a quick readiness check, you can run:

~/.openclaw/skills/peekaboo-macos-automation/scripts/doctor.sh

Preferred workflow

1. Detect/install Peekaboo.
2. Check permissions.
3. If the task targets a specific native app window, bring that app to the foreground first.
4. Inspect the target UI.
5. Prefer semantic action APIs.
6. Fall back to synthetic input only when needed.
7. Re-inspect after state-changing actions when the next step depends on fresh UI state.

Before see, click, set-value, or other window-level actions on a named app, do this first:

peekaboo app switch --to "AppName"

If needed, unhide or launch it first:

peekaboo app unhide --app "AppName"
peekaboo app launch "AppName"

Do not assume a running app has a capturable front window. Bring it forward first, then inspect it.

Inspection patterns

Use structured inspection first. Start broad, then narrow:

peekaboo list apps
peekaboo app switch --to "Safari"
peekaboo see --app "Safari" --json
peekaboo image --mode screen --retina --path ~/Desktop/peekaboo-screen.png

When a command returns a snapshot_id, reuse it for follow-up actions.

A good first-pass sequence is:

peekaboo list apps
peekaboo app switch --to "TargetApp"
peekaboo see --app "TargetApp" --json

Action preference order

Prefer actions in this order:

1. set-value
2. perform-action
3. click --on <id/query>
4. type
5. coordinate-based actions only as a fallback

Examples:

peekaboo set-value --on T1 --value "hello" --snapshot "$SNAPSHOT"
peekaboo perform-action --on B1 --action AXPress --snapshot "$SNAPSHOT"
peekaboo click --on "Save" --snapshot "$SNAPSHOT"
peekaboo type --text "hello world"

Common commands

• Focus + inspect UI: peekaboo app switch --to "AppName" && peekaboo see --app "AppName" --json
• List windows: prefer app-focused inspection over raw window listing
• Click element: peekaboo click --on "Label" --snapshot "$SNAPSHOT"
• Set text directly: peekaboo set-value --on T1 --value "text" --snapshot "$SNAPSHOT"
• Trigger AX action: peekaboo perform-action --on B1 --action AXPress --snapshot "$SNAPSHOT"
• Hotkey: peekaboo hotkey cmd,shift,t
• Switch apps: peekaboo app switch --to "Notes"
• Work with dialogs: peekaboo dialog list
• Menu bar: peekaboo menubar list
• Dock: peekaboo dock list
• Spaces: peekaboo space list
• Natural language agent: peekaboo agent "..."

When to ask first

Ask before:

• destructive or irreversible actions
• sending messages, emails, or posts through another app
• entering secrets, passwords, or 2FA codes
• bulk actions that could affect user data
• installing Homebrew if it is not present

Do not ask before installing Peekaboo itself if Homebrew is already available and the user asked to use or set up Peekaboo.

Troubleshooting

If peekaboo see fails:

• switch/focus the app first with peekaboo app switch --to "AppName"
• if the app may be hidden, run peekaboo app unhide --app "AppName"
• retry with peekaboo see --app frontmost --json
• retry with a simpler target or a full-screen capture

If a semantic action fails:

• refresh the snapshot
• try perform-action
• only then fall back to click or keyboard input

If permission output is unclear, report the raw missing permission names to the user.

Reporting

Keep updates short and concrete:

• whether Peekaboo was already installed or was installed now
• whether permissions are ready
• what app/UI was inspected
• what action succeeded or what blocked progress

Safety notes

• Native desktop automation is high-impact; avoid risky guesses.
• Do not claim success without command evidence.
• If Peekaboo errors, retry once with a simpler or more direct command before concluding.
• If the task is clearly a browser page task, use the browser tool instead of forcing Peekaboo.