Macos Gui Automation

Automate macOS GUI via screen capture and OCR text reading, mouse and keyboard control, window management, and app launching using cliclick, screencapture, t...

Security Scan

The skill's code and instructions match its stated purpose (macOS GUI automation); it requires Accessibility and Screen Recording permissions as expected and does not request unrelated credentials or network access.

Purpose & Capability

Name/description, SKILL.md, README, and script all align: they perform macOS GUI automation via cliclick, screencapture, tesseract, and osascript. Minor metadata mismatch: registry declares no OS restriction even though the skill only works on macOS and requires macOS-specific tools and permissions.

Instruction Scope

Runtime instructions and the helper script perform only screenshot capture, OCR, mouse/keyboard actions, and AppleScript calls. They explicitly require Accessibility and Screen Recording permissions (appropriate for this capability). These permissions grant powerful control (mouse/key events and full-screen reading) so granting them should be a conscious decision.

Install Mechanism

No install spec; this is instruction-only plus a small helper script. Nothing is downloaded or installed by the skill itself.

Credentials

The skill requests no environment variables, no credentials, and no config paths. The permissions it asks the user to grant (Accessibility, Screen Recording) are proportionate to GUI automation.

Persistence & Privilege

always:false and normal invocation settings. The skill does not modify other skills or system-wide configs and does not request permanent background presence.

Assessment

This skill appears to do what it says: it captures the screen, runs OCR, and sends mouse/keyboard/window commands. Before installing: (1) confirm you trust the source — the registry 'source' is unknown; (2) install cliclick/tesseract from trusted upstreams if needed; (3) be careful granting Accessibility and Screen Recording — those permissions let the skill control your mouse/keyboard and read screen contents (avoid running while sensitive windows or credentials are visible); (4) note the metadata lacks an OS restriction even though it only works on macOS; and (5) review the included script to ensure it meets your expectations or test in a restricted environment if you are unsure.

macOS GUI Automation Skill

Capabilities

• Screen Reading: Capture screenshots and OCR text
• Mouse Control: Click, double-click, right-click, move, drag
• Keyboard Input: Type text, press keys, shortcuts
• Window Management: List windows, focus, resize, close
• App Control: Launch, quit, bring to front

Tools Available

cliclick (Mouse/Keyboard)

# Click at coordinates
cliclick c:x,y

# Double click
cliclick dc:x,y

# Right click
cliclick rc:x,y

# Move mouse
cliclick m:x,y

# Drag from to
cliclick dr:x1,y1:x2,y2

# Type text
cliclick t:"text"

# Press key (Enter, Tab, etc.)
cliclick kp:enter

screencapture + tesseract (Screen Reading)

# Capture region to file
screencapture -R x,y,w,h /tmp/screen.png

# Capture full screen
screencapture /tmp/screen.png

# OCR from image
tesseract /tmp/screen.png stdout

# OCR with Chinese support
tesseract /tmp/screen.png stdout -l chi_sim+eng

osascript (AppleScript - Window/App Control)

# List all windows
osascript -e 'tell application "System Events" to get name of every process'

# Get window position/size
osascript -e 'tell application "Finder" to get bounds of window of front window'

# Click menu item
osascript -e 'tell application "System Events" to click menu item "Save" of menu "File" of process "TextEdit"'

Usage Patterns

Read Screen Text

# 1. Capture screen
screencapture -R 100,100,800,600 /tmp/region.png

# 2. OCR
tesseract /tmp/region.png stdout

Click Button at Position

cliclick c:500,300

Type in Field

# Click field first
cliclick c:400,200
# Then type
cliclick t:"hello world"
cliclick kp:enter

Find and Click (OCR + Click)

# 1. Capture and OCR
screencapture /tmp/screen.png
text=$(tesseract /tmp/screen.png stdout)

# 2. Parse coordinates from OCR result or use image recognition

# 3. Click
cliclick c:x,y

Limitations

• Coordinates are absolute (screen resolution dependent)
• No built-in image recognition (need to add OpenCV/sikuli for that)
• OCR accuracy depends on screen DPI and font
• Some apps may not be scriptable via AppleScript

Security Notes

• Requires Accessibility permissions in System Settings
• Run tccutil reset Accessibility if permissions issues
• Some apps (browsers, secure apps) may block automation