Atu Desktop Control

Full desktop automation - screenshots, mouse, keyboard, window management, clipboard, screen info

Desktop Control Skill

Automate the desktop: screenshots, mouse, keyboard, window management, clipboard, and screen info. All commands output JSON with "ok": true/false for reliable agent parsing.

Setup

Run the setup script to create a Python venv and install dependencies. The skill directory is wherever this SKILL.md lives — all paths below are relative to the skill root.

Windows (PowerShell):

powershell

powershell -ExecutionPolicy Bypass -File scripts\setup.ps1

Linux / macOS:

bash

bash scripts/setup.sh

How to Run

The Python executable lives in the venv. Resolve it relative to the skill directory:

OS	Python Path
Windows	.venv\Scripts\python.exe
Linux/Mac	.venv/bin/python

All commands follow this pattern:

text

<python> scripts/desktop.py <command> [subcommand] [args]

Agent shorthand — set the working directory to the skill root, then:

text

exec({ command: ".venv\\Scripts\\python.exe scripts\\desktop.py <command> [args]", workdir: "<skillPath>" })

Where <skillPath> is the absolute path to this skill's directory (the folder containing this SKILL.md).

Commands

Screenshot

bash

# Full screen (saves to captures/ dir, timestamped filename)
python scripts/desktop.py screenshot

# Save to specific path
python scripts/desktop.py screenshot -o C:\tmp\shot.png

# Region capture (left top width height)
python scripts/desktop.py screenshot --region 0 0 800 600

Output: {"ok": true, "path": "...", "width": 1920, "height": 1080}

Mouse

bash

# Current position
python scripts/desktop.py mouse pos

# Move to coordinates
python scripts/desktop.py mouse move 500 300
python scripts/desktop.py mouse move 500 300 --duration 0.5

# Click (left/right/middle, single/double/triple)
python scripts/desktop.py mouse click 500 300
python scripts/desktop.py mouse click 500 300 --button right
python scripts/desktop.py mouse click --clicks 2

# Drag from (100,100) to (400,400)
python scripts/desktop.py mouse drag 100 100 400 400 --duration 1.0

# Scroll (positive=up, negative=down)
python scripts/desktop.py mouse scroll 3
python scripts/desktop.py mouse scroll -5
python scripts/desktop.py mouse scroll 3 --direction horizontal

Keyboard

bash

# Type ASCII text
python scripts/desktop.py key type "hello world"

# Type with interval between keys
python scripts/desktop.py key type "slow typing" --interval 0.1

# Type Unicode / CJK text (auto-uses clipboard paste)
python scripts/desktop.py key type "你好世界"

# Press single key (repeat with --times)
python scripts/desktop.py key press enter
python scripts/desktop.py key press tab --times 3

# Hotkey combination
python scripts/desktop.py key hotkey ctrl c
python scripts/desktop.py key hotkey ctrl shift s
python scripts/desktop.py key hotkey alt f4

Note: For non-ASCII text (CJK, emoji, etc.), key type automatically uses clipboard paste via Ctrl+V.

Window

bash

# List all windows (title + hwnd)
python scripts/desktop.py window list

# Activate (bring to front) — matches title substring, case-insensitive
python scripts/desktop.py window activate "Chrome"
python scripts/desktop.py window activate 1234567   # by hwnd

# Minimize / Maximize
python scripts/desktop.py window minimize "Notepad"
python scripts/desktop.py window maximize "Code"

# Close a window
python scripts/desktop.py window close "Notepad"

# Get window info (position, size, state)
python scripts/desktop.py window info "Chrome"

# Resize a window (width height in pixels)
python scripts/desktop.py window resize "Notepad" 800 600

# Move a window (x y position)
python scripts/desktop.py window move "Notepad" 100 100

Clipboard

bash

# Read clipboard content
python scripts/desktop.py clipboard get

# Write to clipboard
python scripts/desktop.py clipboard set "copied text"

Screen

bash

# Get screen resolution
python scripts/desktop.py screen size

# Get pixel color at (x, y)
python scripts/desktop.py screen pixel 100 200

Pixel output: {"ok": true, "x": 100, "y": 200, "r": 255, "g": 128, "b": 0, "hex": "#ff8000"}

Wait

bash

# Wait for N seconds (useful in automation sequences)
python scripts/desktop.py wait 2.5

Version

bash

python scripts/desktop.py --version

Scenarios

"Take a full screenshot and show me the desktop"

text

exec: .venv\Scripts\python.exe scripts\desktop.py screenshot
→ returns JSON with path → use image tool to show the screenshot

"Open Notepad and type some text"

text

exec: .venv\Scripts\python.exe scripts\desktop.py key hotkey win r
exec: .venv\Scripts\python.exe scripts\desktop.py wait 0.5
exec: .venv\Scripts\python.exe scripts\desktop.py key type "notepad"
exec: .venv\Scripts\python.exe scripts\desktop.py key press enter
exec: .venv\Scripts\python.exe scripts\desktop.py wait 1
exec: .venv\Scripts\python.exe scripts\desktop.py key type "Hello from desktop-control!"

"Maximize the Chrome window"

text

exec: .venv\Scripts\python.exe scripts\desktop.py window maximize "Chrome"

"Read clipboard content"

text

exec: .venv\Scripts\python.exe scripts\desktop.py clipboard get

"Move and resize a window"

text

exec: .venv\Scripts\python.exe scripts\desktop.py window move "Notepad" 0 0
exec: .venv\Scripts\python.exe scripts\desktop.py window resize "Notepad" 1024 768

Safety

• Failsafe ON by default: Move mouse to top-left corner (0,0) to abort any pyautogui operation.
• Use --no-failsafe to disable (NOT recommended).
• All actions return structured JSON for audit trail.
• Screenshots saved locally only — no network requests.
• Captures directory: captures/ (relative to skill root).

Optional Dependencies

For advanced workflows, you may also install:

Package	Use Case
openpyxl	Read/write Excel files
python-docx	Read/write Word documents
pywin32	Advanced Windows COM automation

These are not installed by default setup scripts. Install manually if needed:

text

.venv\Scripts\pip install openpyxl python-docx pywin32

Troubleshooting

Problem	Solution
pyautogui not installed	Run scripts/setup.ps1 (Windows) or scripts/setup.sh (Linux/Mac)
Window not found	Use window list to see available windows; matching is case-insensitive substring
Failed to activate	Window may be minimized — the script tries restore() first, but some apps resist
Screenshot is black	Common with GPU-accelerated apps; try capturing a region instead
key type garbled for CJK	Should auto-use clipboard paste; verify pyperclip is installed
FAILSAFE triggered	Mouse hit (0,0) corner; this is intentional safety — reposition mouse and retry
Permission denied on Linux	pyautogui needs X11/Wayland access; run from a GUI session, not SSH

Limitations

• Windows-primary: Full feature set (window management, all shortcuts) works on Windows. Linux/macOS have partial support via pyautogui but pygetwindow behavior may differ.
• Requires GUI session: Must run in a desktop session with a display. Headless servers or SSH sessions without X forwarding will fail.
• Single monitor: screenshot captures the primary monitor by default. Multi-monitor capture requires --region.
• Admin windows: Cannot interact with windows running as Administrator from a non-admin Python process (Windows UAC).
• Screen scaling: DPI scaling (125%, 150%) may cause coordinate mismatches. Use screen size to verify actual resolution.