Windows Control

Full Windows desktop control. Mouse, keyboard, screenshots - interact with any Windows application like a human.

Security Scan

The skill's code and runtime instructions match its stated purpose (full Windows desktop automation); there are no obvious network exfiltration hooks or unrelated credentials requested, but it grants powerful abilities (screenshots, reading windows, sending keystrokes/clicks) and has some operational gaps (no declared OS restriction, no dependency/install guidance, unknown source) that warrant caution.

Purpose & Capability

Name/description (Windows desktop control) align with the included scripts: they perform mouse/keyboard actions, window management, screenshots, UI automation and text extraction. Requested resources (no env vars, no external services) are appropriate for this purpose. One minor mismatch: the package/metadata do not set an OS restriction even though the code uses Windows-only libraries (pywinauto, pygetwindow) — this should be declared.

Instruction Scope

SKILL.md instructs running the included scripts to capture screenshots, enumerate windows, read UI elements and dialog contents, click and type. Those actions are exactly the skill's purpose, but they are high‑privilege from a data-sensitivity perspective (can capture arbitrary on-screen text, dialogs, and screenshots). The instructions do not reference reading unrelated files or environment variables, and they do not send data to external endpoints.

Install Mechanism

No install spec (instruction-only) so nothing is automatically downloaded/written during install — lower risk. However, the code depends on Python packages (pywinauto, pyautogui, pygetwindow, PIL, optional pytesseract) that are not declared in an install spec or requirements file; the lack of explicit dependency/install guidance could lead users to run unprepared or install packages from untrusted sources.

Credentials

No environment variables, credentials, or config paths are requested. The lack of secret/credential requirements is proportional to the described functionality.

Persistence & Privilege

always:false (no forced inclusion). The skill can be invoked autonomously (default platform behavior) — note that autonomous use plus the ability to control the desktop increases blast radius, but autonomous invocation alone is not a disqualifying issue here.

Assessment

This skill appears to do what it claims: full Windows GUI automation and reading of UI/dialog text. Before installing, consider: (1) Source trust — the package has no homepage and an unknown owner; inspect the code yourself or only use from a trusted publisher. (2) Platform — it requires Windows (pywinauto, pygetwindow); ensure you run it on Windows. (3) Dependencies — the README does not declare required Python packages; install only from trusted package indexes and pin versions. (4) Data sensitivity — the skill can screenshot screens and read UI/dialog text (including sensitive contents like emails, passwords visible on screen). Limit where and when the agent can run it and avoid running it on machines with sensitive data unless you trust the skill. (5) Autonomy — if you allow autonomous invocation, consider restricting or auditing automated runs because the skill can perform destructive actions (close windows, send keystrokes). If you want lower risk, run the scripts manually in an isolated environment first.

Windows Control Skill

Full desktop automation for Windows. Control mouse, keyboard, and screen like a human user.

Quick Start

All scripts are in skills/windows-control/scripts/

Screenshot

py screenshot.py > output.b64

Returns base64 PNG of entire screen.

Click

py click.py 500 300              # Left click at (500, 300)
py click.py 500 300 right        # Right click
py click.py 500 300 left 2       # Double click

Type Text

py type_text.py "Hello World"

Types text at current cursor position (10ms between keys).

Press Keys

py key_press.py "enter"
py key_press.py "ctrl+s"
py key_press.py "alt+tab"
py key_press.py "ctrl+shift+esc"

Move Mouse

py mouse_move.py 500 300

Moves mouse to coordinates (smooth 0.2s animation).

Scroll

py scroll.py up 5      # Scroll up 5 notches
py scroll.py down 10   # Scroll down 10 notches

Window Management (NEW!)

py focus_window.py "Chrome"           # Bring window to front
py minimize_window.py "Notepad"       # Minimize window
py maximize_window.py "VS Code"       # Maximize window
py close_window.py "Calculator"       # Close window
py get_active_window.py               # Get title of active window

Advanced Actions (NEW!)

# Click by text (No coordinates needed!)
py click_text.py "Save"               # Click "Save" button anywhere
py click_text.py "Submit" "Chrome"    # Click "Submit" in Chrome only

# Drag and Drop
py drag.py 100 100 500 300            # Drag from (100,100) to (500,300)

# Robust Automation (Wait/Find)
py wait_for_text.py "Ready" "App" 30  # Wait up to 30s for text
py wait_for_window.py "Notepad" 10    # Wait for window to appear
py find_text.py "Login" "Chrome"      # Get coordinates of text
py list_windows.py                    # List all open windows

Read Window Text

py read_window.py "Notepad"           # Read all text from Notepad
py read_window.py "Visual Studio"     # Read text from VS Code
py read_window.py "Chrome"            # Read text from browser

Uses Windows UI Automation to extract actual text (not OCR). Much faster and more accurate than screenshots!

Read UI Elements (NEW!)

py read_ui_elements.py "Chrome"               # All interactive elements
py read_ui_elements.py "Chrome" --buttons-only  # Just buttons
py read_ui_elements.py "Chrome" --links-only    # Just links
py read_ui_elements.py "Chrome" --json          # JSON output

Returns buttons, links, tabs, checkboxes, dropdowns with coordinates for clicking.

Read Webpage Content (NEW!)

py read_webpage.py                     # Read active browser
py read_webpage.py "Chrome"            # Target Chrome specifically
py read_webpage.py "Chrome" --buttons  # Include buttons
py read_webpage.py "Chrome" --links    # Include links with coords
py read_webpage.py "Chrome" --full     # All elements (inputs, images)
py read_webpage.py "Chrome" --json     # JSON output

Enhanced browser content extraction with headings, text, buttons, and links.

Handle Dialogs (NEW!)

# List all open dialogs
py handle_dialog.py list

# Read current dialog content
py handle_dialog.py read
py handle_dialog.py read --json

# Click button in dialog
py handle_dialog.py click "OK"
py handle_dialog.py click "Save"
py handle_dialog.py click "Yes"

# Type into dialog text field
py handle_dialog.py type "myfile.txt"
py handle_dialog.py type "C:\path\to\file" --field 0

# Dismiss dialog (auto-finds OK/Close/Cancel)
py handle_dialog.py dismiss

# Wait for dialog to appear
py handle_dialog.py wait --timeout 10
py handle_dialog.py wait "Save As" --timeout 5

Handles Save/Open dialogs, message boxes, alerts, confirmations, etc.

Click Element by Name (NEW!)

py click_element.py "Save"                    # Click "Save" anywhere
py click_element.py "OK" --window "Notepad"   # In specific window
py click_element.py "Submit" --type Button    # Only buttons
py click_element.py "File" --type MenuItem    # Menu items
py click_element.py --list                    # List clickable elements
py click_element.py --list --window "Chrome"  # List in specific window

Click buttons, links, menu items by name without needing coordinates.

Read Screen Region (OCR - Optional)

py read_region.py 100 100 500 300     # Read text from coordinates

Note: Requires Tesseract OCR installation. Use read_window.py instead for better results.

Workflow Pattern

1. Read window - Extract text from specific window (fast, accurate)
2. Read UI elements - Get buttons, links with coordinates
3. Screenshot (if needed) - See visual layout
4. Act - Click element by name or coordinates
5. Handle dialogs - Interact with popups/save dialogs
6. Read window - Verify changes

Screen Coordinates

• Origin (0, 0) is top-left corner
• Your screen: 2560x1440 (check with screenshot)
• Use coordinates from screenshot analysis

Examples

Open Notepad and type

# Press Windows key
py key_press.py "win"

# Type "notepad"
py type_text.py "notepad"

# Press Enter
py key_press.py "enter"

# Wait a moment, then type
py type_text.py "Hello from AI!"

# Save
py key_press.py "ctrl+s"

Click in VS Code

# Read current VS Code content
py read_window.py "Visual Studio Code"

# Click at specific location (e.g., file explorer)
py click.py 50 100

# Type filename
py type_text.py "test.js"

# Press Enter
py key_press.py "enter"

# Verify new file opened
py read_window.py "Visual Studio Code"

Monitor Notepad changes

# Read current content
py read_window.py "Notepad"

# User types something...

# Read updated content (no screenshot needed!)
py read_window.py "Notepad"

Text Reading Methods

Method 1: Windows UI Automation (BEST)

• Use read_window.py for any window
• Use read_ui_elements.py for buttons/links with coordinates
• Use read_webpage.py for browser content with structure
• Gets actual text data (not image-based)

Method 2: Click by Name (NEW)

• Use click_element.py to click buttons/links by name
• No coordinates needed - finds elements automatically
• Works across all windows or target specific window

Method 3: Dialog Handling (NEW)

• Use handle_dialog.py for popups, save dialogs, alerts
• Read dialog content, click buttons, type text
• Auto-dismiss with common buttons (OK, Cancel, etc.)

Method 4: Screenshot + Vision (Fallback)

• Take full screenshot
• AI reads text visually
• Slower but works for any content

Method 5: OCR (Optional)

• Use read_region.py with Tesseract
• Requires additional installation
• Good for images/PDFs with text

Safety Features

• pyautogui.FAILSAFE = True (move mouse to top-left to abort)
• Small delays between actions
• Smooth mouse movements (not instant jumps)

Requirements

• Python 3.11+
• pyautogui (installed ✅)
• pillow (installed ✅)

Tips

• Always screenshot first to see current state
• Coordinates are absolute (not relative to windows)
• Wait briefly after clicks for UI to update
• Use ctrl+z friendly actions when possible

---

Status: ✅ READY FOR USE (v2.0 - Dialog & UI Elements) Created: 2026-02-01 Updated: 2026-02-02