Handsfree Windows Control

Guide skill for controlling native Windows apps (UIA) and web browsers (Playwright) via the handsfree-windows CLI. Use when you need to automate or test desk...

Security Scan

The skill's files and runtime instructions are consistent with its stated purpose (installing and using a Windows UIA + Playwright CLI); nothing in the package looks malicious, but it will clone a public GitHub repo, pip-install in editable mode, download large Playwright browser binaries, and persist browser profiles/cookies — review and trust the upstream repo before proceeding.

Purpose & Capability

The skill's name/description match the provided SKILL.md and helper scripts: it installs and runs a handsfree-windows CLI for desktop and browser automation. Minor mismatch: registry metadata lists no required binaries/env but the installer expects Python (3.10+), git, and pip to be available.

Instruction Scope

SKILL.md and scripts only perform tasks relevant to installing and validating the CLI and Playwright browsers (clone repo, pip install -e, run playwright install, run checks). The docs explicitly warn about stored browser profiles and ask humans to confirm destructive actions.

Install Mechanism

Install is an included Python script that clones https://github.com/lijinlar/handsfree-windows.git and runs pip install -e, then optionally downloads Playwright Chromium from cdn.playwright.dev. These are standard sources (GitHub, PyPI, Playwright CDN) — not high-risk, but the installer writes source to ~/.handsfree-windows and creates editable pip links (site-packages egg-link).

Credentials

No environment variables or external credentials are requested. The persistent state (browser profiles, browser-state.json, Playwright binaries) is proportionate to browser automation; the skill warns that cookies/login sessions are stored locally.

Persistence & Privilege

The skill does not request always:true and does not alter other skills. It does persist files in the user's home directory and performs a pip editable install (modifies Python site-packages via an egg-link). This is expected for a CLI but is a persistent change the user should be aware of.

Assessment

Before installing: verify you trust the upstream repository (https://github.com/lijinlar/handsfree-windows), because the installer will clone that repo and pip install it in editable mode. Ensure you have Python 3.10+, git, and pip available. Be aware the installer may download ~200–800 MB of Playwright browser binaries and will create persistent browser profiles (cookies and login sessions) under ~/.handsfree-windows; those can contain sensitive data — delete the profile directory to clear saved sessions. If unsure, run the included scripts manually in a disposable environment (or inspect the cloned repo) rather than letting an agent run setup automatically. To remove later: pip uninstall handsfree-windows and delete ~/.handsfree-windows as documented.

A guide skill for automating native Windows apps (UIA) and web browsers (Playwright) via the handsfree-windows CLI.

First use: auto-setup

On first use, run setup before issuing any other commands:

python scripts/setup.py

This will:

1. Clone handsfree-windows from GitHub into ~/.handsfree-windows/cli/ (public repo, read-only)
2. Install it via pip install -e (standard pip editable install)
3. Install Playwright + Chromium browser binaries (~200 MB one-time download from cdn.playwright.dev)
4. Run check_setup.py to verify everything is working

To skip browser installation (browser-* commands will not work):

python scripts/setup.py --no-browser

To install to a custom directory:

python scripts/setup.py --install-dir "C:\your\preferred\path"

Already installed? Verify anytime:

python scripts/check_setup.py

What is written to disk (transparent)

• CLI source code: ~/.handsfree-windows/cli/ (or --install-dir)
• pip editable link: standard site-packages egg-link (pip managed)
• Browser persistent profiles: ~/.handsfree-windows/browser-profiles/<engine>/ Contains cookies and login sessions. Delete to reset browser auth.
• Browser session state: ~/.handsfree-windows/browser-state.json (last visited URL)
• Playwright browser binaries: ~\AppData\Local\ms-playwright\ (~800 MB, Windows)

To fully remove everything:

pip uninstall handsfree-windows -y
Remove-Item -Recurse -Force "$env:USERPROFILE\.handsfree-windows"

---

Core rules

• Do not guess UI controls. Run hf tree or hf inspect first, then act on what is actually there.
• Do not type credentials. Navigate to login screens; let the human complete auth.
• Prefer UIA selectors (name + control_type) over raw coordinates.
• Use drag-canvas only for canvas/ink surfaces (Paint, drawing apps, etc.).
• For destructive actions (delete, submit, send), ask the human for confirmation first.

---

Workflow: Desktop app (UIA)

# Launch any installed app
hf start --app "Outlook"

# Find the window
hf list-windows --json

# Discover controls (no guessing)
hf tree --title-regex "Outlook" --depth 10 --max-nodes 30000

# Act on what was found
hf click --title "Outlook" --name "New mail" --control-type "Button"

# Inspect element under cursor
hf inspect --json

Workflow: Browser (Playwright)

# Open URL - login sessions saved in profile automatically
hf browser-open --url "https://example.com"

# Inspect page before acting
hf browser-snapshot --fmt text

# Act
hf browser-click --text "Sign in"
hf browser-type --selector "#email" --text "[e-mail]"

# Verify
hf browser-screenshot --out result.png

Mixed macro (desktop + web in one YAML)

- action: start
  args:
    app: "My Desktop App"

- action: browser-open
  args:
    url: "https://app.example.com"
    headless: false

- action: browser-click
  args:
    text: "Get Started"

- action: sleep
  args:
    seconds: 1

Run with: hf run macro.yaml

---

References

• Full command reference + selector schema: references/api_reference.md