Wincontrol

AI remote control for Windows desktop. Captures screen on-demand via POST request and provides an HTTP API for mouse/keyboard actions.

WinControl Skill

AI remote control for Windows desktop. Captures screen on-demand via POST request and provides an HTTP API for mouse/keyboard actions.

What It Does

• On-Demand Capture: POST to /capture to save screenshot, returns file path
• Action API: Control mouse and keyboard via HTTP endpoints on port 8767
• Auto-Cleanup: Deletes screenshot on server shutdown (Ctrl+C)
• Cross-Platform: Works on native Windows or WSL
• Self-Contained: Single screenshot.jpg saved in skill folder, overwritten each time, auto-deleted on exit

Quick Start

Option 1: Native Windows (Recommended)

Run directly on Windows without WSL:

powershell

# Install dependencies (one-time)
pip install pywin32 pillow mss

# Start the server
cd %USERPROFILE%\.openclaw\workspace\skills\wincontrol
python server.py

Or use the provided batch file:

powershell

.\start.bat

Screenshots are saved as screenshot.jpg in the skill directory and auto-deleted when the server stops.

Option 2: WSL Integration

If you prefer WSL, the server runs on Windows Python but can be controlled from WSL:

bash

# From WSL
cd ~/.openclaw/workspace/skills/wincontrol
./start.sh

Or start manually with PowerShell 7:

bash

'/mnt/c/Program Files/PowerShell/7/pwsh.exe' -Command \
  "python //wsl.localhost/Ubuntu/home/\$USER/.openclaw/workspace/skills/wincontrol/server.py" &

Verify Installation

bash

# Health check
curl http://localhost:8767/ping
# Output: {"ok": true}

# Capture a screenshot
curl -X POST http://localhost:8767/capture
# Output: {"ok": true, "path": ".../screenshot.jpg"}

Native Windows: Open screenshot.jpg in the skill directory

WSL: Access via the skill folder path

File Structure

text

skills/wincontrol/
├── SKILL.md            # This file
├── server.py           # Main server (runs on Windows)
├── start.bat           # Start script (Native Windows)
├── start.sh            # Start script (WSL)
├── stop.sh             # Stop script (WSL)
└── screenshot.jpg      # Latest screenshot (auto-created, auto-cleaned)

API Reference

Capture Screen

bash

curl -X POST http://localhost:8767/capture

Returns: {"ok": true, "path": ".../screenshot.jpg", "quality": 90}

Each capture overwrites screenshot.jpg in the skill directory. The file is automatically deleted when the server stops.

Optional quality override:

bash

# Lower quality for faster capture/smaller file
curl -X POST http://localhost:8767/capture -H "Content-Type: application/json" -d '{"quality": 60}'

# Higher quality for detailed screenshots
curl -X POST http://localhost:8767/capture -d '{"quality": 95}'

Quality	File Size (1080p)	Use Case
60	~80KB	Fast capture, quick checks
90 (default)	~150KB	Clear UI text, general use
95	~200KB	Maximum detail, presentations

Mouse Actions

bash

# Move cursor (no click)
curl -X POST http://localhost:8767/move -d '{"x": 500, "y": 300}'

# Click
curl -X POST http://localhost:8767/click -d '{"x": 500, "y": 300}'

# Drag
curl -X POST http://localhost:8767/drag -d '{"x1": 100, "y1": 200, "x2": 300, "y2": 400}'

# Scroll
curl -X POST http://localhost:8767/scroll -d '{"x": 500, "y": 300, "direction": "down", "amount": 3}'

Keyboard Input

bash

# Type text
curl -X POST http://localhost:8767/enter -d '{"keys": ["Hello World"]}'

# Press special key
curl -X POST http://localhost:8767/enter -d '{"keys": ["Enter"]}'

# Key combination (Ctrl+C)
curl -X POST http://localhost:8767/enter -d '{"keys": ["Ctrl", "C"]}'

# Mixed sequence: type, press key, then combo
curl -X POST http://localhost:8767/enter -d '{"keys": ["Hello", "Enter", "Ctrl", "A"]}'

The /enter endpoint accepts a list of keys and handles them sequentially:

• Text strings → Typed as-is
• Special keys → Pressed once (Enter, Escape, Tab, etc.)
• Modifier + key → Treated as combination (Ctrl+C, Alt+Tab, etc.)

Special Keys Reference

Single keys:

• Enter, Return, Escape, Esc
• Backspace, Tab, Space
• Delete, Del
• Up, Down, Left, Right
• Home, End, PageUp, PageDown
• F1 through F12

Modifiers (combine with other keys):

• Ctrl, Control
• Alt
• Shift
• Win, Windows

Common Workflows

Capture and View (Native Windows)

powershell

# Using PowerShell
$response = Invoke-RestMethod -Uri "http://localhost:8767/capture" -Method Post
Start-Process $response.path

Capture and View (WSL)

bash

FILE=$(curl -s -X POST http://localhost:8767/capture | python3 -c "import sys,json; print(json.load(sys.stdin)['path'])")
read "$FILE"

Click and Verify

bash

# Click somewhere
curl -X POST http://localhost:8767/click -d '{"x": 500, "y": 300}'
sleep 0.5

# Capture to see result
curl -X POST http://localhost:8767/capture

Open Notepad and Type

bash

# Win+R to open Run
curl -X POST http://localhost:8767/enter -d '{"keys": ["Win", "R"]}'
sleep 0.5

# Type notepad and press Enter
curl -X POST http://localhost:8767/enter -d '{"keys": ["notepad", "Enter"]}'
sleep 1

# Type message
curl -X POST http://localhost:8767/enter -d '{"keys": ["Hello from WinControl!"]}'

Installation

Native Windows

1. Install Python 3 from python.org
2. Install dependencies:

   powershell

   pip install pywin32 pillow mss
   

3. Clone/download the skill to ~/.openclaw/workspace/skills/wincontrol/
4. Start the server:

   powershell

   python server.py
   

WSL2 (Legacy)

See WSL2 Configuration section below.

Frame Management

• Capture: On-demand via POST /capture
• Quality: 90% JPEG for clear text/UI
• File: screenshot.jpg in skill directory
• Behavior: Overwritten on each capture
• Auto-cleanup: Deleted when server stops (Ctrl+C)

To change quality, edit server.py:

python

QUALITY = 90      # 1-100

Stopping the Server

The server automatically cleans up the screenshot when stopped.

Native Windows

Press Ctrl+C in the PowerShell window, or:

powershell

# Find and stop the process
Get-Process python | Where-Object {$_.CommandLine -like '*wincontrol*'} | Stop-Process

WSL

bash

./stop.sh

Or manually:

bash

# Kill Python process on Windows using PowerShell 7
'/mnt/c/Program Files/PowerShell/7/pwsh.exe' -Command \
  "Get-Process python -ErrorAction SilentlyContinue | Where-Object {\$_.CommandLine -like '*wincontrol*'} | Stop-Process -Force"

Troubleshooting

Native Windows

Issue: pip install pywin32 fails

• Try: pip install pywin32 --upgrade --force-reinstall
• Or download from GitHub releases

Issue: Port 8767 already in use

• Find the process: netstat -ano | findstr :8767
• Kill it: taskkill /PID <PID> /F

Issue: Cannot access from WSL

• Ensure Windows Firewall allows Python through
• Try disabling Windows Defender Firewall temporarily for testing

WSL2

Prerequisites

1. WSL2 with Ubuntu (or your preferred distro)
2. Python 3 on Windows side
3. Dependencies: pywin32, pillow, mss (auto-installed)

Path Configuration

Screenshots are saved as screenshot.jpg in the skill folder, accessible from both Windows and WSL.

Troubleshooting

Issue: Server starts but curl fails

• Check if port 8767 is in use: lsof -i :8767
• Kill existing process: kill <PID>

Issue: Python module errors

• Manually install deps on Windows:

bash

'/mnt/c/Program Files/PowerShell/7/pwsh.exe' -Command "pip install pywin32 pillow mss"

How It Works

text

┌─────────────┐     ┌──────────────┐     ┌────────────────┐
│  Client     │────▶│   server.py  │────▶│  screenshot.jpg│
│  (curl/ps)  │     │  (localhost) │     │  (skill dir)   │
└─────────────┘     └──────────────┘     └────────────────┘
                            │
                     ┌─────┴─────┐
                     │  Port     │
                     │  8767     │
                     └───────────┘

The server runs on Windows Python. Screenshots are saved as screenshot.jpg in the skill directory and automatically deleted when the server stops (Ctrl+C).

Python Client Example

python

import requests
import time

API = "http://localhost:8767"

def capture():
    """Capture screen and return file path"""
    r = requests.post(f"{API}/capture")
    return r.json().get("path")

def click(x, y):
    requests.post(f"{API}/click", json={"x": x, "y": y})

def enter(keys):
    requests.post(f"{API}/enter", json={"keys": keys})

# Example workflow
if __name__ == "__main__":
    # Type text, press Enter, then select all
    enter(["Hello World", "Enter", "Ctrl", "A"])
    time.sleep(0.5)
    
    # Capture result
    screenshot_path = capture()
    print(f"Screenshot saved to: {screenshot_path}")

Security Notes

• Server binds to localhost only (not accessible from network)
• No authentication - anyone with local access can control your desktop
• Screenshot is auto-deleted when server stops
• Screenshot saved as single screenshot.jpg in skill folder, not system directories
• Be careful running this on shared machines

Integration with OpenClaw

Native Windows

javascript

// Capture and view
const result = await exec("curl -s -X POST http://localhost:8767/capture");
const data = JSON.parse(result.stdout);
// Path will be Windows format: C:\Users\...
await read(data.path);  // OpenClaw handles Windows paths

WSL

javascript

// Capture and view
const result = await exec("curl -s -X POST http://localhost:8767/capture");
const data = JSON.parse(result.stdout);
await read(data.path);  // Path: /mnt/c/Users/...

// Or take action then capture
await exec("curl -X POST http://localhost:8767/click -d '{\"x\":500,\"y\":300}'");
await exec("sleep 0.5");
const screenshot = await exec("curl -s -X POST http://localhost:8767/capture");

License

MIT-0

Version

2.0.1