System Controller

Control Windows desktop software, hardware, and IoT devices. This skill should be used when the user wants to open/close/resize windows, start/stop/monitor p...

Security Scan

The skill's code matches its stated Windows-focused purpose, but it includes runtime package installs and unrestricted HTTP/IO capabilities combined with system-information and control functions that could be misused to collect and send sensitive data — review before installing or run in an isolated environment.

Purpose & Capability

The name/description claim Windows desktop, hardware, serial, and IoT control; the included scripts (window_manager, process_manager, hardware_controller, serial_comm, iot_controller, gui_controller) implement exactly those capabilities. Minor incoherence: SKILL.md and scripts assume Windows-only behavior (PowerShell, Win32 APIs) but registry metadata lists no OS restriction.

Instruction Scope

SKILL.md instructs the agent to run the shipped Python scripts via a managed Python path and to always ask for confirmation before destructive actions. The scripts themselves implement a wide range of actions (process kill, shutdown, network adapter disable, screenshots, OCR, serial comm). The agent must enforce the promised confirmation steps — nothing in the scripts forces user confirmation. The SKILL.md also hard-codes an example Python path under C:\Users\wave which may not exist on other hosts (likely a template artifact).

Install Mechanism

There is no registry install spec, but the scripts perform runtime dependency installation (pip install) for pyautogui, pillow, requests, pyserial, etc. Installing PyPI packages at runtime is common here but raises supply-chain risk (malicious or compromised packages). No downloads from unknown custom hosts were detected.

Credentials

The skill declares no required environment variables or credentials (ok), but it can collect detailed local info (process list, system/network info, USB devices, screenshots) and includes a generic HTTP client (iot_controller.http_* and Home Assistant support) that can send requests to arbitrary URLs. Combining local data collection with the ability to POST to arbitrary endpoints is a high-risk capability if misused or invoked without strict user confirmation or network restrictions.

Persistence & Privilege

The skill does not set always:true and does not request system-wide config or other skills' credentials. It will run only when invoked. It does write screenshot files under the skill path when taking screenshots (normal for this functionality).

What to consider before installing

This skill is functionally coherent for controlling Windows apps, hardware, serial devices, and local IoT, but it comes with significant power: it can list processes and hardware, take screenshots, install Python packages at runtime, and make arbitrary HTTP requests. Before installing: 1) Inspect the code yourself (you have the scripts) to verify no hard-coded external endpoints or data-exfiltration logic; 2) Only run it in a trusted, isolated environment (or restrict its network access) if you plan to allow it to make HTTP requests; 3) Ensure the agent enforces explicit user confirmation for any destructive action (shutdown, kill, disable network); 4) Consider preventing runtime pip installs or pre-provisioning required packages from trusted sources; 5) If you don't need generic HTTP or system-info collection, prefer a narrower skill or ask for a version that restricts network destinations and sensitive reads.

Unified control interface for Windows desktop software, system hardware, serial devices, and IoT platforms.

Architecture

User Request → Natural Language Understanding → Script Execution → System Action

Six control modules, each with a dedicated Python script:

Module	Script	Scope
Window Manager	scripts/window_manager.py	Desktop window control
Process Manager	scripts/process_manager.py	System process management
Hardware Controller	scripts/hardware_controller.py	System hardware settings
Serial Communication	scripts/serial_comm.py	Arduino / serial devices
IoT Controller	scripts/iot_controller.py	Smart home / HTTP APIs
GUI Controller	scripts/gui_controller.py	Mouse, keyboard, screenshot, OCR

All scripts are standalone CLI tools. No inter-script dependencies.

Execution Model

Prerequisites

• OS: Windows 10/11
• PowerShell: 5.1+ (built-in)
• Python: Managed runtime (auto-provided by WorkBuddy)
• Optional: pyserial (auto-installed by serial_comm.py), requests (auto-installed by iot_controller.py), pyautogui (auto-installed by gui_controller.py), pillow (auto-installed by gui_controller.py), pytesseract (optional, for OCR), nircmd (for precise volume control), tesseract-ocr (system-level, for Chinese OCR: choco install tesseract)

Python Path

Use the managed Python runtime for all script executions:

C:\Users\wave\.workbuddy\binaries\python\versions\3.13.12\python.exe

For GUI controller (which needs pyautogui/pillow), use the venv Python:

C:\Users\wave\.workbuddy\binaries\python\envs\default\Scripts\python.exe

If the venv does not exist, create it and install packages:

C:\Users\wave\.workbuddy\binaries\python\versions\3.13.12\python.exe -m venv C:\Users\wave\.workbuddy\binaries\python\envs\default
C:\Users\wave\.workbuddy\binaries\python\envs\default\Scripts\pip install pyautogui pillow pyserial requests

Script Path

All scripts are located at:

~/.workbuddy/skills/system-controller/scripts/

Execution Pattern

Always use execute_command to run scripts. Never try to run them inline.

Pattern:

{python_path} {script_path} {action} {flags}

Safety Rules

1. NEVER execute destructive actions without explicit user confirmation (shutdown, restart, kill processes, close windows, disable network adapters).
2. Always list/query first before taking action. Example: run list before close, list --name before kill.
3. Warn the user before power operations (shutdown, restart, sleep, hibernate) and require explicit confirmation.
4. Never disable critical network adapters (the one used for active internet connection) without warning.
5. For serial communication, always list ports first to confirm the correct port name.

Module Details

1. Window Manager (window_manager.py)

Control desktop application windows via Windows UI Automation and Win32 API.

Capabilities: list, activate, close, minimize, maximize, resize, send-keys

Decision flow:

1. User says "open/close/focus X" → list to find the window → confirm with user → execute action
2. User says "resize/move X" → list to find PID → resize with coordinates
3. User says "type/send X to Y" → activate target window → send-keys

Common examples:

• "打开记事本" → process_manager.py start "notepad.exe"
• "关闭Chrome" → window_manager.py list → find Chrome → window_manager.py close --title "Chrome"
• "把微信调到前台" → window_manager.py activate --title "微信"
• "全屏当前窗口" → window_manager.py maximize --title "..."

2. Process Manager (process_manager.py)

List, start, stop, and monitor system processes.

Capabilities: list, kill, start, info, system

Decision flow:

1. User says "查看进程" → list or list --name
2. User says "结束X进程" → list --name X → confirm → kill --name X
3. User says "启动X" → start "X"
4. User says "系统状态" → system

Common examples:

• "有哪些程序在运行" → process_manager.py system
• "关掉所有记事本" → process_manager.py kill --name notepad
• "启动VS Code" → process_manager.py start "code"

3. Hardware Controller (hardware_controller.py)

Control system hardware settings via PowerShell and WMI.

Capabilities:

• Volume: get, set, mute
• Screen: brightness (get/set), display info
• Power: lock, sleep, hibernate, shutdown, restart, cancel
• Network: list adapters, enable/disable, WiFi scan, network info
• USB: list devices

Decision flow:

1. User says "音量/声音" → volume commands
2. User says "亮度/屏幕" → screen/brightness commands
3. User says "关机/锁屏/睡眠" → power commands (always confirm)
4. User says "网络/WiFi" → network commands
5. User says "USB" → usb list

Common examples:

• "把音量调到50" → hardware_controller.py volume set --level 50
• "静音" → hardware_controller.py volume mute
• "屏幕调暗一点" → hardware_controller.py screen brightness --level 30
• "锁屏" → hardware_controller.py power lock
• "扫描WiFi" → hardware_controller.py network wifi

4. Serial Communication (serial_comm.py)

Communicate with Arduino, ESP32, and other serial devices via pyserial.

Capabilities: list ports, detect baud rate, send, receive, chat, monitor

Decision flow:

1. User says "串口/Arduino/COM" → list ports first
2. User says "发送到Arduino" → send --port COMx --data "..."
3. User says "读取传感器" → chat --port COMx --data "READ"

Auto-install: Automatically installs pyserial on first use.

Common examples:

• "有哪些串口" → serial_comm.py list
• "给Arduino发指令开灯" → serial_comm.py send --port COM3 --data "LED_ON"
• "读取温度传感器" → serial_comm.py chat --port COM3 --data "GET_TEMP"
• "监听串口数据" → serial_comm.py monitor --port COM3 --duration 30

5. IoT Controller (iot_controller.py)

Control smart home devices via Home Assistant REST API, Mijia, or generic HTTP endpoints.

Capabilities:

• Home Assistant: list entities, get state, turn on/off/toggle, call any service
• Generic HTTP: GET, POST, PUT to any REST endpoint
• Mijia: device discovery guidance

Decision flow:

1. User mentions "智能家居/Home Assistant/灯光/温度" → IoT commands
2. User says "控制设备/开关灯" → requires URL and token from user
3. User says "调用API" → generic HTTP commands

Auto-install: Automatically installs requests on first use.

Common examples:

• "列出所有智能设备" → iot_controller.py homeassistant --url ... --token ... list
• "打开客厅灯" → iot_controller.py homeassistant --url ... --token ... on --entity-id light.living_room
• "关掉卧室空调" → iot_controller.py homeassistant --url ... --token ... off --entity-id climate.bedroom
• "调用这个API" → iot_controller.py http --url ... get --path ...

6. GUI Controller (gui_controller.py)

Full GUI automation: mouse control, keyboard input, screenshots, OCR, and visual recognition.

Capabilities:

• Mouse: move, click (left/right/double), drag, scroll, get position
• Keyboard: type text, press hotkeys, key down/up (hold/release)
• Screenshot: full screen, region, active window, list saved screenshots
• OCR: extract text from screen regions (requires pytesseract or Windows OCR fallback)
• Visual: find image template on screen, click by image, find by color, get pixel color

Auto-install: Automatically installs pyautogui and pillow on first use.

Decision flow:

1. User says "click/鼠标点击" → determine position (by coordinates, image template, or color), then click
2. User says "type/输入文字" → keyboard type --text "..."
3. User says "screenshot/截图" → screenshot full or screenshot active-window
4. User says "read screen/OCR/识别文字" → visual ocr on the relevant region
5. User says "find/查找图标" → visual find --template "icon.png"
6. User says "scroll/滚动" → mouse scroll
7. User says "drag/拖拽" → mouse drag --start-x ... --start-y ... --end-x ... --end-y ...

Common examples:

• "截图" → gui_controller.py screenshot full
• "截取当前窗口" → gui_controller.py screenshot active-window
• "鼠标移到(500,300)" → gui_controller.py mouse move --x 500 --y 300
• "点击(500,300)" → gui_controller.py mouse click --x 500 --y 300
• "右键点击" → gui_controller.py mouse right-click --x 500 --y 300
• "双击" → gui_controller.py mouse double-click --x 500 --y 300
• "拖拽文件从(100,200)到(500,400)" → gui_controller.py mouse drag --start-x 100 --start-y 200 --end-x 500 --end-y 400
• "向下滚动" → gui_controller.py mouse scroll --direction down --clicks 10
• "输入Hello World" → gui_controller.py keyboard type --text "Hello World"
• "按Ctrl+C" → gui_controller.py keyboard press --keys "ctrl+c"
• "按Alt+Tab" → gui_controller.py keyboard press --keys "alt+tab"
• "识别屏幕上的文字" → gui_controller.py visual ocr
• "识别(100,100,800,600)区域的文字" → gui_controller.py visual ocr --x 100 --y 100 --width 800 --height 600
• "在屏幕上找这个图标" → gui_controller.py visual find --template "button.png"
• "找到并点击这个图片" → gui_controller.py visual click-image --template "submit.png"
• "获取(200,200)位置的颜色" → gui_controller.py visual pixel --x 200 --y 200
• "获取鼠标位置" → gui_controller.py mouse position
• "获取屏幕分辨率" → gui_controller.py screenshot size

Safety Rules:

• Mouse and keyboard operations affect whatever is on screen — always screenshot first if unsure.
• The failsafe is enabled: moving mouse to any screen corner will abort all pyautogui operations.
• OCR and image recognition are best-effort — accuracy depends on screen resolution, language, and image quality.
• For OCR, install Tesseract for best results: choco install tesseract then pip install pytesseract.

Handling Unknown Devices

When the user requests control of a device or software not directly covered:

1. Check if it can be started as a process: Use process_manager.py start "app_name"
2. Check if it has windows: Use window_manager.py list to find it
3. Take a screenshot: Use gui_controller.py screenshot full to see what's on screen
4. Use OCR to read text: Use gui_controller.py visual ocr to extract text from screen
5. Click by image matching: Save a template, then gui_controller.py visual click-image --template icon.png
6. Use mouse/keyboard: gui_controller.py mouse click --x 100 --y 200 for direct control
7. Check if it has an API: Use iot_controller.py http to interact
8. Check if it's USB-connected: Use hardware_controller.py usb list then serial_comm.py list
9. Suggest alternatives: If none of the above work, explain the limitation and suggest MCP Server or custom script

Quick Reference

For detailed command syntax, read references/command_reference.md.