System Controller

Control Windows desktop software, hardware, and IoT devices. This skill should be used when the user wants to open/close/resize windows, start/stop/monitor processes, adjust volume/brightness, manage power settings, control network adapters, communicate with Arduino or serial devices, or interact with smart home platforms like Home Assistant. Also covers mouse/keyboard automation, screenshots, OCR, and visual recognition. Trigger phrases include "open app", "close window", "set volume", "adjust brightness", "lock screen", "shutdown", "list processes", "send serial command", "control light", "connect Arduino", "enable WiFi", "USB devices", "click here", "screenshot", "OCR", "find image on screen", "type text", "mouse click", "drag", "scroll", or any request involving controlling software, hardware, or external devices.

Unified control interface for Windows desktop software, system hardware, serial devices, and IoT platforms.

Architecture

text

User Request → Natural Language Understanding → Script Execution → System Action

Six control modules, each with a dedicated Python script:

Module	Script	Scope
Window Manager	scripts/window_manager.py	Desktop window control
Process Manager	scripts/process_manager.py	System process management
Hardware Controller	scripts/hardware_controller.py	System hardware settings
Serial Communication	scripts/serial_comm.py	Arduino / serial devices
IoT Controller	scripts/iot_controller.py	Smart home / HTTP APIs
GUI Controller	scripts/gui_controller.py	Mouse, keyboard, screenshot, OCR

All scripts are standalone CLI tools. No inter-script dependencies.

Execution Model

Prerequisites

• OS: Windows 10/11
• PowerShell: 5.1+ (built-in)
• Python: Managed runtime (auto-provided by WorkBuddy)
• Optional: pyserial (auto-installed by serial_comm.py), requests (auto-installed by iot_controller.py), pyautogui (auto-installed by gui_controller.py), pillow (auto-installed by gui_controller.py), pytesseract (optional, for OCR), nircmd (for precise volume control), tesseract-ocr (system-level, for Chinese OCR: choco install tesseract)

Python Path

Use the managed Python runtime for all script executions:

text

C:\Users\wave\.workbuddy\binaries\python\versions\3.13.12\python.exe

For GUI controller (which needs pyautogui/pillow), use the venv Python:

text

C:\Users\wave\.workbuddy\binaries\python\envs\default\Scripts\python.exe

If the venv does not exist, create it and install packages:

text

C:\Users\wave\.workbuddy\binaries\python\versions\3.13.12\python.exe -m venv C:\Users\wave\.workbuddy\binaries\python\envs\default
C:\Users\wave\.workbuddy\binaries\python\envs\default\Scripts\pip install pyautogui pillow pyserial requests

Script Path

All scripts are located at:

text

~/.workbuddy/skills/system-controller/scripts/

Execution Pattern

Always use execute_command to run scripts. Never try to run them inline.

Pattern:

text

{python_path} {script_path} {action} {flags}

Safety Rules

1. NEVER execute destructive actions without explicit user confirmation (shutdown, restart, kill processes, close windows, disable network adapters).
2. Always list/query first before taking action. Example: run list before close, list --name before kill.
3. Warn the user before power operations (shutdown, restart, sleep, hibernate) and require explicit confirmation.
4. Never disable critical network adapters (the one used for active internet connection) without warning.
5. For serial communication, always list ports first to confirm the correct port name.

Module Details

1. Window Manager (window_manager.py)

Control desktop application windows via Windows UI Automation and Win32 API.

Capabilities: list, activate, close, minimize, maximize, resize, send-keys

Decision flow:

1. User says "open/close/focus X" → list to find the window → confirm with user → execute action
2. User says "resize/move X" → list to find PID → resize with coordinates
3. User says "type/send X to Y" → activate target window → send-keys

Common examples:

• "打开记事本" → process_manager.py start "notepad.exe"
• "关闭Chrome" → window_manager.py list → find Chrome → window_manager.py close --title "Chrome"
• "把微信调到前台" → window_manager.py activate --title "微信"
• "全屏当前窗口" → window_manager.py maximize --title "..."

2. Process Manager (process_manager.py)

List, start, stop, and monitor system processes.

Capabilities: list, kill, start, info, system

Decision flow:

1. User says "查看进程" → list or list --name
2. User says "结束X进程" → list --name X → confirm → kill --name X
3. User says "启动X" → start "X"
4. User says "系统状态" → system

Common examples:

• "有哪些程序在运行" → process_manager.py system
• "关掉所有记事本" → process_manager.py kill --name notepad
• "启动VS Code" → process_manager.py start "code"

3. Hardware Controller (hardware_controller.py)

Control system hardware settings via PowerShell and WMI.

Capabilities:

• Volume: get, set, mute
• Screen: brightness (get/set), display info
• Power: lock, sleep, hibernate, shutdown, restart, cancel
• Network: list adapters, enable/disable, WiFi scan, network info
• USB: list devices

Decision flow:

1. User says "音量/声音" → volume commands
2. User says "亮度/屏幕" → screen/brightness commands
3. User says "关机/锁屏/睡眠" → power commands (always confirm)
4. User says "网络/WiFi" → network commands
5. User says "USB" → usb list

Common examples:

• "把音量调到50" → hardware_controller.py volume set --level 50
• "静音" → hardware_controller.py volume mute
• "屏幕调暗一点" → hardware_controller.py screen brightness --level 30
• "锁屏" → hardware_controller.py power lock
• "扫描WiFi" → hardware_controller.py network wifi

4. Serial Communication (serial_comm.py)

Communicate with Arduino, ESP32, and other serial devices via pyserial.

Capabilities: list ports, detect baud rate, send, receive, chat, monitor

Decision flow:

1. User says "串口/Arduino/COM" → list ports first
2. User says "发送到Arduino" → send --port COMx --data "..."
3. User says "读取传感器" → chat --port COMx --data "READ"

Auto-install: Automatically installs pyserial on first use.

Common examples:

• "有哪些串口" → serial_comm.py list
• "给Arduino发指令开灯" → serial_comm.py send --port COM3 --data "LED_ON"
• "读取温度传感器" → serial_comm.py chat --port COM3 --data "GET_TEMP"
• "监听串口数据" → serial_comm.py monitor --port COM3 --duration 30

5. IoT Controller (iot_controller.py)

Control smart home devices via Home Assistant REST API, Mijia, or generic HTTP endpoints.

Capabilities:

• Home Assistant: list entities, get state, turn on/off/toggle, call any service
• Generic HTTP: GET, POST, PUT to any REST endpoint
• Mijia: device discovery guidance

Decision flow:

1. User mentions "智能家居/Home Assistant/灯光/温度" → IoT commands
2. User says "控制设备/开关灯" → requires URL and token from user
3. User says "调用API" → generic HTTP commands

Auto-install: Automatically installs requests on first use.

Common examples:

• "列出所有智能设备" → iot_controller.py homeassistant --url ... --token ... list
• "打开客厅灯" → iot_controller.py homeassistant --url ... --token ... on --entity-id light.living_room
• "关掉卧室空调" → iot_controller.py homeassistant --url ... --token ... off --entity-id climate.bedroom
• "调用这个API" → iot_controller.py http --url ... get --path ...

6. GUI Controller (gui_controller.py)

Full GUI automation: mouse control, keyboard input, screenshots, OCR, and visual recognition.

Capabilities:

• Mouse: move, click (left/right/double), drag, scroll, get position
• Keyboard: type text, press hotkeys, key down/up (hold/release)
• Screenshot: full screen, region, active window, list saved screenshots
• OCR: extract text from screen regions (requires pytesseract or Windows OCR fallback)
• Visual: find image template on screen, click by image, find by color, get pixel color

Auto-install: Automatically installs pyautogui and pillow on first use.

Decision flow:

1. User says "click/鼠标点击" → determine position (by coordinates, image template, or color), then click
2. User says "type/输入文字" → keyboard type --text "..."
3. User says "screenshot/截图" → screenshot full or screenshot active-window
4. User says "read screen/OCR/识别文字" → visual ocr on the relevant region
5. User says "find/查找图标" → visual find --template "icon.png"
6. User says "scroll/滚动" → mouse scroll
7. User says "drag/拖拽" → mouse drag --start-x ... --start-y ... --end-x ... --end-y ...

Common examples:

• "截图" → gui_controller.py screenshot full
• "截取当前窗口" → gui_controller.py screenshot active-window
• "鼠标移到(500,300)" → gui_controller.py mouse move --x 500 --y 300
• "点击(500,300)" → gui_controller.py mouse click --x 500 --y 300
• "右键点击" → gui_controller.py mouse right-click --x 500 --y 300
• "双击" → gui_controller.py mouse double-click --x 500 --y 300
• "拖拽文件从(100,200)到(500,400)" → gui_controller.py mouse drag --start-x 100 --start-y 200 --end-x 500 --end-y 400
• "向下滚动" → gui_controller.py mouse scroll --direction down --clicks 10
• "输入Hello World" → gui_controller.py keyboard type --text "Hello World"
• "按Ctrl+C" → gui_controller.py keyboard press --keys "ctrl+c"
• "按Alt+Tab" → gui_controller.py keyboard press --keys "alt+tab"
• "识别屏幕上的文字" → gui_controller.py visual ocr
• "识别(100,100,800,600)区域的文字" → gui_controller.py visual ocr --x 100 --y 100 --width 800 --height 600
• "在屏幕上找这个图标" → gui_controller.py visual find --template "button.png"
• "找到并点击这个图片" → gui_controller.py visual click-image --template "submit.png"
• "获取(200,200)位置的颜色" → gui_controller.py visual pixel --x 200 --y 200
• "获取鼠标位置" → gui_controller.py mouse position
• "获取屏幕分辨率" → gui_controller.py screenshot size

Safety Rules:

• Mouse and keyboard operations affect whatever is on screen — always screenshot first if unsure.
• The failsafe is enabled: moving mouse to any screen corner will abort all pyautogui operations.
• OCR and image recognition are best-effort — accuracy depends on screen resolution, language, and image quality.
• For OCR, install Tesseract for best results: choco install tesseract then pip install pytesseract.

Handling Unknown Devices

When the user requests control of a device or software not directly covered:

1. Check if it can be started as a process: Use process_manager.py start "app_name"
2. Check if it has windows: Use window_manager.py list to find it
3. Take a screenshot: Use gui_controller.py screenshot full to see what's on screen
4. Use OCR to read text: Use gui_controller.py visual ocr to extract text from screen
5. Click by image matching: Save a template, then gui_controller.py visual click-image --template icon.png
6. Use mouse/keyboard: gui_controller.py mouse click --x 100 --y 200 for direct control
7. Check if it has an API: Use iot_controller.py http to interact
8. Check if it's USB-connected: Use hardware_controller.py usb list then serial_comm.py list
9. Suggest alternatives: If none of the above work, explain the limitation and suggest MCP Server or custom script

Quick Reference

For detailed command syntax, read references/command_reference.md.