Desktop Automation Pro

Desktop GUI automation toolkit for browser, mobile devices, and native applications. 桌面 GUI 自动化工具包，支持浏览器、移动设备和原生应用。 Use this skill when: 使用此技能的场景： - Automati...

Security Scan

The skill's code, files, and runtime instructions align with its stated desktop/mobile GUI automation purpose and do not request unrelated credentials or perform hidden network operations.

Purpose & Capability

Name/description advertise browser, mobile and native app automation and the included files and instructions implement exactly that: pyautogui and pywinauto demos for local input/screenshot and guidance to use 'browser' and 'nodes' tools for browser and paired-device control. No unrelated credentials, binaries, or config paths are requested.

Instruction Scope

SKILL.md instructs the agent to use local Python scripts (exec) and platform tools ('browser', 'nodes', 'screenshot') to perform automation. That matches the declared purpose. Note: these instructions assume the agent environment provides the browser/nodes/screenshot tools and a Python interpreter; the skill also instructs installing packages with pip (pyautogui, pywinauto, pillow). The instructions do not ask the agent to read unrelated system files or export data to external endpoints.

Install Mechanism

There is no install specification; this is instruction-only with two included demo scripts. The SKILL.md suggests pip installing common packages from PyPI (pyautogui, pywinauto, pillow) which is proportionate and traceable. No downloads from arbitrary URLs or extract steps are present.

Credentials

The skill requires no environment variables, credentials, or config paths. The capabilities (local input simulation, launching apps, screenshots, paired-device control) legitimately do not require external secrets. No excessive or unrelated env/credential requests are present.

Persistence & Privilege

always is false and the skill is user-invocable; model invocation is allowed (platform default). This skill can simulate global mouse/keyboard events and start/connect to local apps, which are expected for desktop automation but are powerful actions—users should be aware of that operational risk before allowing autonomous runs.

Assessment

This skill appears to do what it says: local GUI automation using pyautogui/pywinauto plus instructions to use 'browser' and 'nodes' tools. Before installing or enabling it: (1) review and run the included demo scripts manually in a safe environment (VM) to confirm behavior; (2) only install the listed Python packages from the official PyPI registry; (3) be cautious about letting the agent invoke the skill autonomously, since input simulation and app-launching can perform destructive actions if misused; and (4) ensure the host provides the referenced 'browser'/'nodes'/'screenshot' tools or the workflows will fail.

Desktop GUI Automation / 桌面 GUI 自动化

Cross-platform GUI automation toolkit with browser control, mobile device management, and native app automation capabilities.

跨平台 GUI 自动化工具包，支持浏览器控制、移动设备管理和原生应用自动化。

Capabilities Overview / 能力概览

Platform 平台	Tool 工具	Use Case 使用场景
Browser 浏览器	browser	Web automation, screenshots, form filling
Mobile 移动设备	nodes	Device control, screen recording, camera
Desktop 桌面	screenshot skill	Window/region capture
Native 原生应用	Python scripts	Windows app automation, mouse/keyboard simulation

---

1. Browser Automation / 浏览器自动化

Use the browser tool for Chromium-based browser control.

使用 browser 工具控制 Chromium 浏览器。

Common Actions / 常用操作

# Open URL / 打开网址
browser action=open url="https://example.com"

# Take screenshot / 截图
browser action=screenshot

# Click element / 点击元素
browser action=act kind=click ref="e12"

# Type text / 输入文本
browser action=act kind=type text="Hello" ref="e15"

# Get page snapshot / 获取页面快照
browser action=snapshot

Workflow Example / 工作流示例

1. Start browser: browser action=start
2. Navigate: browser action=open url="..."
3. Snapshot to find elements: browser action=snapshot
4. Interact: browser action=act kind=click ref="..."
5. Capture result: browser action=screenshot

---

2. Mobile Device Control / 移动设备控制

Use the nodes tool for paired devices (Android/iOS/macOS).

使用 nodes 工具控制配对设备。

Available Actions / 可用操作

Action 操作	Description 描述
camera_snap	Take photo / 拍照
screen_record	Record screen / 录屏
location_get	Get GPS location / 获取位置
device_info	Device status / 设备状态
run	Execute command (iOS/macOS) / 执行命令

Example Usage / 使用示例

# List paired devices / 列出配对设备
nodes action=status

# Take screenshot / 截图
nodes action=camera_snap node="my-iphone" facing="back"

# Record screen / 录屏
nodes action=screen_record node="my-android" durationMs=10000

---

3. Native App Automation / 原生应用自动化

For Windows native applications, use Python scripts via exec.

Windows 原生应用通过 Python 脚本执行。

Option A: Mouse/Keyboard Simulation / 鼠标键盘模拟

Use pyautogui for global input simulation.

使用 pyautogui 进行全局输入模拟。

# scripts/pyautogui_demo.py
import pyautogui
import time

# Safety: move mouse to corner to abort / 安全：移到角落中止
pyautogui.FAILSAFE = True

# Get screen size / 获取屏幕尺寸
width, height = pyautogui.size()

# Move mouse / 移动鼠标
pyautogui.moveTo(100, 200, duration=0.5)

# Click / 点击
pyautogui.click(x=100, y=200)

# Type text / 输入文本
pyautogui.write('Hello World', interval=0.1)

# Hotkey / 快捷键
pyautogui.hotkey('ctrl', 'c')

# Screenshot / 截图
screenshot = pyautogui.screenshot()
screenshot.save('screenshot.png')

# Locate image on screen / 在屏幕上定位图片
position = pyautogui.locateOnScreen('button.png')
if position:
    pyautogui.click(position)

Option B: Windows App Control / Windows 应用控制

Use pywinauto for native Windows application control.

使用 pywinauto 控制 Windows 原生应用。

# scripts/pywinauto_demo.py
from pywinauto import Application
import time

# Method 1: Start new app / 方式1：启动新应用
app = Application().start('notepad.exe')

# Method 2: Connect to existing app / 方式2：连接已有应用
# app = Application().connect(title='Untitled - Notepad')

# Get main window / 获取主窗口
win = app.window(title='Untitled - Notepad')

# Type text / 输入文本
win.Edit.type_keys('Hello World{ENTER}', with_spaces=True)

# Menu operations / 菜单操作
win.menu_select('File->Save')

# Click button / 点击按钮
# win.Button1.click()

# Close window / 关闭窗口
win.close()

Installation / 安装依赖

pip install pyautogui pywinauto pillow

---

4. Screenshot Capture / 截图捕获

Use the screenshot skill for desktop capture.

使用 screenshot 技能进行桌面截图。

# Full screen / 全屏
Use screenshot skill

# Specific window / 特定窗口
Use browser or nodes tool

# Region / 区域
Use pyautogui.screenshot(region=(x, y, w, h))

---

Quick Reference / 快速参考

Task 任务	Recommended Tool 推荐工具
Web form filling 网页表单填充	browser
Web scraping 网页抓取	browser + snapshot
Mobile screen record 手机录屏	nodes
Windows app control Windows应用控制	pywinauto script
Global mouse/keyboard 全局鼠标键盘	pyautogui script
Desktop screenshot 桌面截图	screenshot skill

---

Best Practices / 最佳实践

1. Browser: Always use snapshot before interacting to get fresh element refs. 浏览器：交互前先 snapshot 获取最新元素引用。

2. Mobile: Check device status with nodes action=status first. 移动设备：先检查设备状态。

3. Native Apps: Enable FAILSAFE for pyautogui; add delays between actions. 原生应用：启用 FAILSAFE 安全机制；操作间添加延迟。

4. Error Handling: Use try-except and validate element presence before clicking. 错误处理：使用 try-except，点击前验证元素存在。

---

Troubleshooting / 故障排查

Issue 问题	Solution 解决方案
Browser won't start 浏览器无法启动	Check if Chrome/Edge is installed
Device not found 设备未找到	Run nodes action=status to check pairing
pyautogui click misses 点击偏移	Check DPI scaling; use pyautogui.position() to verify
pywinauto connection failed 连接失败	Run as admin; check app window title