FlexArm Robot Arm: Physical Tap, Swipe, OCR for Real Phones

Give AI agents physical eyes and a hand to operate any smartphone. Built-in 50+ atomic actions (click, swipe, OCR, screenshot). AI can call them directly. Also supports YAML scripts. Script Store: once one AI learns an app, all AI agents instantly master it. For strong-reasoning AI models: they can autonomously explore new apps via camera + robot arm — see, reason, act, verify. This closed-loop lets AI use any phone function, almost infinitely.

When to Use This Skill

An AI Agent MUST use this skill when the user wants to physically interact with a real smartphone screen using a robot arm. This includes:

• Clicking, tapping, swiping, or typing on a phone.
• Opening an app on a real phone.
• Finding text or icons on the phone screen (OCR or template matching).
• Automating a sequence of actions on a phone (e.g., daily check‑in, repeated workflow).
• Writing, running, or managing YAML scripts for automation.
• Any mention of: robot arm, FlexArm, calibration, camera view, physical click, or real phone.

Example user utterances that should trigger this skill:

• “Click the WeChat icon on my phone.”
• “Swipe down on the screen.”
• “Open TikTok on the real device.”
• “Find the ‘Settings’ button on the screen and tap it.”
• “Write a YAML script to check weather every morning at 8 AM.”
• “Run my automation script for Qishui Music.”
• “Use the robot arm to type ‘Hello’ into the search box.”

Do NOT use this skill if:

• The user asks a pure knowledge question (e.g., “What’s the capital of France?”).
• The user wants to operate a virtual or simulated phone (e.g., Android emulator without physical arm).
• The user simply asks for code generation without any intent to execute it on a real phone.

FlexArm Robot — AI Agent Skill Reference

Phone screen automation via robot arm + camera vision. Uses a camera to detect the phone screen area, maps pixel coordinates to physical arm coordinates, and performs precise clicks and swipes.

Environment & Initialization

All API calls in this skill depend on the RobotArmServer.exe service. Before using this skill, the following conditions must be met:

• Calibration Tool: RobotArmCalibration.exe Download: Official Release Page
  Latest version: v2.0.0
  Size: ~160 MB (compressed)

• Server Program: RobotArmServer.exe (included in RobotArmServer.zip) Download: FlexArm v2.0.1 Release
  Latest version: v2.0.0
  Size: ~231 MB (compressed)

• Installation:

  1. Download RobotArmServer.zip from the link above
  2. Extract to any directory (e.g., D:\FlexArm) — this becomes the project root
  3. Run RobotArmServer.exe as Administrator (first run requires admin privileges to install the driver)
  4. Verify: visit http://127.0.0.1:7826/api/health — should return {"ok":true}

• Directory Convention: All relative paths (e.g., scripts/, icons/) are relative to the project root above. Do not modify files inside the _internal/ directory.

⚠️ If the service is not running, this skill cannot perform any operations. Before starting a task, always check /api/health status.

Important: Fixed Port

All API requests must use port 7826, not 5000.

text

http://127.0.0.1:7826/api/*

The port is fixed at 7826 and cannot be changed. Flask's default port 5000 does not apply.

Important: Chinese Characters in curl

Do NOT use curl to send Chinese characters in JSON. curl corrupts UTF-8 encoding and the server won't correctly recognize Chinese keywords, causing lookup failures.

bash

# ❌ Wrong: curl corrupts Chinese characters in JSON
curl -X POST http://127.0.0.1:7826/api/find_text -d '{"text_keyword":"领取"}'

# ✅ Correct: use Python requests for Chinese parameters
python -c "import requests; r = requests.post('http://127.0.0.1:7826/api/find_text', json={'text_keyword': '领取'}); print(r.text)"

APIs with English-only parameters (e.g., detect_desktop, click_icon, run_script, click_at) may use curl. APIs involving Chinese keywords (find_text, click_text, detect_page page names) must use Python.

---

Complete API Index (54 endpoints)

All endpoints below are accessible via HTTP POST/GET at http://127.0.0.1:7826.

System & Status (8)

#	Method	Endpoint	Description
1	GET	/	Service root
2	GET	/api/health	Health check (service, arm, camera status)
3	GET	/api/arm_status	Arm status (COM port, service, calibration)
4	GET	/api/get_frame_info	Get frame dimensions
5	GET	/api/get_overlay	Get current overlay (vision match result)
6	GET	/api/get_phone_corners	Get phone screen 4-corner coordinates
7	GET	/api/is_phone_present	Check if phone is in frame
8	GET	/api/is_screen_on	Check if screen is on

Display & Control (5)

#	Method	Endpoint	Description
9	GET/POST	/api/show_window	Open camera display window
10	GET/POST	/api/hide_window	Close camera display window
11	POST	/api/toggle_phone_corners	Toggle phone screen border overlay
12	POST	/api/change_focus	Adjust camera focus (delta value)
13	GET	/api/screenshots	List historical screenshot files

Actions (17)

#	Method	Endpoint	Description
14	POST	/api/go_home	Home — return to desktop
15	POST	/api/go_back	Back navigation
16	POST	/api/go_forward	Forward navigation
17	POST	/api/reset	Reset robot arm to origin
18	POST	/api/clear_overlay	Clear vision match overlay boxes
19	GET/POST	/api/run_app	Launch a specified app
20	POST	/api/swipe_up	Swipe up (large/small)
21	POST	/api/swipe_down	Swipe down (large/small)
22	POST	/api/swipe_up_normal	Standard swipe up (~80% success)
23	POST	/api/swipe_down_normal	Standard swipe down
24	POST	/api/swipe	Custom swipe (start/end percentages)
25	POST	/api/close_all_apps	Close all background apps
26	POST	/api/click_icon	Template-matching icon click
27	POST	/api/click_icons	Click multiple icons sequentially
28	POST	/api/click_icon_many_times	Click same icon multiple times
29	POST	/api/click_text	OCR text search and click
30	POST	/api/click_at	Click at frame pixel coordinates
31	POST	/api/click	Click at phone percentage coordinates
32	POST	/api/click_roi	Click center of an ROI area
33	POST	/api/screenshot	Screenshot (save file / return base64)
34	POST	/api/reload_gestures	Reload gesture config (hot-reload)

Vision Detection (12)

#	Method	Endpoint	Description
35	POST	/api/find_template	Full-screen template matching
36	POST	/api/find_template_roi	ROI-based template matching
37	POST	/api/find_text	OCR text search
38	POST	/api/find_text_roi	ROI-based OCR text search
39	POST	/api/find_all_text	Recognize all text on screen
40	POST	/api/find_all_templates	All templates must match
41	POST	/api/find_any_template	Any template match is sufficient
42	POST	/api/count_template	Count template occurrences
43	POST	/api/detect_desktop	Detect current desktop page
44	POST	/api/detect_page	Detect current app page
45	POST	/api/wait_for_template	Poll until template appears
46	POST	/api/wait_for_page	Poll until target page appears

Script Control (4)

#	Method	Endpoint	Description
47	POST	/api/run_script	Execute YAML script (async)
48	GET	/api/script_status	Check if a script is running
49	GET	/api/script_progress	Get script execution progress
50	POST	/api/stop_script	Force-stop a running script

Configuration (3)

#	Method	Endpoint	Description
51	GET/PUT	/api/config/daily	Read/update daily automation config
52	GET/PUT	/api/config/app/<name>	Read/update app page config
53	GET/PUT	/api/config/gesture	Read/update gesture config

System Management (1)

#	Method	Endpoint	Description
54	POST	/api/shutdown	Graceful service shutdown

---

System Architecture

text

User / AI Agent
    │
    ├── HTTP API (POST/GET http://127.0.0.1:7826/api/*)
    │       Controls robot arm, camera, script execution
    │
    └── YAML Scripts (scripts/*.yaml)
            Define automation workflows (click icons, find text, loops, conditions)

For AI Agents: You cannot see the camera feed directly. Use these APIs to understand the phone screen state:

• GET /api/get_frame_info — frame metadata
• GET /api/is_phone_present — detect if phone is in frame
• GET /api/is_screen_on — detect if screen is lit
• POST /api/screenshot {"return_base64": true} — get base64 image data
• POST /api/detect_page — detect current page name

Core Principles:

• Arm / camera / script engine are exclusive resources — only one task may use them at a time
• All vision operations rely on template matching (icons) and OCR (text)
• Synchronous blocking — except /api/run_script, all API commands are synchronous and blocking. You must wait for the HTTP response (with "ok": true indicating completion) before sending the next command. run_script launches a background thread and returns immediately; monitor it with script_status, script_progress, stop_script
• Resource protection: action commands are rejected with "script is running" during script execution

✅ Safe to call while script runs	❌ Rejected while script runs
health, arm_status, script_status, script_progress	run_script (only one at a time)
get_frame_info, get_overlay, get_phone_corners	click, click_icon, click_text, click_at, click_roi
is_phone_present, is_screen_on	swipe, swipe_up, swipe_down, swipe_up_normal, swipe_down_normal
find_template, find_all_templates, find_any_template, count_template (cv2, thread-safe)	go_home, go_back, go_forward
screenshot (base64 or file)	find_text, find_all_text, find_text_roi (PaddleOCR singleton, not thread-safe)
wait_for_template (uses cv2 internally)	detect_desktop, detect_page, wait_for_page (may call OCR)
	reset, clear_overlay, close_all_apps, run_app, reload_gestures

• Scripts are interpreted, no compilation needed, edit-and-run
• The robot arm auto-resets after script completion

---

Quick Start: Hello FlexArm

Step 1: Confirm service is running

bash

# Start the service
RobotArmServer.exe
# Default port: 7826

On startup the program auto-detects and initializes:

1. Checks if the robot arm Windows service is running
2. If not, tries to auto-install and start it
3. Auto-detects the arm's COM port and connects
4. Auto-loads the latest calibration file (or starts auto-calibration if none exists)
5. Checks license (shows activation dialog if unlicensed)

Note: First-time use requires Administrator privileges to install the Windows service. Run RobotArmServer.exe as Administrator. In daily use, administrator rights are not needed if the service is already installed.

If service installation fails, the program still starts but the arm is unavailable. You can manually run robot-arm-service\安装.bat as Administrator to install the service.

Step 2: Test connectivity

bash

curl http://127.0.0.1:7826/api/health
# Returns: {"ok": true, "data": {"status": "running", ...}}

Step 3: Open camera window, confirm phone is visible

bash

curl -X POST http://127.0.0.1:7826/api/show_window -H "Content-Type: application/json" -d '{}'

After opening the window, you should see the phone screen. Press ESC to close.

For AI Agents: Check screen state without seeing the window

AI Agents cannot see the window. Use these APIs instead:

bash

# Get frame metadata
curl http://127.0.0.1:7826/api/get_frame_info
# Returns: {"ok":true,"data":{"width":960,"height":540}}

# Detect if phone is in frame (brightness check)
curl http://127.0.0.1:7826/api/is_phone_present
# Returns: {"ok":true,"data":{"present":true}}

# Detect if screen is on
curl http://127.0.0.1:7826/api/is_screen_on
# Returns: {"ok":true,"data":{"screen_on":true}}

# Get current frame (base64, parseable by AI)
curl -X POST http://127.0.0.1:7826/api/screenshot -H "Content-Type: application/json" -d '{"return_base64":true,"phone_only":true}'
# Returns: {"ok":true,"data":{"base64":"iVBORw0KGgoAAAANSUhEUg..."}}

# Save screenshot to file
curl -X POST http://127.0.0.1:7826/api/screenshot -H "Content-Type: application/json" -d '{"filename":"screenshots/test.png"}'
# Returns: {"ok":true,"data":{"path":"E:\\robot_arm\\screenshots\\test.png"}}

# List historical screenshots
curl http://127.0.0.1:7826/api/screenshots?limit=5
# Returns: [{"filename":"...","size":123456,"time":"2026-05-24 18:00:12"},...]

Step 4: Execute your first action — click center of screen

bash

curl -X POST http://127.0.0.1:7826/api/click \
  -H "Content-Type: application/json" \
  -d '{"x": 0.5, "y": 0.5}'

The robot arm automatically moves to and clicks the center of the phone screen (x: 0.5, y: 0.5 are percentage coordinates, range 0-1).

No icon templates or configuration needed — just calibrate and go.

Step 5: Write your first script

⚠️ Important: All YAML script files must use UTF-8 without BOM encoding. UTF-8 with BOM causes parse failures or garbled Chinese parameters.

Create scripts/hello_flexarm.yaml:

yaml

name: hello_flexarm
description: "First FlexArm script — experience clicking, swiping, waiting"

steps:
  # 1. Click center of screen
  - action: click
    x: 0.5
    y: 0.5

  # 2. Wait 1 second
  - action: wait
    seconds: 1

  # 3. Click bottom of screen (back navigation)
  - action: click
    x: 0.5
    y: 0.95

  # 4. Wait 2 seconds
  - action: wait
    seconds: 2

  # 5. Large swipe up (page turn)
  - action: swipe_up
    large: true

  # 6. Wait 1 second
  - action: wait
    seconds: 1

  # 7. Small swipe down
  - action: swipe_down
    large: false

  # 8. Click top-right corner
  - action: click
    x: 0.85
    y: 0.1

Run it:

bash

curl -X POST http://127.0.0.1:7826/api/run_script \
  -H "Content-Type: application/json" \
  -d '{"path": "scripts/hello_flexarm.yaml"}'

The example above only needs calibration — no icon templates or page config required.

To learn more advanced actions (icon clicks, OCR text clicks, page switching, conditional branches), continue reading to understand icon templates and page definitions.

---

Configuration: Teach the Program About Your Phone

Phone Desktop Config — scripts/configs/app_desktop.yaml

Required, filename is fixed as app_desktop.yaml. The program uses it to identify pages on the phone desktop.

yaml

app_name: Phone Desktop

pages:
  - name: desktop_page0           # Page name, arbitrary string
    min_match: 2                  # At least 2 features must match
    must_features:                # Required (all must pass)
      - name: Phone
        type: image
        path: icons/app_phone.png
        mask: false               # false=4-corner sampling, loose matching
      - name: Camera
        type: image
        path: icons/app_camera.png
        mask: false
    features:                     # Optional (need min_match to pass)
      - name: Messages
        type: image
        path: icons/app_message.png
        mask: false
      - name: Settings
        type: image
        path: icons/app_settings.png
        mask: false

  - name: desktop_page1
    min_match: 3
    must_features:
      - name: Phone
        type: image
        path: icons/app_phone.png
        mask: false
      - name: Camera
        type: image
        path: icons/app_camera.png
        mask: false
    features:
      - name: Qishui Music
        type: image
        path: icons/app_qishui.png
        mask: false
      - name: WeChat
        type: image
        path: icons/app_wechat.png
        mask: false

  - name: TaskSwitcher          # Multi-task switching view
    min_match: 0
    must_features:
      - name: RecentApps
        type: image
        path: icons/task_show.png
        mask: false
      - name: Trash
        type: image
        path: icons/task_delete.png
        mask: false
    features: []

Key Fields:

• must_features: all must match, or the page is skipped
• features: optional; passes if matched count >= min_match
• mask: false: uses 4-corner background sampling, tolerates icon size/position variation
• mask: true (or omitted): strict template matching, suitable for fixed UI elements

App Page Config — scripts/configs/app_xxx.yaml

One config file per app, defining all recognizable pages within that app.

yaml

app_name: Qishui Music

pages:
  - name: Music
    min_match: 1
    must_features: []
    features:
      - name: BottomPlayerBar
        type: image
        path: icons/qishui/music_playing.png
        mask: false

  - name: Rewards
    min_match: 1
    must_features:
      - name: RewardsTitle
        type: text
        text: "福利"
    features: []

When switch_page in a script doesn't match any page, it auto-executes the default branch — no separate config needed.

How to Create Icon Templates

1. Send an email with your license code to [e-mail] to get the helper tools, including screen capture and script testing features.

Or manually: take a phone screenshot → crop the icon → place it in the icons/ directory.

Icon Requirements:

• PNG format
• Icon body should be complete with 2-3px empty border
• Do not include dynamic content (countdowns, animations)
• Use lowercase English + underscores for filenames, e.g., app_qishui.png

---

YAML Script Language

Basic Structure

yaml

name: script_name
description: "Script description"

steps:
  - action: action_type
    param1: value1
    param2: value2

Supported Actions

action	Parameters	Description
click_icon	path, threshold, roi, mask	Template-matching icon click
click_icons	paths, interval	Click multiple icons sequentially (arm exits frame between clicks)
click_icon_many_times	path, count, interval	Click same spot multiple times without reset
dial_number	number, interval	Smart dialing (maps number to digit icons, supports # and *)
click_text	text, roi, min_score	OCR text search and click
click	x, y	Click phone percentage coordinates (0-1), ±30px random offset
click_at	cam_x, cam_y	Click frame pixel coordinates (precise, no offset)
click_roi	roi, label	Click center of ROI area (phone screen percentage)
find_all_text	roi, min_score	Recognize all text, return list + positions + confidence
swipe	sx, sy, ex, ey, steps, step_wait_ms	Custom swipe (start/end percentages)
swipe_up	large: true/false	Swipe up (large/small)
swipe_down	large: true/false	Swipe down (large/small)
swipe_up_normal	none	Standard swipe up (~80% success)
swipe_down_normal	none	Standard swipe down
go_home	max_retries	Return to desktop (detect → swipe up → detect loop)
go_back	none	Back navigation
go_forward	none	Forward navigation
reset	none	Reset robot arm
clear_overlay	none	Clear vision overlay boxes
run_app	app_name	Launch app (go_home → detect page → swipe → click icon)
close_all_apps	max_swipes	Close all background apps
screenshot	filename, phone_only, show_board, return_base64	Screenshot
reload_gesture	none	Reload gesture config (hot-reload)
set_video_to_coin	value	Set video-to-coin earning mode
wait	seconds	Wait (supports ranges like 2-5)
loop	count, steps	Loop sub-steps (supports random ranges like count: 3-5)
if_found	type, path/text, then, else	Conditional: if target found, run then; else run else
if_found_roi	type, path/text, roi, then, else	Same as above but with ROI-limited search
if_progress_stop	template, roi, then, else	Progress bar stall detection
if_video_to_coin	then, else	Branch based on video-to-coin mode state
if_random	chance, then, else	Random probability branch
detect_desktop	config	Detect if currently on desktop (no assertion)
detect_page	config	Detect current page name (no assertion)
is_screen_on	none	Check if screen is lit
assert_desktop	config	Must be on desktop, error if not
switch_page	config, cases	Detect page → match cases → default if no match
run_script	path	Execute sub-script (synchronous, returns on completion)
stop_loop	none	Break current loop
stop_script	none	Stop current script level (sub-script only stops itself)
log	message	Print log message

Parameter Details

click_icon

yaml

- action: click_icon
  path: icons/app_qishui.png     # Icon path (relative to project root)
  threshold: 0.75                # Match threshold (default 0.75)
  roi: [0.1, 0.2, 0.5, 0.6]     # Search area [sx, sy, ex, ey] (phone percentage, 0-1)
  mask: false                    # false=loose, true=strict (default true)

click_text

yaml

- action: click_text
  text: "领取"                   # Text to find
  roi: [0.3, 0.5, 0.7, 0.8]     # Optional search area
  min_score: 0.5                 # OCR minimum confidence (default 0.3)

click (percentage coordinates)

yaml

- action: click
  x: 0.5                         # X percentage (0=left, 1=right)
  y: 0.96                        # Y percentage (0=top, 1=bottom)

loop

yaml

- action: loop
  count: 10                      # Fixed count
  # count: 3-5                   # Random range also supported
  steps:
    - action: click_text
      text: "领取"
    - action: wait
      seconds: 2

if_found (conditional branch)

yaml

- action: if_found
  type: image/text               # image=template matching, text=OCR
  path: icons/qishui/cross.png   # For type=image
  text: "继续观看"               # For type=text
  roi: [0.7, 0.0, 1.0, 0.15]    # Optional search area
  then:
    - action: click_icon
      path: icons/qishui/cross.png
      roi: [0.7, 0.0, 1.0, 0.15]
  else:
    - action: wait
      seconds: 2

if_random (random branch)

yaml

- action: if_random
  chance: 0.4                    # 40% chance to take then branch
  then:
    - action: log
      message: "Took then branch"
  else:
    - action: log
      message: "Took else branch"

switch_page (page switching)

yaml

- action: switch_page
  config: scripts/configs/app_qishui.yaml   # Page config file
  cases:
    Music:                                   # When "Music" page matches
      - action: click
        x: 0.5
        y: 0.96
    Rewards:
      - action: click_text
        text: "福利"
    default:                                 # When no page matches
      - action: swipe_up
        large: true

Script Nesting

yaml

steps:
  - action: run_script
    path: qishui/run_ad_card.yaml     # Execute sub-script
  - action: wait
    seconds: 5

After a sub-script finishes, execution returns to the parent script.

Random Wait

yaml

- action: wait
  seconds: 2-5          # Random wait between 2~5 seconds

Execution Model

• Scripts execute sequentially — each action completes before the next begins
• loop repeats its sub-steps for the specified count
• switch_page iterates through all page definitions until a match is found
• run_script is a sub-call — returns to the parent when done
• stop_loop breaks the current loop
• stop_script stops the current level (in a sub-script, only stops that sub-script)
• After script completion or error, the robot arm auto-resets

---

HTTP API Reference

Basic Info:

• Address: http://127.0.0.1:7826
• All POST endpoints expect JSON body
• Success: {"ok": true, "data": {...}}
• Failure: {"ok": false, "error": "error message"}

1. Health Check

text

GET /api/health

Returns service status, port, uptime, etc.

2. Arm Status

text

GET /api/arm_status

Returns COM port, connection status, movement range, etc.

3. Camera Frame

Get frame info

text

GET /api/get_frame_info

Returns:

json

{"ok": true, "data": {"width": 540, "height": 960, "fps": 29.5}}

Show/hide window

text

POST /api/show_window
POST /api/hide_window

Detect phone presence

text

GET /api/is_phone_present
GET /api/is_phone_present?bright_threshold=60&bright_ratio=0.08

Returns:

json

{"ok": true, "data": {"present": true}}

Detect screen on/off

text

GET /api/is_screen_on
GET /api/is_screen_on?dark_threshold=30&dark_ratio=0.7

Returns:

json

{"ok": true, "data": {"screen_on": true}}

Toggle phone corners

text

POST /api/toggle_phone_corners

Overlays a green phone screen border on the display window.

Screenshot

text

POST /api/screenshot {"path": "screenshots/test.png"}    # Save to file
POST /api/screenshot {"return_base64": true}             # Return base64 (recommended for AI Agents)
POST /api/screenshot {"phone_only": true}                # Crop to phone area only
POST /api/screenshot {"show_board": true}                # Full view with ruler

List screenshots

text

GET /api/screenshots
GET /api/screenshots?limit=10

Returns:

json

{"ok":true,"data":[{"filename":"20260524_1800_phone.png","path":"E:\\robot_arm\\screenshots\\...","size":123456,"time":"2026-05-24 18:00:12"},...]}

4. Page Detection

Detect desktop

text

POST /api/detect_desktop {"desktop_config": "scripts/configs/app_desktop.yaml"}

Returns:

json

{"ok": true, "data": {"matched": true, "page_name": "desktop_page1", "score": 0.84}}

Detect specific page

text

POST /api/detect_page {"config_path": "scripts/configs/app_qishui.yaml", "threshold": 0.75}

Returns:

json

{"ok": true, "data": {"matched": true, "page_name": "Rewards", "score": 0.82}}

5. Vision Search

Find icon

text

POST /api/find_template
{"path": "icons/app_qishui.png", "threshold": 0.75, "roi": [0.1, 0.2, 0.5, 0.6], "auto_mask": false}

Returns:

json

{"ok": true, "data": {"x": 257, "y": 453, "w": 52, "h": 53, "score": 0.9446}}

Find text

text

POST /api/find_text {"text_keyword": "领取", "roi": [0.3, 0.5, 0.7, 0.8], "min_score": 0.5}

Returns:

json

{"ok": true, "data": {"x": 300, "y": 600, "w": 40, "h": 20, "text": "领取奖励", "score": 0.91}}

ROI-based text search

text

POST /api/find_text_roi {"roi": [0.0, 0.6, 1.0, 1.0], "text_keyword": "夸克", "min_score": 0.3}

Similar to find_text but requires roi (array format [sx, sy, ex, ey], 0-1).

ROI-based template matching

text

POST /api/find_template_roi {"path": "icons/app_qishui.png", "roi": [0.1, 0.2, 0.5, 0.6], "threshold": 0.75}

Similar to find_template but requires a roi region.

Adjust camera focus

text

POST /api/change_focus {"value": 2}     # Focus near +2
POST /api/change_focus {"value": -2}    # Focus far -2

Incremental adjustment (range 0~500). Returns {"ok": true, "data": {"focus": 310.0}}.

Find all text

text

POST /api/find_all_text {"min_score": 0.5}

Returns all recognized text on screen.

Performance Warning: find_all_text does a full-screen OCR scan using CPU inference. Time varies by text density:

• Sparse text (<20 items): ~15 seconds
• Dense text (novel apps, etc.): 90~120+ seconds

Best Practice: Use find_text with a specific roi whenever possible — it's orders of magnitude faster.

Find all templates

text

POST /api/find_all_templates {"template_paths": ["icons/a.png", "icons/b.png"], "threshold": 0.75}

Returns true only if all templates are found.

Find any template

text

POST /api/find_any_template {"template_paths": ["icons/a.png", "icons/b.png"], "threshold": 0.75}

Returns the first matched icon.

6. Actions

Return to desktop

text

POST /api/go_home {"max_retries": 5}

max_retries: maximum retry attempts, default 5 (detect desktop → swipe up → re-detect).

Back / Forward

text

POST /api/go_back
POST /api/go_forward

Reset

text

POST /api/reset

Swipe up / down

text

POST /api/swipe_up {"large": true}     # Large swipe up
POST /api/swipe_down {"large": false}  # Small swipe down

Click icon

text

POST /api/click_icon
{"path": "icons/app_qishui.png", "threshold": 0.75, "roi": [0.1, 0.2, 0.5, 0.6], "mask": false, "reset": true}

Click multiple icons

text

POST /api/click_icons
{"paths": ["icons/phone/num1.png", "icons/phone/num3.png", "icons/phone/num2.png"], "interval": 1}

Clicks each icon sequentially. After each click the arm exits the frame, then resets after all clicks. Returns {"ok": true, "data": {"clicked": true, "success_count": N, "failed": []}}.

Click same icon multiple times

text

POST /api/click_icon_many_times
{"path": "icons/qishui/like.png", "count": 3, "interval": 0.5}

Searches for the icon once, then clicks the same position multiple times without moving or resetting. Resets only after all clicks. Returns {"ok": true, "data": {"clicked": true, "clicks": 3}}.

Click text

text

POST /api/click_text
{"text": "领取", "roi": [0.3, 0.5, 0.7, 0.8], "min_score": 0.5}

Click coordinates

text

POST /api/click {"x": 0.5, "y": 0.96}

Click ROI center

text

POST /api/click_roi {"roi": [0.3, 0.5, 0.7, 0.8]}

Custom swipe

text

POST /api/swipe
{"sx": 0.5, "sy": 0.8, "ex": 0.5, "ey": 0.1, "steps": 5, "duration": 0.3}

sx/sy/ex/ey are phone percentage coordinates, steps is the number of steps, duration is total swipe time (seconds).

Close all apps

text

POST /api/close_all_apps {"max_swipes": 15}

Launch app

text

GET /api/run_app?app_name=汽水音乐
# or
POST /api/run_app {"app_name": "汽水音乐"}

Looks up the app icon in app_desktop.yaml and clicks it.

7. Script Control

Run script

text

POST /api/run_script {"path": "scripts/qishui_daily.yaml"}

Returns:

json

{"ok": true, "data": {"script": "D:\\...\\scripts/qishui_daily.yaml", "status": "started"}}

Check script status

text

GET /api/script_status

Returns:

json

{"ok": true, "data": {"running": false, "current_script": null}}

Get script progress

text

GET /api/script_progress

Returns full execution log + stats:

json

{
  "ok": true,
  "data": {
    "running": true,
    "script": "scripts/xxx.yaml",
    "current_step": {"step_index": 5, "action": "switch_page", "target": "拨号页", "status": "ok", "detail": "Matched branch: 拨号页", "timestamp": 1780329450.91},
    "steps_log": [
      {"step_index": 0, "action": "script_start", "target": "test_66", "status": "ok", "detail": "5 top-level steps", "timestamp": ...},
      ...
    ],
    "stats": {
      "total_steps": 24,
      "completed_steps": 23,
      "failed_steps": 0,
      "elapsed": 61.6,
      "status": "running"
    }
  }
}

• steps_log: complete step history with timestamps
• stats: progress stats including elapsed time, completed/failed counts
• Idle state: {"running": false, "script": null, "current_step": null, "steps_log": [], "stats": {}}

Stop script

text

POST /api/stop_script

Wait for template

text

POST /api/wait_for_template {"path": "icons/qishui/reward_popup.png", "timeout": 10, "interval": 0.5}

Polls until the template appears or timeout. Checks every interval seconds within timeout seconds.

Wait for page

text

POST /api/wait_for_page {"config_path": "scripts/configs/app_xxx.yaml", "target_name": "RewardsPage", "timeout": 15}

Polls until the specified page appears or timeout.

Clear overlay

text

POST /api/clear_overlay

8. Configuration

Read/update daily config

text

GET  /api/config/daily
PUT  /api/config/daily  {"windows": [...]}

Read/update app page config

text

GET  /api/config/app/qishui           # Returns app_qishui.yaml content
PUT  /api/config/app/qishui           # Update config (body is YAML text)

Read/update gesture config

text

GET  /api/config/gesture
PUT  /api/config/gesture  {...}

9. Shutdown

text

POST /api/shutdown

Graceful shutdown: detect running script → safe terminate → arm reset → release resources → process exit.

Note: Do not kill the process directly — the COM port won't be released, and you'll need to reinstall the driver on next startup.

10. System Tray

RobotArmServer.exe minimizes to the system tray on launch:

• Left click / double-click: Show/hide console window
• Right-click menu: "Show/Hide Console", "Exit Service"
• Tray exit = API /api/shutdown (same graceful shutdown flow)

---

Directory Structure

text

RobotArmServer/
├── RobotArmServer.exe          ← Main program
├── _internal/                  ← Program libraries (do not touch)
├── scripts/                    ← YAML scripts directory
│   ├── hello_flexarm.yaml      ← Your first script
│   ├── daily_config.yaml       ← Daily scheduled task config
│   ├── configs/
│   │   ├── app_desktop.yaml    ← Phone desktop config (required)
│   │   ├── app_qishui.yaml     ← Qishui Music page config
│   │   └── app_kuaishou.yaml   ← Kuaishou page config
│   └── qishui/                 ← Sub-scripts directory
│       ├── run_*.yaml
│       └── music_actions.yaml
├── icons/                      ← Icon templates directory
│   ├── app_phone.png
│   ├── app_camera.png
│   ├── app_qishui.png
│   └── qishui/
│       ├── cross.png
│       └── ...
├── calibrations/               ← Calibration results (auto-generated)
├── screenshots/                ← Screenshot save directory
├── camera_config.json          ← Camera focus config
├── device_config.json          ← Device config
├── gesture_config.json         ← Swipe gesture config
└── robot-arm-service/          ← Windows service driver

---

FAQ

Q: The robot arm doesn't move?

1. Make sure robot-arm-service\安装.bat has been run as Administrator
2. Verify GET /api/arm_status returns "connected": true
3. Ensure calibration is complete (a JSON file exists in calibrations/)

Q: Icon not found?

1. Confirm the icon file exists in icons/
2. Lower the threshold (e.g., 0.65)
3. Set mask: false (4-corner background sampling, more tolerant)
4. Specify a roi to narrow the search area
5. Check that the icon template matches the on-screen icon (size, color, background)

Q: Text not found?

1. Improve image clarity (adjust camera focus: POST /api/change_focus {"value": 5})
2. Raise min_score to 0.5+ to reduce false matches
3. Specify a roi to narrow the search area
4. Use find_all_text to see what text the OCR actually recognizes

Q: Script execution interrupted?

1. Check GET /api/script_progress to see which step is stuck
2. Check log output (service console)
3. Ensure the phone hasn't shown a system dialog (permissions, notifications, etc.)

Q: How to add automation for a new app?

1. Capture the app icon → place it in icons/
2. Update scripts/configs/app_desktop.yaml (add the new icon to features)
3. Create scripts/configs/app_xxx.yaml (define each page inside the app)
4. Write scripts/run_xxx.yaml (define the workflow)
5. Run: POST /api/run_script {"path": "scripts/run_xxx.yaml"}

---

Error Handling Guide

All APIs return a uniform format: {"ok": true/false, "data": {...}, "error": "..."}

Common Errors & Strategies

Error	Cause	Resolution
"error": "Script is running"	A script is executing in the background	Check script_status, wait for completion, or call stop_script
"error": "RobotActions not initialized"	Arm not connected / service not started	Guide user to check robot-arm-service\安装.bat
"error": "Missing parameter: path"	Incomplete request parameters	Check API call parameters
"ok": false, "data": null (find_template)	Icon not found	Lower threshold or check icon file, do not retry indefinitely
"ok": false, "data": null (find_text)	OCR text not found	Widen roi or lower min_score, try at most 2-3 times then report
"error": "Unauthorized"	License check failed	Guide user to activate

Agent Retry Recommendations

• Icon lookup: fail → lower threshold → retry once → still fail → report to user
• Text lookup: fail → widen ROI → retry once → still fail → report to user
• Page detection: no match → try go_back or go_home → re-detect → still no match → report to user
• Script conflict: received "script running" → check script_status → if running, wait or report to user

---

Agent Conversation Example: Find and Open Qishui Music

Below is a complete example showing how an AI Agent combines APIs to "find and open the Qishui Music app on the desktop":

Step 1: Check service status

bash

curl http://127.0.0.1:7826/api/health
# Returns: {"ok":true,"data":{"status":"running","arm_connected":true,...}}

Step 2: Detect current desktop

bash

curl -X POST http://127.0.0.1:7826/api/detect_desktop -H "Content-Type: application/json" -d '{}'
# Returns: {"ok":true,"data":{"page_name":"desktop_page1","score":0.94,"matched":true}}

Step 3: Find the Qishui Music icon

bash

curl -X POST http://127.0.0.1:7826/api/find_template -H "Content-Type: application/json" -d '{"path":"icons/app_qishui.png","threshold":0.75}'
# Returns: {"ok":true,"data":{"x":242,"y":516,"w":55,"h":55,"score":0.92}}

Step 4: Click the icon

bash

curl -X POST http://127.0.0.1:7826/api/click_icon -H "Content-Type: application/json" -d '{"path":"icons/app_qishui.png"}'
# Returns: {"ok":true,"data":{"clicked":true,"score":0.92,...}}

Step 5: Wait for app launch, detect page

bash

python -c "import requests,time; time.sleep(2)"
python -c "import requests; r=requests.post('http://127.0.0.1:7826/api/detect_page',json={'config_path':'scripts/configs/app_qishui.yaml'}); print(r.text)"
# Returns: {"ok":true,"data":{"page_name":"Music","score":0.85,"matched":true}}

Step 6: Confirm phone is in frame

bash

curl http://127.0.0.1:7826/api/is_phone_present
# Returns: {"ok":true,"data":{"present":true}}

✅ Task complete: Qishui Music is open, currently on the Music page.

Or more directly, if app_desktop.yaml is properly configured and the Qishui Music icon exists, you can use the run_app API endpoint directly. It will intelligently auto-navigate, find the correct desktop page, and click the icon.