application-use

Automate macOS tasks by opening apps, clicking elements, filling forms, typing, scrolling, and controlling the desktop via CLI commands.

macOS Desktop Automation (application-use)

A CLI to automate macOS applications.

install

bash

npm i -g application-use

🚀 Core Workflow

1. Open: application-use open --appName "<Name>" -- Auto-shows initial snapshot.
2. Interact: Use hints (e.g., [A]) to click, type, or scroll.
3. Auto-Update: Commands (open, click, type, sendkey, scroll) automatically display an updated snapshot.

[!TIP] Use application-use snapshot only when you need a fresh view without any action (e.g., after waiting for dynamic content).

🛠 Command Reference

Command	Usage	Description
open	open --appName "Safari"	Opens app and shows snapshot.
click	click A [--right] [--double]	Clicks element by hint letter.
fill	fill A "text" or fill "text"	Fills text into hint or focused element.
sendkey	sendkey enter, sendkey cmd+v	Sends single or shortcut keys.
scroll	scroll "Main" down [500]	Scrolls area by name/hint.
screenshot	screenshot [path] [--frame]	Captures window or specific coordinates.
search	search "Safari"	Finds installed application names.
close	close --appName "Safari"	Quits the specified application.
type	type "text"	Types text into the focused element.

🏷 Hint System

• Elements: [A], [B], [AA]... (Interactive elements).
• Areas: [a], [b]... (Targets for scrolling).
• Special Marks: (*) = OCR-detected; (+) = Newly added since last snapshot.

⚡️ Common Automation Patterns

Form Filling & Submission & scroll

bash

application-use open --appName "Safari"
application-use fill A "[e-mail]"    # Fill email
application-use sendkey tab                   # Next field
application-use fill "password"               # Focus-fill password
application-use sendkey enter                 # Submit
application-use scroll d down 1000            # Scroll down area d

Browser Navigation

bash

application-use open --appName "Safari"
application-use fill A "https://google.com" && application-use sendkey enter
# Wait for load, then interact with hints from auto-snapshot

File Operations (Finder)

bash

application-use open --appName "Finder"
application-use sendkey cmd+shift+g           # Go to Folder
application-use fill "/Users/name/Downloads" && application-use sendkey enter

# Open in New Tab via right-click
application-use click A --right               # Right-click folder/file (hint A)
application-use click B                       # Click "Open in New Tab" from menu (hint B)