Free Web Search Js

Web search & content fetching via Playwright + HTTP — zero API keys. Searches Bing CN (domestic) or DDG (international), auto-fetches top page contents. For real-time search, fact-checking, news, tutorials, documentation lookup.

Pure search — no rewriting, no filtering, noise handled by the AI. Uses real browser automation (Playwright) for Bing CN anti-bot, with HTTP fallback for international queries. Zero API keys required.

Quick start

bash

# Search (auto-fetches top 3 page contents)
node scripts/search.js "silver price"
node scripts/search.js "how to deploy docker" --max=5
node scripts/search.js "xxx" --region=cn
node scripts/search.js "xxx" --fetch=5         # fetch top 5
node scripts/search.js "xxx" --no-fetch        # search only

# Fetch specific URLs
node scripts/fetch.js "https://example.com/page1" "https://example.com/page2"

Architecture

text

Domestic (CN IP):
  Playwright headless → Bing CN → home page for cookie → search box submit
  → 0 results? fallback to DDG HTML
  → auto-fetch top N page contents

International:
  Pure HTTP → DDG HTML parse
  → fetch fails? fallback to Playwright request
  → DDG empty? fallback to Bing
  → auto-fetch top N page contents

Install

All dependencies are installed upfront by the setup script. The skill never runs npm install or pip install at runtime. This avoids audit flags and supply-chain risks.

Prerequisites

Dependency	Purpose	Size
Node.js >= 18	Runtime	—
cheerio	HTML parsing	small
commander	CLI arg parsing	small
iconv-lite	GBK charset conversion	small
playwright	Browser automation	~50 MB
Chromium (Playwright)	Browser engine for Bing search + headed fetch	~150 MB

One-command setup

bash

# Linux / macOS
bash scripts/setup.sh

# Windows
powershell -File scripts/setup.ps1

The scripts auto-detect region and use npmmirror.com mirrors inside China for faster downloads.

Manual install

bash

cd skills/free-web-search-js
npm install
npx playwright install chromium    # ~150 MB

# Verify
node scripts/check-env.js

If you already have Chrome/Chromium installed:

bash

export CHROMIUM_EXECUTABLE_PATH="/path/to/chrome"
node scripts/check-env.js

check-env.js will confirm whether the provided executable is usable.

Environment variables

Variable	Purpose
CHROMIUM_EXECUTABLE_PATH	Path to system Chrome/Chromium, skips 150 MB download
PLAYWRIGHT_DOWNLOAD_HOST	Mirror for Chromium download (auto-set by setup script)
BROWSER_DAEMON_IDLE_MS	Daemon idle timeout in ms (default 600000 = 10 min)

Search engines

Engine	Method	Region	Notes
Bing CN	Playwright search box submit	Domestic	Visits home page first for cookie, then types into search box
DDG	Playwright search box submit	International	Full browser automation

Strategy

Region	Search	Fetch
Domestic	Bing CN, fallback DDG	Auto-fetch top 3
International	DDG HTML, fallback Playwright request; DDG empty→Bing	Auto-fetch top 3

IP detection

On each search, three probes run in parallel — first responder wins:

Probe	Service	Logic
1	myip.ipip.net / cip.cc	Reachable from China
2	ipinfo.io / ipapi.co	International
3	cn.bing.com	Reachable → likely CN
Fallback	—	Defaults to domestic

Override with --region=cn or --region=intl if your proxy causes misdetection.

Fetch mode

After search, auto-fetches the top N results (default 3).

Tier	Method	Speed	Notes
1	Lightweight HTTP + cheerio	Fast	No browser launch
2	Playwright headed	Slower	Full JS rendering

Tier 1 enhancements:

• JSON API response detection with structured content extraction
• JSON-LD (<script type="application/ld+json">) articleBody/description
• __NEXT_DATA__ embedded data extraction
• Meta tag fallback (og:description / description)
• Auto GBK charset detection & conversion

Deduplication

Smart dedup: domain + path stem (ignores www/m subdomains, tracking params, trailing slash, .html suffix). Bing redirect URLs (bing.com/ck/) auto-decoded.

Browser daemon (optional, ~70% faster)

Avoid 1.5s+ browser launch overhead by reusing a persistent Chromium instance:

bash

node scripts/browser-daemon.js            # Start (run with & or as background job)
node scripts/browser-daemon.js --status   # Show status
node scripts/browser-daemon.js --stop     # Stop

The daemon auto-exits after 10 minutes of inactivity (configurable via BROWSER_DAEMON_IDLE_MS). Activity is tracked via a .browser-heartbeat file that search.js and fetch.js touch on each connection. The daemon checks the heartbeat every 60 seconds.

Verification & troubleshooting

bash

cd skills/free-web-search-js
node scripts/check-env.js
node scripts/search.js "OpenClaw" --max 2 --region cn --fetch 0
node scripts/search.js "OpenClaw" --max 2 --region intl --fetch 0

Common issues:

• node_modules not found → run npm install
• Executable doesn't exist ... → run bash scripts/setup.sh or set CHROMIUM_EXECUTABLE_PATH
• browserType.launch ... closed → headless+no-sandbox is default; run node scripts/check-env.js to diagnose
• 0 search results → try --region=cn / --region=intl, or --fetch 0 to test search alone

Privacy & network

• Searches connect to Bing CN / DDG HTML / Sogou directly from the skill's runtime — no intermediate proxy
• The daemon heartbeat file (.browser-heartbeat) and endpoint file (.browser-endpoint) are written to the skill root directory
• No analytics, no telemetry, no third-party logging beyond the search engine's own server logs
• Headed warm-up fetches: Cookie warm-up for Bing uses a headed browser context to obtain session cookies from the search engine home page; no cookies are persisted to disk between sessions

Known limitations

• First domestic search slower: Chromium cold start takes 3–6s; daemon avoids this
• Bing CN instant answers: Weather/calculator cards don't use li.b_algo, return 0 results
• Sogou HTTP unstable: No cookie, easily blocked by anti-crawl
• JS-required pages: HTTP tier returns empty for SPA pages — headed tier catches them
• Region-specific sites: Some domestic sites time out from international IPs
• Proxy interference: IP detection may misclassify behind proxies — use --region to override
• DDG blocked in China: DDG HTML is not reachable from mainland China; domestic strategy avoids it