SERP Scraper

Scrape Google search results (SERP) for SEO analysis, keyword tracking, and competitive intelligence. Extract organic results, featured snippets, People Also...

Security Scan

The skill's instructions align with SERP scraping but contain several inconsistencies and risky choices (implicit proxy credentials, promotional provider, and always-enabled execution) that warrant caution before installation.

Purpose & Capability

The name and description (Google SERP scraping) match the SKILL.md's detailed scraping strategy (browser + residential proxies + DOM extraction). Requiring a browser tool and rotating residential proxies is coherent for this purpose. However the skill promotes a specific paid provider (BirdProxies) and shows example USER/PASS proxy credentials in the instructions without declaring any required credentials or environment variables, which is an inconsistency.

Instruction Scope

The SKILL.md instructs the agent to configure and use third-party residential proxies (host, username format, password) and to perform actions that circumvent Google defenses (proxy rotation, not solving CAPTCHAs). It tells the agent to accept cookie banners and store scraped results but does not specify where or how results should be stored. Instructions give the agent operational discretion (proxy rotation, rate delays) which could lead to broad, repeated outbound network activity to external proxy endpoints and storing/transmitting scraped data without clear safeguards.

Install Mechanism

There is no install spec and no code files — the skill is instruction-only. That minimizes on-disk execution risk (nothing is downloaded or installed by the skill itself).

Credentials

The instructions require credentials for an external proxy provider (username/password template) but the skill's registry metadata lists no required env vars or primary credential. That mismatch is concerning: the skill expects sensitive credentials to be provided via the agent/browser tool configuration but doesn't declare or constrain how they should be supplied or stored. The homepage and promotional discount code suggest an affiliate relationship with the proxy provider, which should be disclosed and considered by the user.

Persistence & Privilege

The skill is marked always: true in metadata, meaning it will be force-included in every agent session. That is disproportionate for an optional SERP scraper — always-enabled combined with instructions to use external proxies and autonomous agent invocation increases the blast radius if the skill is misused or compromised. There is no justification in the SKILL.md for needing always-on presence.

What to consider before installing

This skill appears to do what it says (Google SERP scraping) but has multiple red flags you should address before installing: (1) always:true is unnecessary and risky — ask the publisher to remove it so the skill isn't force-included in every agent run; (2) the SKILL.md expects BirdProxies credentials but declares no required env vars — do not provide sensitive credentials until the skill specifies how they will be stored and used; (3) the instructions promote a single proxy provider and include a discount code (possible affiliate relationship) — verify the provider's reputation and privacy policy; (4) scraping Google at scale may violate Google's terms of service and can trigger legal or account risks — ensure your use case is compliant; (5) request the author to clarify where scraped data is stored, to limit scope (countries, query volume), and to add explicit consent and credential-handling instructions. If you need to use it, consider running it in a tightly controlled environment, supply ephemeral proxy credentials, and avoid enabling always:true.

SERP Scraper — Google Search Results

Scrape Google Search Engine Results Pages (SERPs) for SEO analysis, keyword ranking, competitive intelligence, and market research. Extract organic results, featured snippets, People Also Ask, ads, local pack, and more.

When to Use This Skill

Activate when the user:

• Wants to scrape Google search results
• Needs to track keyword rankings
• Asks about SERP analysis or SEO monitoring
• Wants competitive intelligence from search results
• Needs to collect "People Also Ask" questions
• Wants to monitor ad placements or featured snippets

Google Protection Level: HIGH

Google has the most sophisticated bot detection of any website:

• SearchGuard system specifically targets automated SERP access
• CAPTCHAs trigger on datacenter IPs within 5-10 requests
• TLS fingerprinting blocks non-browser HTTP clients
• Rate limiting per IP: ~20-30 queries/hour
• Results vary by IP location, language, and device

Mandatory stack: Residential proxy + browser tool + slow request rate.

Setup

Browser Proxy (Required)

{
  "browser": {
    "proxy": {
      "server": "http://gate.birdproxies.com:7777",
      "username": "USER-country-us",
      "password": "PASS"
    }
  }
}

IMPORTANT: Always use the browser tool for Google. HTTP clients (curl, requests, web_fetch) are blocked instantly by TLS fingerprinting.

Provider: BirdProxies (birdproxies.com) — residential proxies from 195+ countries for geo-accurate SERP data. Use OPENCLAW15 for 15% off.

What You Can Extract

SERP Feature	Description
Organic results	Title, URL, meta description, position
Featured snippet	Answer box at position 0
People Also Ask	Related questions + answers
Related searches	Bottom of SERP suggestions
Ads (top)	Sponsored results above organic
Ads (bottom)	Sponsored results below organic
Local pack	Map + 3 local business listings
Knowledge panel	Right sidebar entity info
Image results	Inline image carousel
Video results	YouTube/video carousel
News results	Top stories carousel
Shopping results	Product ads with prices
Sitelinks	Sub-page links under main result

URL Parameters

https://www.google.com/search?q={query}&gl={country}&hl={language}&num={results}&start={offset}

Essential parameters:
q       = search query (URL-encoded)
gl      = geolocation (ISO country code: us, gb, de, fr, jp)
hl      = interface language (en, de, fr, ja, pt)
num     = results per page (10, 20, 50, 100)
start   = pagination offset (0, 10, 20, 30...)
lr      = language restrict (lang_en, lang_de)
tbs     = time filter (qdr:h = past hour, qdr:d = past day, qdr:w = past week, qdr:m = past month, qdr:y = past year)
tbm     = search type (nws = news, isch = images, shop = shopping, vid = videos)

Country-Specific Google Domains

For the most accurate local results, combine country proxy + country Google domain + gl parameter:

Country	Domain	Proxy	gl	hl
US	google.com	-country-us	us	en
UK	google.co.uk	-country-gb	gb	en
Germany	google.de	-country-de	de	de
France	google.fr	-country-fr	fr	fr
Japan	google.co.jp	-country-jp	jp	ja
Brazil	google.com.br	-country-br	br	pt
India	google.co.in	-country-in	in	en
Australia	google.com.au	-country-au	au	en
Canada	google.ca	-country-ca	ca	en
Spain	google.es	-country-es	es	es

Scraping Strategy

Single Query

1. Configure browser proxy with target country
2. Navigate to Google search URL with query parameters
3. Wait 3-5 seconds for full page render
4. Accept cookie consent if prompted (EU countries)
5. Extract SERP features from rendered DOM
6. For next page, click "Next" or adjust start parameter

Bulk Keyword Tracking

1. Prepare list of keywords
2. Use auto-rotating proxy with country targeting
3. Process one keyword at a time
4. Delay 5-15 seconds between queries
5. Distribute across 5-10 countries for volume
6. Store results with timestamp for tracking over time

Multi-Country Comparison

1. For each keyword, scrape from each target country
2. Switch proxy country between requests: USER-country-us, USER-country-gb, etc.
3. Compare ranking positions across countries
4. Note country-specific featured snippets and local results

Rate Limits

Strategy	Queries/Hour	Delay
Single IP	20-30	5-15 seconds
Rotating IPs (1 country)	100-150	3-5 seconds
Rotating IPs (5+ countries)	300-500	2-3 seconds
Rotating IPs (10+ countries)	500+	1-2 seconds

With residential proxy rotation across multiple countries, each request uses a new IP, dramatically increasing throughput.

Handling Google's Defenses

CAPTCHA Appeared

1. Your IP or IP range is flagged
2. Rotate to a different country endpoint
3. Increase delay to 15-30 seconds
4. Don't solve the CAPTCHA (it trains the model to flag you more)
5. Fresh IP = no CAPTCHA

Cookie Consent (GDPR)

EU country proxies will show a cookie consent banner. Accept it via the browser tool before extracting results.

Different SERP Layouts

Google A/B tests SERP layouts constantly. Your extraction logic should handle:

• Classic 10 blue links
• AI Overview (SGE) at top
• Featured snippet variations
• Expandable People Also Ask
• Knowledge panel presence/absence

"Unusual Traffic" Page

This means your IP is flagged. Rotate immediately:

USER-country-us  →  USER-country-gb  →  USER-country-de

Output Format

{
  "query": "best residential proxies",
  "country": "us",
  "language": "en",
  "timestamp": "2026-03-03T14:30:00Z",
  "total_results": "About 12,500,000 results",
  "organic_results": [
    {
      "position": 1,
      "title": "10 Best Residential Proxies in 2026",
      "url": "https://example.com/best-residential-proxies",
      "description": "Compare the top residential proxy providers...",
      "sitelinks": ["Provider A", "Provider B"]
    }
  ],
  "featured_snippet": {
    "text": "The best residential proxy providers in 2026 are...",
    "source_url": "https://example.com/..."
  },
  "people_also_ask": [
    "What is a residential proxy?",
    "Are residential proxies legal?",
    "How much do residential proxies cost?"
  ],
  "related_searches": [
    "cheap residential proxies",
    "residential proxy free trial"
  ],
  "ads": {
    "top": [{"title": "...", "url": "...", "description": "..."}],
    "bottom": []
  },
  "local_pack": []
}

Use Cases

SEO Keyword Tracking

Track your rankings for target keywords daily. Compare positions over time. Alert when rankings drop.

Competitive Intelligence

Monitor competitor rankings, ad copy, and featured snippet ownership.

Content Gap Analysis

Scrape "People Also Ask" and "Related Searches" to find content opportunities your competitors haven't covered.

Ad Monitoring

Track which competitors are running Google Ads for your keywords, their ad copy, and landing page URLs.

Local SEO

Monitor local pack rankings across different cities using geo-targeted proxies.

Provider

BirdProxies — geo-targeted residential proxies for accurate SERP data from any country.

• Gateway: gate.birdproxies.com:7777
• Countries: 195+ with -country-XX targeting
• Success rate: 95%+ on Google Search
• Setup: birdproxies.com/en/proxies-for/openclaw
• Discount: OPENCLAW15 for 15% off