hfpclawer-paper-search

Discover, download, and organize academic papers from arXiv, HuggingFace Papers, and OpenReview. Multi-source search → dedup → PDF download → Markdown conversion → optional wiki sync. Designed for researchers who want to monitor new papers daily.

hfpclawer Paper Search & Download

🔒 Sanitization: This skill ships in the public repo. Never embed private LAN IPs, real person names, or machine codenames in examples — use <placeholder> / Jane Doe / [e-mail]. Real values live in the repo's gitignored .hermes/internal-guide.md. See repo AGENTS.md "Public-Release Sanitization".

A multi-source academic paper pipe: search across arXiv / HuggingFace Papers / OpenReview / PapersWithCode, deduplicate by title, download PDFs, convert to Markdown, and optionally sync to a wiki.

Who this is for: Researchers who want a daily "new papers on my topic" feed without manually checking multiple websites.

Overview

Typical workflow in one command:

text

hfpclawer search           # Discover new papers across sources
   └── ranked by relevance to your keywords
hfpclawer download         # Download PDFs for matched papers  
   └── 8 concurrent streams
hfpclawer convert --to-wiki # PDF → readable Markdown + wiki sync

Direct fetch of a known arXiv ID (v0.16.11+)

hfpclawer fetch <arxiv_id> downloads one paper by ID with a built-in network escape ladder — TCP quick-probe → QUIC/HTTP-3 → browser-hint. Use it whenever arXiv TCP/443 is RST-reset (common on CN networks); QUIC over UDP/443 survives the reset and is the primary escape channel.

bash

hfpclawer fetch 2609.02737                    # PDF, auto transport
hfpclawer fetch 2608.06013 -k source -t quic  # TeX bundle, force QUIC

Robustness built in: bounded-memory streaming (512KB flush to .part), resume on interruption (fresh .part continues via Range; >24h stale .part reclaimed), and a sha256 integrity anchor printed on every completed fetch and appended to data/download_audit.jsonl.

Channel-verification trick: fetch the same arXiv ID twice — once from the official arXiv endpoint and once from a mirror (e.g. AlphaXiv; its real PDFs live at pdfs.assets.alphaxiv.org) — and compare sha256. Byte-identical hashes verify the QUIC channel end-to-end (cross-channel MITM detection).

Or run the full pipeline at once:

bash

hfpclawer full --max-pages 3 --to-wiki

Multi-Source Registry (v0.18+)

Beyond the default HF/arXiv path, three biomedical adapters are registered — europepmc, biorxiv, medrxiv. They run only when enabled:

yaml

search:
  enabled: [hf_cli, arxiv_api, europepmc, biorxiv, medrxiv]

bash

hfpclawer source-list                               # which adapters exist
hfpclawer source-search europepmc "CRISPR screen"   # query one adapter directly

• Registration is not enablement — an adapter in the registry does nothing until its key is in search.enabled, so adding one cannot change existing behaviour.
• Keep personal query lists and real names in the gitignored config.local.yaml; search.biomed_queries: [] in the tracked file is the declared placeholder slot.
• Europe PMC returns no abstract unless resultType=core is set (the adapter sets it); bioRxiv/medRxiv expose a date-range API only, so keyword filtering happens at ingest.
• Transient failures retry through anti_crawl.max_retries / retry_http_codes / retry_delay_base.

Prerequisites

Python ≥ 3.10. Pick the install that matches how you work — uv is recommended, because the CLI then lives in its own environment (no conflicts with your project's dependencies):

bash

# 1) Recommended — uv tool: isolated CLI install, `hfpclawer` on your PATH
uv tool install hfpclawer
hfpclawer init                    # writes config.yaml

# 2) Try it without installing anything (ephemeral, one-off runs)
uvx hfpclawer --help

# 3) Inside an existing project / venv (uv-managed)
uv pip install hfpclawer

# 4) No uv yet — pip and pipx both work
pip install hfpclawer             # or: python -m pip install hfpclawer
pipx install hfpclawer            # CLI-style install, functionally like `uv tool`

Optional extras — the core install stays deliberately small:

Extra	Adds	Install
zotero	pyzotero — the Zotero read / write / ingest paths	uv tool install "hfpclawer[zotero]"
nlp	spaCy pipeline for entity enrichment	uv tool install "hfpclawer[nlp]"
graph	networkx + geopy for the citation graph	uv tool install "hfpclawer[graph]"
llm	litellm for opt-in LLM helpers (sniff)	uv tool install "hfpclawer[llm]"

⚠️ nlp extra + PyPI (checked against the published 0.19.0 metadata): PyPI strips the direct-URL spaCy model, so that extra installs spacy only — fetch the model yourself with python -m spacy download en_core_web_sm. The loader falls back configured → en_core_web_md → en_core_web_sm and logs one actionable hint when none is present: entity enrichment degrades, nothing else breaks.

Where to find it: repo https://github.com/diamond2nv/hfpapers-crawler · PyPI https://pypi.org/project/hfpclawer/ · registry: clawhub inspect <slug>

Edit config.yaml with your search interests (see Configuration section below).

Quick Start

1. First-time Setup

bash

# Create default config
hfpclawer init

# Edit the config to match your research interests
vim config.yaml
# → Change: search.queries, keywords.include_high, keywords.exclude

2. One-Shot Full Pipeline (daily use)

bash

# Discover → Download → Convert → Wiki sync in one command
hfpclawer full

# Limit pages for a quick check
hfpclawer full --max-pages 3 --to-wiki

3. Step-by-Step (for debugging)

bash

# Step 1: Search across all sources
hfpclawer search --max-pages 5

# Step 2: Download PDFs for matched papers
hfpclawer download

# Step 3: Convert PDFs to Markdown
hfpclawer convert

# Step 4: Sync to wiki directory
hfpclawer convert --to-wiki

4. Monitor New Papers Regularly

bash

# Check what papers have been downloaded
hfpclawer list

# Show paper store statistics
hfpclawer store stats

# Start the real-time download monitor
hfpclawer monitor start

Configuration

The config file config.yaml controls what papers are searched and downloaded:

yaml

search:
  max_per_dim: 50           # Papers per search query per source
  queries:
    - query: "neural operator"
      category: neural-operator
    - query: "physics-informed"
      category: physics-informed
    - query: "PDE solver deep learning"
      category: pde-solver

keywords:
  include_high:              # Papers must match these (OR)
    - "neural operator"
    - "pde"
    - "deep learning"
  include_low:               # Optional bonus keywords
    - "fourier"
    - "self-attention"
  exclude:                   # Exclude these topics
    - "quantum"
    - "llm"

classification:
  threshold_pass: 30         # Relevance score threshold (0-100)
  title_similarity_min: 0.40 # Dedup threshold

paths:
  data_dir: "data"           # SQLite DB location
  pdf_dir: "pdfs"            # Downloaded PDFs
  md_dir: "mds"              # Converted Markdown files

Available Commands

Command	Purpose	Common Flags
hfpclawer search	Discover new papers	--max-pages, --dry-run
hfpclawer download	Download PDFs	(runs from search results)
hfpclawer convert	Convert PDF → MD	--to-wiki syncs to raw/papers/
hfpclawer full	All-in-one pipeline	--max-pages, --to-wiki
hfpclawer list	List downloaded papers
hfpclawer store stats	Paper store statistics
hfpclawer store export	Export store as JSON/CSV	--format json
hfpclawer store verify	Cross-verify paper metadata	--arxiv-id
hfpclawer config	Show current config
hfpclawer mcp	Start MCP server	(for LLM integration)
hfpclawer monitor	Download daemon control	start, stop, status
hfpclawer dedup	Show dedup statistics

Daily Routine Examples

Morning — Check What's New

bash

# Quick scan (3 pages per query, ~50 papers)
hfpclawer search --max-pages 3

# View results
hfpclawer store stats

Afternoon — Download & Read

bash

# Download all new papers
hfpclawer download

# Convert to readable markdown
hfpclawer convert

# Read the best one
cat mds/2010.08895.md | head -80

Weekly — Full Pipeline

bash

# Full sweep with wiki sync
hfpclawer full --max-pages 10 --to-wiki

# Validate references in newly added papers
hfpclawer audit verify "Key cited paper" --source openalex

Data Storage

hfpclawer uses three tiers:

Storage	Location	Content	Persistence
SQLite	data/papers.db	Metadata, dedup, cross-ref	Persistent
PDFs	pdfs/	Raw paper PDFs	Download once, keep
Markdown	mds/	Converted text	Regeneratable from PDFs

The paper store tracks:

• arXiv ID, title, authors, abstract
• Source of discovery (HF / arXiv / OpenReview)
• Download status, conversion status
• Wikified path (if synced)
• Cross-verification with Crossref (DOI validation)

Search · Breadth · Recommend (搜广推, v0.18+)

Three families, each usable on its own — combined they form the discovery loop:

Layer	Commands	What it does
搜 Search	search, convert-tex, sniff, fetch	multi-source discovery (HF Papers + arXiv + OpenReview + biomedical), TeX source → Markdown with formulas preserved, opt-in LLM abstract triage, CN-aware single-paper transport
广 Breadth	graph (expand-hub --audit), pool, dedup, batch	citation-graph hub expansion with an audit trail of what was adopted vs truncated, positive-example pool, dedup statistics, queue-based batch download
推 Recommend	recommend, profile, rank	scored recommendations that show why (layer + query per row), interest profile, opt-in learned re-ranking

bash

# Recommend papers for this repo (fuses config queries + repo/user profile, keeps suspect out)
hfpclawer recommend --limit 10
hfpclawer recommend --path ~/my-project        # read the profile from another repo

# Show the profile that drives it (repo block + private user profile)
hfpclawer profile            # repo: hfpclawer: YAML block in the nearest AGENTS.md
hfpclawer profile --user     # private ~/.hfpclawer/profile.yaml

# Grow the positive-example pool, then train the learned re-ranker (local lightgbm)
hfpclawer pool sync-favorited                  # Zotero favorites → positives
hfpclawer pool ingest-verified                 # human-approved papers
hfpclawer pool export --out data/train.jsonl
hfpclawer rank train --audit data/expand-audit.jsonl --out data/rank_model.txt

Why it is auditable: every recommendation row carries its provenance (which layer, which query, verification state). Suspect papers are gated out; verified ones are preferred.

Zotero Integration (read · write · ingest)

Talks to Zotero Desktop's local server on port 23119 — the local server needs no cloud account; the zotero pyzotero extra is still required for these paths. Zotero must be running.

Direction	Commands	Endpoint
Read	zotero check, list, search, get, tags, children	local API /api/
Write	zotero push, zotero push-batch	Connector protocol /connector/
Ingest	zotero ingest	Zotero local PDF → paper_store + wiki/raw + annotations
Learn	pool sync-favorited	Zotero favorites feed the recommendation pool

bash

hfpclawer zotero check                        # verify connectivity first
hfpclawer zotero list --limit 10 --tag hfpclawer
hfpclawer zotero search "neural operator" --limit 5
hfpclawer zotero get ABC123
hfpclawer zotero push <arxiv-id>              # paper_store → Zotero
hfpclawer zotero ingest ABC123                # Zotero PDF → store + wiki + annotations

Override ZOTERO_API_URL only if Zotero listens elsewhere. Real Zotero user ids are private — keep them in gitignored config, never in tracked files.

Hermes Agent Environment

hfpclawer is built to run inside Hermes Agent (and OpenCode) as a first-class tool — the agent discovers the skill, and every command below is callable without leaving the session.

1. Install the skill — place this folder under ~/.hermes/skills/research/<slug>/ (or install from ClawHub: clawhub inspect <slug> --file SKILL.md); Hermes loads it automatically and skill_view(name='<slug>') returns this file.

2. Register the MCP server so the agent calls the CLI as tools:

yaml

# ~/.hermes/config.yaml
mcp:
  servers:
    hfpclawer:
      command: "hfpclawer"
      args: ["mcp"]        # stdio mode — Hermes native MCP client

For OpenCode / debugging use HTTP mode: hfpclawer mcp --mode http --port 8765.

3. Environment variables — every one is optional; the pipeline runs with none set:

Variable	Purpose
HFPAPERS_DATA_DIR	state/DB root (XDG ~/.local/share/hfpclawer when installed; the checkout when run from source)
HFPAPERS_CONFIG / HFPAPERS_LOCAL_CONFIG	config file + private overlay
S2_API_KEY	Semantic Scholar — 10x faster (anonymous tier works)
OPENALEX_POLITE_EMAIL	OpenAlex polite pool — 10x faster
ZOTERO_API_URL	Zotero local API base (default http://127.0.0.1:23119/api/)
HFPCLAWER_PEER_REPOS / HFPCLAWER_REPO_MAP	address private sibling repos by tag instead of hard-coded paths

4. Cost profile — the mechanical layer (search / dedup / verify / audit / Zotero) needs low-cost by design — not zero-cost. The core path (search / dedup / verify / audit / Zotero) issues no LLM call and needs no API key, so the marginal cost per run is small; it is still not zero — bandwidth, disk and CPU are spent, and rate-limited upstreams (arXiv / OpenAlex / Semantic Scholar) can throttle or expect a key at volume. LLM features (sniff, abstract triage) are opt-in and metered where they run; rank trains locally with lightgbm. Describe the cost as low, and keep the LLM steps explicit — do not advertise the stack as zero-token.

5. Keep private data private — real author lists, ORCIDs and Zotero ids belong in config.local.yaml (gitignored) or ~/.hfpclawer/profile.yaml, never in tracked files. The repo-side profile (the hfpclawer: block in a project AGENTS.md) is deliberately public-safe: neutral academic keywords only.

Common Pitfalls

1. pip install needs to be in the right venv. If hfpclawer command is not found, check the active Python environment.
2. HuggingFace CLI rate limits. Too many queries per minute will trigger 429s. Reduce max_per_dim to 10 if this happens.
3. Scrapy spiders need scrapy extra installed. If you see ModuleNotFoundError: scrapy, run pip install hfpclawer[scrapy].
4. PDF conversion needs pymupdf4llm. Run pip install hfpclawer[pdf] if hfpclawer convert complains about missing pymupdf4llm.
5. Wiki sync defaults to raw/papers/. If you do not have a wiki directory, skip --to-wiki and read from mds/ directly.
6. First run creates a config.yaml. Edit it before running hfpclawer full, otherwise the default queries may not match your research area.

Verification Checklist

• hfpclawer init creates a valid config.yaml
• hfpclawer search --dry-run validates config without network calls
• hfpclawer search --max-pages 3 returns real papers
• hfpclawer download downloads PDFs correctly
• hfpclawer convert produces readable Markdown
• hfpclawer store stats shows non-zero counts
• hfpclawer store verify --arxiv-id 2010.08895 cross-checks via Crossref