SF Civic Digest

Track San Francisco city government activity — Board of Supervisors, Land Use committee, Public Safety committee, SFMTA Engineering Public Hearings, SF Planning Commission, Historic Preservation Commission, Zoning Administrator hearings, Planning project notices, and SF.gov district events. Fetches agendas and recaps from SF Legistar, sfmta.com, sfplanning.org, the sf.gov Wagtail CMS API (api.sf.gov), and Socrata open data via curl-based scraping. Filtered by supervisorial district, neighborhood, streets, or topic keywords. Use when a user asks what's happening at City Hall, wants to track legislation or development in their neighborhood, wants a weekly or daily SF civic digest, asks about upcoming public hearings, planning variances, or wants to know what their supervisor is working on. SF-specific.

Sean Gillen@sgillen

Install

openclaw skills install @sgillen/sf-civic-digest

SF Civic Digest — SKILL.md

Cold-start note: This file is designed to be self-sufficient. An agent reading only SKILL.md should be able to run reports, generate digests, and understand the full data stack. No other context required.

Quick Start

bash

# Step 1: Find the absolute path to this skill's scripts/ directory
#         (it's always a sibling of this SKILL.md file)
SKILL_DIR="/absolute/path/to/sf-civic-digest"

# Step 2: Run scripts using that absolute path (--district N required)
python3 "$SKILL_DIR/scripts/sf_weekly_digest.py" --district 5 --json

# Step 3: Synthesize the JSON output into a narrative report (see STYLE.md)

Note: sf_weekly_digest.py makes ~22 network calls sequentially and can take 2-3 minutes. For a quick test, try a single script first: python3 "$SKILL_DIR/scripts/sf_journalism.py" --district 5 --days 7 --json

District: Pass --district N (1–11) to any script. User preferences and neighborhood context live in USER.md — read it before writing a report. There is no civic_config.json or civic_config.example.json; those files have been removed.

⚠️ Path rule — critical for any agent (any agent, IDE, or human):

All scripts must be run with the absolute path to the scripts/ directory
Find it: scripts/ is always a sibling of this SKILL.md file
Example: if SKILL.md is at /home/alice/.openclaw/workspace/skills/sf-civic-digest/SKILL.md, then scripts are at /home/alice/.openclaw/workspace/skills/sf-civic-digest/scripts/
Never use bare relative paths like scripts/foo.py — they break when the working directory is anything other than the skill root
When in doubt: realpath on any script path gives you the absolute path

Architecture

Scripts are data pipelines. The agent writes the narrative.

The scripts fetch, parse, dedup, and output JSON. The agent (you) reads that JSON and synthesizes it into a report following the style guide. There is no auto-generated report — the editorial judgment is yours.

text

sf_weekly_digest.py --json
    └── calls all subscripts → returns combined JSON
            │
            ▼
    Agent reads JSON + STYLE.md
            │
            ▼
    Agent writes narrative report

For report generation, see Report Generation section below.

All Scripts

Canonical script location: skills/sf-civic-digest/scripts/

Script	What it does	Method	Notes
`sf_weekly_digest.py`	Master aggregator — calls all subscripts	—	Use `--json` for machine-readable output; pass to agent for synthesis
`sf_civic_digest.py`	Board of Supervisors + all committees via Legistar	curl	`--daily` for delta only; `--days N`; state in `sf_civic_state.json`
`sfmta_hearings.py`	SFMTA Engineering Public Hearings	curl + PDF	Street changes, parking meters, signals. Every ~2 weeks.
`sf_planning_notices.py`	SF Planning project notices (Section 311)	curl	Earliest signal layer — before commission hearings. Needs `Accept: text/html` header.
`sf_planning_commission.py`	Planning Commission, HPC, ZA full agendas	curl + PDF	Server-rendered HTML (no browser). `--body hpc` for HPC, `--body za` for ZA.
`sf_board_of_appeals.py`	Board of Appeals	Wagtail API	sf.gov CMS API (`api.sf.gov`). 1st & 3rd Wednesdays, 5pm. No browser needed.
`sf_building_permits.py`	Building permits citywide	Socrata REST	Every permit filed. Classified HIGH/NOTABLE/noise. State-tracked.
`sfgov_events.py`	Supervisor office hours, district events	REST/JSON	Very sparse — supervisor events only. Low signal.
`sf_311.py`	311 service requests by district	Socrata REST	Spike detection. Query two 7-day windows for trend comparison.
`sf_ethics.py`	Lobbyist contacts + campaign contributions	RSS + Socrata	Contacts: `5f5n-tdbf`. Contributions: `e6py-fg8b`. Column names have NO underscores.
`sf_rent_board.py`	Rent Board commission meetings	Wagtail API	sf.gov CMS API (`api.sf.gov`). No browser needed. Rent increase: links to sf.gov (no hardcoded value).
`sf_sfmta_board.py`	SFMTA Board of Directors	curl	1st & 3rd Tuesdays 1pm. Service changes, fare decisions, bike/ped safety.
`sf_rec_park.py`	Rec & Park Commission	curl (Granicus)	`view_id=91`. sfrecpark.org is unreachable from server — Granicus only.
`sf_housing_pipeline.py`	AB 2011 / SB 35 / density bonus housing pipeline	Socrata REST	Cross-refs parcels dataset for district. Watchlist tracks 400 Divisadero. State-tracked.
`sf_journalism.py`	News aggregator — 5 SF outlets	RSS	Mission Local, Streetsblog SF, SF YIMBY, 48 Hills, SF Standard. SF Examiner 404s. Archive: `sf_journalism_archive.json`.
`sf_volunteer_cleanups.py`	Community cleanup events	curl/JS bundle	Refuse Refuse SF + DPW Love Our City NBDs. GoDaddy SPA — data is in JS bundle, not HTML.
`sf_bart_board.py`	BART Board of Directors + advisory bodies	Legistar API	`webapi.legistar.com/v1/bart/Events`. Board = HIGH, advisory = MEDIUM. `--next` for next meeting.
`sf_sfusd_board.py`	SFUSD Board of Education meetings	Google Sheets + BoardDocs	2nd & 4th Tuesdays 5pm. Falls back to projected schedule if sources unavailable.
`sf_evictions.py`	Eviction notices by district	Socrata REST	Dataset `5cei-gny5`. Ellis Act + OMI = displacement signals. Trend comparison (two windows). `--district N`.
`sf_sfcta.py`	SF County Transportation Authority	curl (sfcta.org)	Congestion pricing, Muni Forward, bike/ped funding, Prop L sales tax. Board + CAC + committees.
`sf_civic_actions.py`	Protests, rallies, civic actions	Mobilize.us API + Indybay	No auth. Rallies/marches = HIGH, canvass/phone bank = MEDIUM. `--type rally` to filter.
`sf_mission_local.py`	Mission Local news only	RSS	Subset of sf_journalism.py. Use sf_journalism.py for multi-outlet.
`config_loader.py`	District config loader	—	All 11 districts pre-populated. Pass `--district N` to any script.

No browser required

All scripts now work without a browser. sf_board_of_appeals.py and sf_rent_board.py use the sf.gov Wagtail CMS API (api.sf.gov/api/v2/pages/). sf_planning_commission.py uses curl against server-rendered Drupal HTML (the sfplanning.org calendar pages are not JS-dependent despite initial assumptions).

District Configuration

All scripts accept --district N (1–11). That's all you need for a basic run.

User preferences and neighborhood context (streets, addresses, people to flag) live in USER.md — a human-readable profile file. There is no civic_config.json; that file has been removed. Read USER.md to understand what the user cares about before writing any report.

District→neighborhood mappings: references/sf-districts.md Build timeline benchmarks: references/sf-build-timelines.md — Read before writing about any housing project. Includes typical phase durations, financing tracking methods, and context for whether a project is on track or stalled.

Report Generation

Agent-driven (recommended for one-off reports)

bash

# 1. Collect data (use absolute paths — see path rule above)
python3 /absolute/path/to/sf-civic-digest/scripts/sf_weekly_digest.py --json > /tmp/civic_data.json
python3 /absolute/path/to/sf-civic-digest/scripts/sf_housing_pipeline.py --district 5 --json >> /tmp/civic_data.json
python3 /absolute/path/to/sf-civic-digest/scripts/sf_volunteer_cleanups.py --district 5 --days 14 --json >> /tmp/civic_data.json

# 2. Agent synthesizes following STYLE.md
# Read STYLE.md, read the JSON, write narrative to reports/

Agent-driven reports (quarterly, annual, deep dives)

Give your agent the workspace path and let it orchestrate:

Read STYLE.md and USER.md. Run the scripts in scripts/ to collect data for District 5, then write a Q1 2026 civic digest. Save to reports/.

⚠️ Path gotcha: Agents default to wrong relative paths. Always resolve the absolute path to scripts/ from the location of this SKILL.md before running anything. Use full absolute paths — e.g. python3 /full/path/to/scripts/sf_housing_pipeline.py. Never just scripts/SCRIPTNAME.py.

Report style

Read STYLE.md before writing any report. Key rules:

TLDR (5-7 bullets) first — last bullet is always one action
Officials box: YOUR elected officials only (supervisor, mayor, state reps)
Short sections (3-5 sentences), scan-and-skip
Connect threads across sources (lobbying + permits + 311 + journalism = one story)
End each thread with a specific action (date, email address, what to say)
Rates over counts. "Up 15%" beats "95 reports."
Tone: pro-building, pro-tech, car-free lens. See STYLE.md for full framing guidance.

Reference: see STYLE.md for rules and examples.

Key Data Gotchas

These will burn you if you don't know them:

Socrata column names:

311 (vw6y-z8j6): supervisor_district = '5.0' (float string), date field = requested_datetime
Building Permits (i98e-djp9): supervisor_district = '5'. Address split across street_number/street_name/street_suffix — no combined field. estimated_cost is a string, cast in Python.
Lobbyist Contacts (5f5n-tdbf): NO underscores — officialname, clientname, lobbyistname
Lobbyist Contributions (e6py-fg8b): same — lobbyistname, candidatename, sourceoffunds
Planning Records (qvu5-m3a2): pim_link returns {"url": "..."} dict, not string. No supervisor_district — cross-reference parcels (acdm-wktn) by block/lot.

Legistar: The API is broken for SF (returns "Agenda Draft Status" error). Use the RSS feed instead: https://sfgov.legistar.com/Feed.ashx?M=C&ID=17442&GUID=EEE85B7C-1A1C-4A56-873E-355A0A0DE5C3&Mode=All&Format=rss Returns ~100 meetings (~6 months). Parse meeting IDs from RSS, then fetch_meeting_detail() for agenda items.

Reddit: Direct curl blocked (403, datacenter IP). Use web_fetch tool — routes differently, works.

sfrecpark.org: Completely unreachable from this server. Granicus (view_id=91) is the only path.

311 trend pattern: Query two 7-day windows (current + prior week) and show week-over-week delta. "Encampments up 15%" is useful. "95 encampments" is not.

Journalism archive: sf_journalism.py persists articles to sf_journalism_archive.json (cap 2000). For quarterly history, use --archive --days 90.

Socrata Dataset Reference

Dataset	ID	Key fields
Building Permits	`i98e-djp9`	permit_number, permit_type, address (split), supervisor_district ('5'), estimated_cost (string)
311 Cases	`vw6y-z8j6`	service_name, requested_datetime, supervisor_district ('5.0')
Planning Dept Records	`qvu5-m3a2`	record_id, description, record_status, project_address, block, lot, number_of_units_net, pim_link (dict)
SF Parcels	`acdm-wktn`	block_num, lot_num, supervisor_district, zoning_code
Lobbyist Contacts	`5f5n-tdbf`	officialname, lobbyistname, clientname, outcomesought, subjectarea, filenumber
Lobbyist Contributions	`e6py-fg8b`	lobbyistname, candidatename, sourceoffunds, committeename
Assessor Parcels	`fk72-cxc3`	property_location, parcel_number, assessed_land_value, zoning_code
Eviction Notices	`5cei-gny5`	eviction_id, address, neighborhood, file_date, supervisor_district, ellis_act_withdrawal, owner_move_in

Query pattern: https://data.sfgov.org/resource/{id}.json?$where=supervisor_district='5'&$order=date+DESC&$limit=50

Scraping Method Reference

Site	curl works?	Browser needed?
Legistar (sfgov.legistar.com)	✅	No
SFMTA (sfmta.com)	✅	No
SF Planning notices (sfplanning.org/notices)	✅ (needs Accept header)	No
SF Planning calendars (sfplanning.org/hearings-*)	✅ (use `-list` views)	No
SF.gov meetings (api.sf.gov)	✅ (Wagtail API)	No
Socrata (data.sfgov.org)	✅	No
SF Ethics (sfethics.org)	✅	No
Mission Local / journalism RSS	✅	No
Granicus (Rec & Park)	✅	No
Refuse Refuse SF (refuserefusesf.org)	✅ (JS bundle)	No
sfrb.org	✅ (Wagtail API via api.sf.gov)	No

Scheduling

Weekly digest (Monday morning)

json

{
  "schedule": { "kind": "cron", "expr": "0 14 * * 1", "tz": "America/Los_Angeles" },
  "payload": { "kind": "systemEvent", "text": "Run SF civic weekly digest and send summary" }
}

Daily delta (weekday noon check-in)

json

{
  "schedule": { "kind": "cron", "expr": "0 12 * * 1-5", "tz": "America/Los_Angeles" },
  "payload": { "kind": "systemEvent", "text": "Run SF civic daily delta and report any new agenda items" }
}

File Layout

text

skills/sf-civic-digest/
├── SKILL.md              ← you are here
├── STYLE.md              ← report style guide (read before writing any report)
├── USER.md               ← user profile: district, neighborhoods, streets, interests (fill this in)
├── .gitignore            ← excludes runtime state from version control
├── scripts/              ← all data pipeline scripts
│   ├── sf_weekly_digest.py      ← master aggregator
│   ├── sf_civic_digest.py       ← Board of Supervisors (Legistar)
│   ├── sfmta_hearings.py        ← SFMTA Engineering Hearings
│   ├── sf_planning_notices.py   ← Planning project notices
│   ├── sf_planning_commission.py← Planning Commission / HPC / ZA
│   ├── sf_board_of_appeals.py   ← Board of Appeals
│   ├── sf_building_permits.py   ← Building permits (Socrata)
│   ├── sfgov_events.py          ← Supervisor events
│   ├── sf_311.py                ← 311 service requests (Socrata)
│   ├── sf_ethics.py             ← Lobbyist contacts + contributions
│   ├── sf_rent_board.py         ← Rent Board
│   ├── sf_sfmta_board.py        ← SFMTA Board of Directors
│   ├── sf_rec_park.py           ← Rec & Park Commission
│   ├── sf_housing_pipeline.py   ← Housing pipeline (AB2011/SB35/density bonus)
│   ├── sf_journalism.py         ← News aggregator (5 outlets)
│   ├── sf_volunteer_cleanups.py ← Community cleanups (Refuse Refuse + DPW)
│   ├── sf_evictions.py          ← Eviction notices (Socrata)
│   ├── sf_bart_board.py         ← BART Board of Directors (Legistar)
│   ├── sf_sfusd_board.py        ← SFUSD Board of Education
│   ├── sf_sfcta.py              ← SF County Transportation Authority
│   └── config_loader.py         ← District config loader
├── reports/              ← generated reports
│   └── archive/          ← older versions
└── references/
    ├── sf-districts.md       ← district map, supervisor names, neighborhood lists
    └── sf-build-timelines.md ← build phase benchmarks, financing signals, project context

SF Civic Digest

Install

SF Civic Digest — SKILL.md

Quick Start

Architecture

All Scripts

No browser required

District Configuration

Report Generation

Agent-driven (recommended for one-off reports)

Agent-driven reports (quarterly, annual, deep dives)

Report style

Key Data Gotchas

Socrata Dataset Reference

Scraping Method Reference

Scheduling

Weekly digest (Monday morning)

Daily delta (weekday noon check-in)

File Layout

Participation Guide

How to comment at SFMTA Engineering Hearings

How to comment at Planning Commission / HPC

How to reach your supervisor

SF Civic Digest

Install

SF Civic Digest — SKILL.md

Quick Start

Architecture

All Scripts

No browser required

District Configuration

Report Generation

Agent-driven (recommended for one-off reports)

Agent-driven reports (quarterly, annual, deep dives)

Report style

Key Data Gotchas

Socrata Dataset Reference

Scraping Method Reference

Scheduling

Weekly digest (Monday morning)

Daily delta (weekday noon check-in)

File Layout

Participation Guide

How to comment at SFMTA Engineering Hearings

How to comment at Planning Commission / HPC

How to reach your supervisor

Related skills