Workflow

Build automated pipelines with reusable components, data flow between nodes, and state management.

Architecture

Workflows live in workflows/ with components and flows:

text

workflows/
├── index.md                 # Inventory with tags
├── components/
│   ├── connections/         # Auth configs
│   ├── nodes/               # Reusable operations
│   └── triggers/            # Event sources
└── flows/{name}/
    ├── flow.md              # Definition
    ├── config.yaml          # Parameters
    ├── run.sh               # Executable
    ├── state/               # Persistent between runs
    └── logs/

Quick Reference

Topic	File
Directory layout, naming, formats	structure.md
Data passing between nodes	data-flow.md
Cursor, seen set, checkpoint	state.md
Retry, rollback, idempotency	errors.md
Connections, nodes, triggers	components.md
Create, test, update, archive	lifecycle.md

Requirements

• jq — JSON processing
• yq — YAML config parsing
• curl — HTTP requests
• flock — Lock files to prevent concurrent runs
• Secrets in macOS Keychain (security find-generic-password)

Data Storage

• Location: workflows/ in workspace root
• State: flows/{name}/state/ — cursor.json, seen.json, checkpoint.json
• Logs: flows/{name}/logs/ — JSONL per run
• Data: flows/{name}/data/ — intermediate files between nodes

Core Rules

1. Data Flow Pattern

Each node writes output to data/{NN}-{name}.json. Next node reads it.

bash

curl ... > data/01-fetch.json
jq '...' data/01-fetch.json > data/02-filter.json

Breaking this pattern = nodes can't communicate.

2. Error Declaration

Every node in flow.md MUST declare:

• On error: retry(N) | fail | continue | alert
• On empty: skip | continue | fail

Undefined behavior = unpredictable workflow.

3. Lock Files

Prevent concurrent runs:

bash

LOCKFILE="/tmp/workflow-${NAME}.lock"
exec 200>"$LOCKFILE"
flock -n 200 || exit 0

4. State Files

File	Purpose
cursor.json	"Where did I leave off?"
seen.json	"What have I processed?"
checkpoint.json	"Multi-step recovery"

5. Component Reuse

Before creating new connections/nodes/triggers:

bash

ls workflows/components/connections/
ls workflows/components/nodes/

Use existing. Update "Workflows Using This" when adding.

---

Related: For LLM-driven multi-phase processes, see the cycle skill.