ClawHub

K8s Prefect Flow Builder

Dev Tools

Build, modify, and review Prefect-based offline orchestration in this repository. Use when adding a new Prefect flow, wrapping an existing offline computation as a flow or task, editing prefect.yaml, splitting deployments by resource profile, configuring resources or concurrency, or aligning CI, Prefect, and Kubernetes deployment behavior.

Install

openclaw skills install k8s-prefect-flow-builder

Prefect Flow Builder

Overview

Use this skill to add or refactor Prefect-managed offline workflows in this repository without mixing orchestration concerns into business logic.

Workflow

  1. Classify the change.
  • New flow or major refactor: read references/flow-design.md and references/template-prefect-yaml.md.
  • Deployment-only or config change: read references/deployment-patterns.md.
  • Resource or concurrency tuning: read references/resources-and-concurrency.md.
  • If the system model is unclear, read references/architecture.md first.
  1. Keep orchestration separate from compute.
  • Put heavy business logic in reusable job or service modules under src/core/....
  • Keep Prefect wrappers in src/prefect_flows/....
  • Use tasks for independently observable units or meaningful side effects; avoid exploding one logical step into many tiny tasks just for structure.
  • Choose task invocation mode deliberately: direct call for simple serial execution, .submit() or .map() for in-flow concurrency, and .delay() only for background execution on separate infrastructure.
  • If a child flow is intentionally serial, call tasks directly inside the loop and let the child flow own error aggregation and final failure semantics. Do not default to .submit() just because the unit is a task.
  • Reserve .submit() for cases that actually need Prefect futures, parallel fan-out, or non-blocking wait and collection behavior, and make sure terminal futures are resolved before the flow returns.
  • Introduce child flows only when you need a separate scaling, resource, or failure boundary.
  1. Put concurrency controls at the right layer.
  • Use task-runner concurrency for in-flow task execution only; it is not a substitute for deployment or infrastructure throttling.
  • Use deployment, work-pool, worker, or work-queue limits to control how many flow runs the platform launches.
  • Use tag-based concurrency limits when many tasks across flows share an external bottleneck such as a database, API, or memory-heavy resource.
  1. Treat prefect.yaml as deployment source of truth.
  • Put deployment names, schedules, work pool selection, resources, concurrency policies, and deployment parameter defaults in prefect.yaml.
  • Let CI provide only deploy-time values such as PREFECT_API_URL and PREFECT_DEPLOY_IMAGE.
  • Keep runtime business env in Kubernetes via env_from, not CI.
  • Keep infrastructure choices aligned with Prefect's worker model: deployments target work pools, and workers poll compatible pools to execute runs.
  1. Make failures observable.
  • Raise exceptions instead of returning non-zero codes silently.
  • Include key context in exception messages and logs.
  • Use readable task and flow names so Prefect UI can identify the failing unit quickly.
  • When a batch must continue after per-item failures, convert each item into a structured result, finish the batch, then raise once at the flow boundary if the aggregate outcome should be failed.
  1. Validate in this order.
  • Run focused tests for changed flow and job code.
  • Review prefect.yaml for deployment or resource drift.
  • Trigger a small manual run before enabling or changing schedules.
  • If the question is about Prefect semantics rather than repo conventions, verify against the official v3 docs before codifying a pattern.

Repository Anchors

  • Flow entrypoints: src/prefect_flows/
  • Reusable compute logic: src/core/
  • Deployment definitions: prefect.yaml
  • Team guide: docs/cg_offline_prediction/prefect_orchestration_overview.md

Prefect v3 Defaults

  • Tasks support three distinct execution modes in v3: direct call blocks and returns a result, .submit() returns a PrefectFuture for concurrent execution in the same flow, and .delay() is for background execution on separate workers.
  • Task runners are optional. If you are not intentionally using concurrency, do not introduce .submit() or a task runner configuration just to preserve the task decorator.
  • Task states are orchestrated client-side and may appear with eventual consistency in the UI, so design recovery and resume logic around durable business outcomes instead of assuming every intermediate task-state transition is the source of truth.

References

More in Dev Tools