fulcra-ingest

Autonomously orchestrate the ingestion of 3rd-party data exports (e.g., Spotify, Netflix) from the Fulcra File Store into properly mapped Fulcra Annotations.

This skill orchestrates the ingestion of 3rd-party data exports that the user has uploaded to their Fulcra File Store. It profiles the data schemas, creates idempotent Fulcra Annotation mappings, and ingests the data points.

General Guidelines

• Zero User Friction: Assume the user has dumped a raw ZIP/JSON/CSV into their Fulcra ingest. Do not ask them to map schemas manually unless absolutely necessary for a completely unrecognized format.
• Automation: You can set up a cron job, a continuous loop, or a heartbeat reminder (depending on what is appropriate for the running agent) to periodically check the ingest/ directory for new files or to fetch data from APIs and other sources. You must explicitly confirm what data will be collected, how often, where it will be stored, and how to stop the operation before establishing automated fetching.
• Initial Setup Guidance: If the user is just setting up this skill for the first time, help them ingest and verify their first data source so they can see it working end-to-end.
• File Filtering: Strictly ignore the _meta/ subdirectory and any .md files found in the ingest root to prevent attempting to ingest the agent's own OKF tracking files.
• Idempotency: Never create duplicate schemas. Always use the annotation's description field to store the specific namespace (e.g., com.fulcradynamics.annotation.ingest.spotify) and check the catalog first.
• Interactive Feedback & Logging: Maintain a running log of ingest events in ingest/_meta/ingest_log.md detailing what was ingested, record counts, and start/end dates. When completing an interactive ingestion or when asked about recent ingests, use this log to generate a fun, simple ASCII chart summarizing the data (e.g., a timeline or simple visualization showing start/end dates and record counts).

Advanced Data Capabilities

Beyond straightforward 1:1 data transfer, you can perform advanced transformations and enrichments to maximize the value of the user's data:

1. Enhance Data (Augmentation): You can augment raw data by fetching additional context from auxiliary sources before ingestion. For example, when ingesting a Netflix viewing history, you could search the web to append actors, genres, or production dates to the records.
2. Create Computed Data Tracks: You can generate entirely new data types informed by existing or newly ingested data. For example, you could create a "Rolling Weekly Song Count" track, or a "Movie Compatibility" track if you have viewing data for multiple users. Compute these insights and ingest them into their own dedicated Fulcra Annotations.
3. Adjust Existing Data: You can adjust the tag scheme, values, or anything about existing data. To do this, delete the old records via the fulcra-api delete command, update the schema (creating a new data type if necessary), and re-ingest the data. You must obtain explicit user confirmation before running the fulcra-api delete command. You can re-import previously archived files from ingest/_meta/archive/artifact/ to process the new tags. Before modifying or re-processing existing records, confirm the plan with the user.

References

• references/fulcra-ingest-cli.md: Contains the necessary fulcra-api CLI commands for checking the catalog, listing files, and creating new data types, as well as providing the exact batch ingestion syntax, JSON schemas, and tagging instructions required for ingesting records to Fulcra Annotations.
• references/fulcra-ingest-source-mapping.md: Outlines the structure and workflow for maintaining the ingest/_meta/source_map.md file, which tracks data lineage, prevents duplicate schemas, handles ingest versioning, and logs archived files.
• scripts/generate_deterministic_id.py: A python script that takes arbitrary string arguments and returns a consistent, deterministic UUID. Use this to ensure idempotency across ingested records.

The Pipeline

0. The Fetcher (API/CLI Extraction) (If applicable)

   • You may inform the user that you can process manual file uploads or extract data from accessible APIs, local services, and CLI tools on their behalf.
   • You must receive explicit user consent before fetching data from any external API, local service, or CLI tool. Confirm the exact source, destination, frequency, and privacy implications with the user.
   • If the user explicitly requests to ingest data from an API, CLI tool, or local service rather than pointing to a pre-uploaded file, first fetch the data programmatically.
   • Save the fetched data to a structured file (e.g., CSV or JSON) in the local workspace.
   • Upload the file to the Fulcra File Store drop-zone using uvx fulcra-api file upload <local_file> ingest/<filename>.
   • Once uploaded, proceed with the standard triage process below.

1. Triage

   • Use uvx fulcra-api file list to check the ingest/ directory. Explicitly ignore the _meta/ folder and any .md files.
   • If new data files are found, proceed to process them directly.

2. Profiling & Ingestion

   • Retrieval: Execute uvx fulcra-api file download <file_id> ./<filename>.
   • Source Mapping & Schema Resolution: Crucial: You must strictly follow the agent workflow outlined in references/fulcra-ingest-source-mapping.md. Rely on the source_map.md registry to resolve the target schema ID. If you need to create a new schema for an unseen source, consult references/fulcra-ingest-cli.md for the correct CLI commands and base types.
   • Data Ingestion: Write and execute a Python script to parse the file.
     • Generate deterministic UUIDs for id using scripts/generate_deterministic_id.py (ensure you pass the source identifier followed by the specific ID fields to prevent cross-service collisions).
     • Construct the payload and push the records using the fulcra-api record command exactly as specified in references/fulcra-ingest-cli.md.

3. Cleanup & Archive

   • Archive the processed file by downloading it from ingest/ and re-uploading it to ingest/_meta/archive/artifact/. When archiving, prefix the filename with a timestamp in the format YYYYMMDD-HHMMSS.
   • Example commands:
     • uvx fulcra-api file download ingest/NetflixViewingHistory.csv ./NetflixViewingHistory.csv (if not already downloaded)
     • uvx fulcra-api file upload ./NetflixViewingHistory.csv ingest/_meta/archive/artifact/20260625-143000_NetflixViewingHistory.csv
   • Safety Check: Before deleting the original file, you MUST verify that the file was successfully uploaded to the archive path (e.g., uvx fulcra-api file list ingest/_meta/archive/artifact/ or uvx fulcra-api file stat ingest/_meta/archive/artifact/20260625-143000_NetflixViewingHistory.csv). This is a move operation, so ensuring the destination exists is critical to avoid data loss.
   • You may execute fulcra-api file delete autonomously to remove the original file from the ingest/ directory (preventing re-ingestion), but only if the user explicitly approved the full ingest-and-archive lifecycle when the skill or automation was initially configured. If this is a one-off interactive session without prior blanket approval, verify the deletion with the user.
   • Finalize the process by updating the source_map.md in memory. Before overwriting the remote file, ensure you have downloaded the latest version to prevent wiping out concurrent changes. Then, upload it back to Fulcra, as instructed in the source mapping reference.
   • Update the Ingest Log: Append a brief summary of the completed ingestion to ingest/_meta/ingest_log.md (create it if it doesn't exist), including the source, start and end dates of the data, and total records ingested. Save it locally and upload it back to ingest/_meta/ingest_log.md via the Fulcra CLI.
   • User Handoff: If the user is present or asked about the ingestion directly, inform them that their data is being processed. Present them with a fun ASCII chart representing the data you just ingested (using the aggregate metrics from your ingest log). Point them to https://context.fulcradynamics.com/timeline?mode=week&date=YYYY-MM-DD to view their new data, where YYYY-MM-DD is calculated as six days before the latest recorded_at value in the ingested dataset (this ensures the week view includes the newest data point). Mention that for large datasets, it may take a little time for all records to fully appear on the timeline. Remind the user that if they ever want to change the tagging scheme or fix a mistake, they can simply ask you to correct the data and you will automatically handle the deletion and re-ingestion.