Paperless Cloud Intake Hardening

Procedure-only Paperless cloud-folder intake hardening with polling and post-delivery consumer guidance.

Purpose

Use this skill when a Paperless-ngx instance consumes files from a folder fed by cloud storage such as OneDrive, iCloud Drive, Dropbox, Google Drive for Desktop, Synology Drive, Box, or any File Provider-backed folder, and there is risk that Paperless may see placeholder, zero-byte, partial, or not-yet-materialized files.

The goal is to preserve simple local automation where possible while making the Paperless handoff reliable.

Privacy And Anonymization

When documenting or sharing this workflow, keep implementation details generic.

Do not include client names, instance-specific Paperless names, private document names or contents, private usernames, hostnames, filesystem paths, volume names, organization names, secrets, database credentials, URLs, tokens, or internal network details.

Use neutral terms such as source cloud folder, Paperless consume folder, Paperless instance, deployment, state directory, and staging directory.

When examples are needed, use placeholder paths such as:

text

/path/to/source-cloud-folder
/path/to/paperless/consume
/path/to/paperless-intake-state/staging

When To Use

Use this skill when:

• Paperless-ngx is ingesting from a cloud-synced folder.
• Files appear in the consume folder as zero bytes or fail before fully downloading.
• A folder action, Automator workflow, Shortcuts automation, Hazel rule, cron job, launchd job, systemd timer, Synology task, or similar local mechanism moves files into Paperless.
• No trigger exists yet, but the user wants to create a safe cloud-folder-to-Paperless intake path.
• The user wants a pragmatic local fix rather than a new platform or custom daemon.
• Multiple Paperless instances may eventually share the same intake pattern.

Do not use this skill when the issue is Paperless OCR/classification quality rather than file readiness, the files are already on a fully local filesystem and failures are unrelated to cloud placeholders, or no cloud-backed source folder exists and the user does not want to create one.

Core Principle

Never let Paperless see a file until it is complete.

Use a staging directory outside the Paperless consume folder. Force the cloud provider to materialize real bytes by reading or streaming the source file into staging. Verify that the staged file is nonzero and complete enough for the use case. Then atomically move the finished staged file into the Paperless consume folder.

Trigger Strategy

An existing Automator workflow or folder action is not required.

There are two valid modes:

1. Retrofit an existing trigger.

   If an Automator folder action, Shortcuts automation, Hazel rule, launchd job, cron job, systemd timer, Synology task, or other local trigger already exists, inspect it first. Preserve it if it is adequate, and change it to call the safe staging script instead of moving files directly into Paperless.

2. Create a new trigger.

   If no trigger exists, ask for or discover the two required folders: the source cloud folder and the Paperless consume folder. Then create the simplest suitable trigger for the platform.

Prefer an event trigger for responsiveness and a periodic retry scan for reliability. If only one mechanism is appropriate, prefer the periodic retry scan because folder events can fire before cloud files are ready.

On macOS, verify scheduler access separately from scheduler health. A launchd job can be loaded, watch paths, run on interval, and exit with code 0 while still being blocked by privacy controls from enumerating a File Provider or CloudStorage folder. Treat Operation not permitted during scans as a degraded retry state, not as an empty folder.

For macOS File Provider folders, prefer a narrow automation permission boundary before broad Full Disk Access. A good pattern is:

• Keep the portable intake logic in a generic script.
• Add a mode that processes only explicit file paths handed to it, without enumerating the source folder itself.
• Use Automator, Shortcuts, Hazel, Finder AppleScript, or another user-facing automation layer to enumerate the cloud folder.
• Pass the explicit file paths into the generic script for staging, verification, duplicate handling, and atomic handoff.
• Avoid granting Full Disk Access to broad runtimes such as /bin/bash unless the user explicitly accepts that wider security surface.

Deployment Discovery

A Docker Compose file is helpful but not required.

Paperless may be deployed through Docker Compose, Portainer stacks, Synology Container Manager or Synology Docker, direct docker run containers, bare-metal/system package installation, or another orchestrator.

Discover the Paperless consume folder and verification method using whatever evidence is available: Compose or stack files, docker ps, docker inspect, Portainer metadata if available and approved, Synology configuration, known local directory layout, Paperless environment variables, Paperless logs, Paperless API, or database records, preferably read-only.

Do not require a compose file before proceeding. If the instance is stopped and no runtime metadata is available, report what can be inferred from the filesystem and ask before starting containers or changing deployment state.

Preferred Design

1. Keep the existing trigger if it is adequate.

   Examples: Automator folder action, macOS Shortcuts folder automation, Hazel, launchd, cron, systemd timer, Synology task, or simple watcher.

2. Make the trigger lightweight.

   The trigger should call a script and return quickly. It should not do fragile copy or move work inline.

3. Use a staging directory outside Paperless consume.

4. Stream the source file into staging.

   On cloud-backed paths, reading the file usually forces materialization. A simple tool like cat, dd, rsync, or a platform-appropriate equivalent can be enough.

5. Verify before handoff.

   Minimum checks: source exists and is a regular file, extension or MIME type is allowed, source-reported size is greater than zero, staged copy is greater than zero, staged copy is not smaller than the source-reported size when trustworthy enough, and optional checksum is available for idempotency.

6. Use atomic handoff.

   Move the completed staged file into the Paperless consume folder. Paperless should only see the final filename after the file is complete.

7. Add retry outside the event trigger.

   Folder events can fire too early and may not fire again after the cloud provider materializes a file. Add a periodic scan using launchd, cron, systemd timer, Automator, Shortcuts, Hazel, or equivalent when the platform allows reliable source-folder access.

8. Add simple concurrency protection.

   A mkdir lock directory is often enough for local scripts. For longer-term use, include stale-lock detection based on PID and timestamp.

9. Verify with Paperless itself.

   Do not rely only on the consume folder. Check Paperless logs, task outcomes, database records, or API results to confirm documents were actually consumed.

Paperless Consumer Detection

Do not assume that moving a file into the Paperless consume folder is enough for immediate ingestion.

Paperless commonly watches the consume folder using filesystem notifications such as inotify. In Docker Desktop, macOS, network mounts, cloud-backed folders, and some bind-mount scenarios, those filesystem events may not reliably reach the container even when the file is visible inside the container.

If Paperless sees files in consume but does not automatically consume them:

• Confirm the files are visible inside the Paperless container or runtime.
• Confirm the consumer process is running.
• Confirm the files are readable and complete.
• Run Paperless's one-shot consumer or equivalent management command/API to determine whether ingestion works when explicitly triggered.
• Prefer enabling native Paperless polling, such as PAPERLESS_CONSUMER_POLLING, as the stack-level baseline when filesystem notifications are unreliable.
• Consider adding a post-delivery hook from the intake automation to explicitly trigger Paperless consumption after successful handoff.

A good long-term pattern is:

• Native Paperless polling for general reliability, including files placed manually in consume.
• A post-delivery one-shot consumer hook for this automation path, so files handed off by the staging script are processed promptly.

Keep the post-delivery hook optional and per deployment. Do not assume every Paperless deployment needs it.

Investigation Checklist

Before changing anything:

1. Identify the Paperless consume folder.
2. Identify the source cloud-backed folder, or confirm that the user wants to create one.
3. Locate the current trigger or automation, if one exists.
4. If there is no cloud-backed source folder and the user does not want to create one, stop and report that the hardening pattern is not currently applicable.
5. If there is no trigger but a source folder and consume folder are known, propose creating the simplest suitable trigger.
6. Inspect existing logs and consume leftovers.
7. Count zero-byte or failed files in consume.
8. Determine how Paperless is deployed; do not assume Docker Compose.
9. Identify Paperless logs and database/API access method.
10. Check whether Paperless's consumer detects files automatically or needs polling/explicit triggering.
11. Confirm whether the user wants cleanup of old failed files or only future hardening.

Implementation Checklist

A minimal robust implementation should include:

• Configurable source folder.
• Configurable consume folder.
• Configurable state/log folder.
• Staging outside consume.
• Allowlist for file extensions or MIME types.
• Stream/read copy to force materialization.
• Nonzero staged file validation.
• Optional size comparison.
• Checksum-based duplicate skip.
• Atomic move into consume.
• Log messages for delivered, skipped duplicate, waiting, timeout, and error states.
• A periodic retry mechanism when the scheduler can access and enumerate the source folder.
• For macOS/File Provider cases where background enumeration is blocked, an explicit-path retry mode plus a user-facing automation enumerator.
• Optional post-delivery command or API call to trigger Paperless consumption when native filesystem notifications are unreliable.
• Explicit diagnostics when retry cannot access the source folder.
• Locking to prevent concurrent scans.

Verification Checklist

After implementation:

1. Run syntax/lint checks for the script and scheduler config.
2. Run a manual scan or explicit-path retry test.
3. Confirm no new zero-byte files appear in consume.
4. Confirm the scheduler is loaded/enabled and exits successfully.
5. Confirm the scheduler can actually enumerate or receive paths from the source cloud folder; on macOS, do not assume launchd access just because the job is healthy.
6. Confirm files appear inside Paperless's consume path as seen by the Paperless runtime/container.
7. Confirm Paperless automatically consumes the delivered files, or confirm polling/post-delivery triggering is in place.
8. Confirm Paperless successfully creates document records.
9. Confirm duplicate handling does not re-submit already consumed files unnecessarily.
10. Leave old failed files untouched unless the user explicitly approves cleanup.

Cleanup Guidance

Old zero-byte files in consume should not be deleted blindly.

First check whether the real versions have already been consumed. Good evidence includes matching Paperless document checksum, matching title or original filename, Paperless task logs showing successful consumption, or user confirmation that copies were re-added and consumed.

If all failed placeholders are accounted for, move or delete the zero-byte leftovers only with user approval.

Design Bias

Prefer simple existing local tools over bespoke services.

Good fits include Automator, Shortcuts, Hazel, launchd, cron, systemd timers, Synology tasks, a small staging script, Paperless native polling, optional one-shot consumer triggering, and Paperless logs/API/database reads for verification.

Avoid a custom long-running daemon, queue database, direct database edits for intake, complex file watching frameworks, or notification systems before basic logs prove insufficient.

Generalization Notes

For one instance, hardcoded paths may be acceptable if the script is private and local.

For several Paperless instances, use per-instance config files and one shared script or template. Instance-specific values should include source folder, consume folder, state/log folder, allowed extensions, scheduler label/name, maximum copy timeout, duplicate policy, and whether to trigger Paperless after delivery.

Keep platform-specific wrappers separate from the generic intake script. For example, a macOS AppleScript or Shortcut can enumerate a File Provider folder and pass explicit paths to the shared script, while Linux can use cron or systemd to run a normal scan.

Example Pattern

bash

# Trigger command used by Automator, Shortcuts, Hazel, or another folder action.
/path/to/paperless-cloud-intake.sh --config /path/to/instance.conf --kick "$@"

# Periodic retry command used when the scheduler can enumerate the source folder.
/path/to/paperless-cloud-intake.sh --config /path/to/instance.conf --scan

# Retry command used by a macOS automation wrapper that already enumerated the files.
/path/to/paperless-cloud-intake.sh --config /path/to/instance.conf --paths-only /path/to/file1 /path/to/file2

Example optional per-deployment post-delivery behavior:

text

After a successful atomic handoff, trigger Paperless's own one-shot consumer, API import, or another approved native ingestion mechanism.

Do not publish real deployment commands, hostnames, paths, credentials, or client-specific labels in community examples.

Lessons Learned

Cloud sync settings such as "keep this folder locally" are not always sufficient. Some providers still expose files through placeholder semantics and may not materialize bytes until a user or process explicitly reads the file.

Paperless consume folders should be treated as final handoff locations, not staging areas.

Folder events are hints, not reliable proof that a file is ready.

A scheduler being loaded and exiting successfully is not the same as proving it can access a cloud-backed folder. Verify the actual scan result and log access failures explicitly.

On macOS, a narrower user-facing automation enumerator can be safer than granting broad Full Disk Access to a shell runtime. Keep the permission-sensitive enumeration separate from the generic staging-and-handoff logic.

In Docker Desktop, macOS, network-mounted, or cloud-adjacent deployments, Paperless may see files in consume but miss the filesystem event. Use native polling as the baseline fix, and consider a post-delivery consumer trigger for files delivered by the automation.

A small staging-and-atomic-move pattern plus native Paperless consumption verification is often more reliable than trying to perfectly detect every cloud provider's placeholder state.