docker-fclones-scheduler

README

docker-fclones-scheduler

Find and deduplicate files on a schedule — reclaim wasted disk space automatically.

What it does

Wraps the fclones duplicate file finder in a Go scheduler daemon with interval-based scheduling and a CLI health probe. Supports the group, link, remove, and dedupe actions with configurable arguments. Reports scan statistics including duplicates found, space reclaimable, and files processed. All output goes to stdout/stderr for collection by log aggregators (Alloy, Promtail, etc.) and alerting via Grafana or similar.

• Mount your media directory and schedule periodic scans — fclones finds duplicates and can replace them with hardlinks or remove them entirely
• Pipe container logs to your observability stack for alerting
• Supports the group (report only), link (replace with hardlinks), remove (delete duplicates), and dedupe (reflink/copy-on-write) actions
• Configurable scan interval, paths, and fclones arguments
• Built-in scheduler, or hand scheduling to an external scheduler (cron, Ofelia, etc.) via the scan subcommand
• Built-in Docker healthcheck with automatic recovery

Why this design

• Scheduler your way — ships with a self-contained Go interval scheduler so you don't need external cron, systemd timers, or orchestrator-level scheduling. If you already run a central scheduler (Ofelia, cron), set FCLONES_INTERVAL=off and trigger scans with docker exec fclones /app/wrapper scan instead
• One owner for every run — the daemon (PID 1) executes every scan; the scan subcommand is a thin client on the daemon's local unix socket that relays the run's result as its exit code. Runs are serialized in one queue, and every run's logs land on the container's own log stream in both scheduling modes, so the same alert rules work everywhere
• Machine-readable report contract — the scan consumes fclones' JSON report (-f json) with a strict decoder, so an upstream output-format change fails the run loudly instead of silently zeroing the duplicate stats your alerting reads
• Distroless and rootless — runs as nonroot (UID 65532) on gcr.io/distroless/static-debian13 with no shell or package manager, minimizing attack surface
• Dangerous flags blocked by default — --command, --transform, --in-place, and --no-copy are rejected unless you explicitly opt in with FCLONES_ALLOW_UNSAFE=true, preventing command injection via environment variables
• Structured logs for observability — all output goes to stdout/stderr in a format ready for log aggregators, enabling alerting on scan failures or duplicate detection without custom exporters. Timestamps are UTC, so log lines are zone-stable regardless of the container's TZ

Quick start

The image is published to both GHCR (ghcr.io/cplieger/docker-fclones-scheduler) and Docker Hub (cplieger/docker-fclones-scheduler) — identical contents, use whichever you prefer.

services:
  fclones:
    image: ghcr.io/cplieger/docker-fclones-scheduler:latest
    container_name: fclones
    restart: unless-stopped
    user: "1000:1000"  # match your host user

    environment:
      FCLONES_INTERVAL: "1h"  # Go duration (e.g. 1h, 30m, 12h)
      FCLONES_SCAN_PATHS: "/scandir"
      FCLONES_ARGS: "--rf-over 1"
      FCLONES_ACTION: "link"  # group (report), link (hardlink), remove (delete), or dedupe (reflink/copy-on-write)
      FCLONES_ACTION_ARGS: "--priority bottom"

    volumes:
      - "/path/to/media:/scandir"
      - "/opt/appdata/fclones:/cache"

Scheduling modes

The container runs in one of three modes, selected by FCLONES_INTERVAL.

Built-in scheduler (default)

Set FCLONES_INTERVAL to a positive Go duration (1h, 30m, 12h, …). The container runs a scan at startup and then every interval. This is the zero-dependency default; nothing else is required. An unset, unparseable, or negative value falls back to the 3h default cadence in this mode (a negative value is treated as a typo and logged as a warning).

External scheduler

Set FCLONES_INTERVAL=off (alias: disabled). The container stays running but idle, and you trigger each scan out-of-band by exec'ing the scan subcommand:

docker exec fclones /app/wrapper scan

The scan subcommand is a thin client on the daemon's local unix socket: it submits one run request, blocks until the daemon has executed the scan, and exits non-zero on failure — the run itself executes inside the daemon (PID 1), so its full output lands on the container's log stream while the client relays only lifecycle lines and the result to the trigger. The daemon updates the same health marker the healthcheck reports. This lets a central scheduler own the cadence. Example with Ofelia labels:

services:
  fclones:
    image: ghcr.io/cplieger/docker-fclones-scheduler:latest
    container_name: fclones
    restart: unless-stopped
    user: "1000:1000"
    environment:
      FCLONES_INTERVAL: "off"   # disable built-in loop; Ofelia drives it
      FCLONES_SCAN_PATHS: "/scandir"
      FCLONES_ACTION: "link"
    labels:
      ofelia.enabled: "true"
      ofelia.job-exec.fclones-scan.schedule: "@every 6h"
      ofelia.job-exec.fclones-scan.command: "/app/wrapper scan"
      ofelia.job-exec.fclones-scan.no-overlap: "true"
    volumes:
      - "/path/to/media:/scandir"
      - "/opt/appdata/fclones:/cache"

Runs never overlap: the daemon executes all requests — scheduled ticks and triggered scans alike — strictly in order from one queue (a trigger that arrives mid-run queues behind it and reports its own run's result). An advisory file lock (flock) on /cache/.fclones.lock additionally guards the cache against a scan from a different container or a manual docker run sharing the same /cache volume, which skips rather than corrupting the shared fclones cache. Ofelia's no-overlap is still recommended to avoid queuing redundant triggers. Note the exec must run as the container's own user (the trigger socket is owner-only); a mismatched exec user fails loudly at connect.

Run once

Set FCLONES_INTERVAL=0 (or 0s). The container runs exactly one scan and dedup action, then exits — non-zero if the scan failed, timed out, was interrupted (SIGTERM/SIGINT) before it finished, or was skipped because another process held the /cache scan lock. This suits a batch or one-shot context (a Kubernetes Job, a CI step, or a manual docker run --rm) where an external system, not the container, decides when to run again: a run that did not complete a scan — whether cut short or skipped on lock contention — surfaces as a failed run (logged with outcome=skipped for the lock case) so the orchestrator retries rather than recording success. In the long-running modes a SIGTERM is a clean shutdown and a lock conflict a benign no-op, both exiting 0; only run-once treats a scan that never ran as a failure, because there the exit code is the job result.

Configuration reference

Environment variables

Variable	Description	Default	Required
FCLONES_INTERVAL	Built-in scan interval as a Go duration (e.g. 1h, 30m, 12h). The first scan runs at startup; subsequent scans fire every interval thereafter. Set to off (or disabled) to idle and trigger scans externally, or to 0 (or 0s) to run a single scan and exit — see Scheduling modes. Falls back to 3h on an unset, unparseable, or negative value.	3h	No
FCLONES_SCAN_PATHS	Paths inside the container to scan for duplicates. Must match the volume mounts. Multiple paths can be space-separated (e.g. /media /photos), each requiring a corresponding volume mount.	/scandir	No
FCLONES_ARGS	Extra arguments passed to the fclones group scan phase. The wrapper owns --cache and the report format (-f json); passing --cache, -f, or --format here is rejected at startup.	(none)	No
FCLONES_ACTION	Dedup action after scan — group (report only), link (hardlink), remove (delete), or dedupe (reflink/copy-on-write)	group	No
FCLONES_ACTION_ARGS	Extra arguments for the dedup action phase	(none)	No
FCLONES_ALLOW_UNSAFE	Set to true to allow dangerous flags (--command, --transform, --in-place, --no-copy)	false	No
FCLONES_SCAN_TIMEOUT	Per-phase timeout (Go duration) applied to each fclones scan and action phase. A phase exceeding it is terminated and the run is marked unhealthy. Set to 0 for no timeout (unbounded — the phase runs until it finishes or the container stops). Raise for large filesystems whose initial scan can exceed 12h.	12h	No
FCLONES_LOG_LEVEL	slog level: debug, info, warn/warning, or error. Unrecognized values fall back to info.	info	No

Volumes

Mount	Description
/scandir	Directory to scan for duplicate files. Must match the paths in FCLONES_SCAN_PATHS (space-separated for multiple mounts). The group action needs read access only; link/remove/dedupe modify files here, so /scandir must be writable by the user: UID (not a :ro mount) for those actions.
/cache	fclones cache and state directory. Must be writable by the UID set in user: (the example uses 1000:1000). The wrapper write-probes /cache at startup; if it is read-only or owned by another UID the container logs cache directory verification failed uid=<n> and exits (crash-looping under restart: unless-stopped).

Alerting

docker-fclones-scheduler has no metrics endpoint; its operational state is in its logs. Ship the container's logs to Loki (Grafana Alloy's Docker log discovery does this with no configuration) and evaluate these with Loki's ruler; firing alerts deliver through your Alertmanager exactly like Prometheus metric alerts.

groups:
  - name: docker-fclones-scheduler
    rules:
      - alert: FclonesLinkEstablished
        expr: |
          sum by (files_deduped, reclaimed_human) (
            count_over_time(
              {container="fclones"} |= "action complete" | logfmt | action="link" | files_deduped > 0 [15m]
            )
          ) > 0
        for: 0m
        labels:
          severity: info
        annotations:
          summary: "fclones linked {{ $labels.files_deduped }} duplicate files (reclaimed {{ $labels.reclaimed_human }})"
          description: >
            fclones established hardlinks for {{ $labels.files_deduped }} duplicate
            files, reclaiming {{ $labels.reclaimed_human }}. The individual linked
            paths are in the same run's `duplicate file` log lines (capped at 500
            pairs / 64 KB), viewable in Loki: {container="fclones"} |= "duplicate
            file" (filter by the run's scan_id). Success notification, no action
            required.
      - alert: FclonesFormatDrift
        expr: |
          sum(count_over_time({container="fclones"} |= "possible fclones format drift" [2h])) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "fclones output-format drift detected (duplicate stats may be unreliable)"
          description: >
            The wrapper logged "possible fclones format drift": a group-count
            mismatch, an unreadable group total, or an unrecognized action
            summary. Its report parsers no longer recognize fclones' output
            format, so the duplicate and reclaim stats in its log lines may be
            reported as zero even when duplicates exist or work happened. Any
            configured dedup action still runs (the wrapper falls back to
            piping fclones' own report to it) and the run still exits 0, so a
            job-failure or container-restart alert would not catch it; duplicate
            reporting is silently degraded. Check the fclones version and the
            wrapper's report parsers.

Thresholds and the severity labels are starting points. Adjust the container selector (or job / service, depending on your log collector) to your deployment; if you run remove or dedupe instead of link, change action="link" in the first rule to match your FCLONES_ACTION. Route by whatever labels your Alertmanager uses.

These rules work in both scheduling modes: every run executes in the daemon (PID 1), so its logs always land under the app's own container name. Under FCLONES_INTERVAL=off the exec'd scan client emits only its own lifecycle lines (queued / started / result) to the trigger's log — an Ofelia job log, for example — and exits with the run's result, so you can additionally alert on your scheduler's own job outcome for belt-and-braces failure coverage.

Healthcheck

The built-in healthcheck (/app/wrapper health) checks for a marker file the daemon maintains after each run — the daemon is the marker's single writer, so a triggered scan client never races it. The container becomes unhealthy when fclones exits non-zero (e.g. scan path missing, permission denied, corrupted cache), the action phase fails (e.g. hardlink across filesystems), the report cannot be decoded, or startup verification fails (e.g. /cache is full or read-only). It recovers automatically on the next successful scan — no restart required. In built-in mode the container begins unhealthy, transitions to healthy after the first successful scan completes, and additionally arms a freshness deadline of 2 x FCLONES_INTERVAL + 2 x FCLONES_SCAN_TIMEOUT: a marker that old means the interval loop is wedged, and the probe reports unhealthy so Docker restarts the container (disabled automatically when FCLONES_SCAN_TIMEOUT=0, since the worst-case run duration is then unbounded). In external mode the container starts healthy (idle, nothing has failed), each triggered run updates the marker, and no deadline applies — an idle container between sparse triggers is healthy.

The image bakes a 15s start_period, which suits small libraries and external mode. If your first built-in scan takes minutes, raise start_period in your own compose file so the container isn't reported unhealthy — and doesn't fire spurious alerts — during that initial scan:

services:
  fclones:
    healthcheck:
      start_period: 10m # size to your library's first-scan duration

Security

No vulnerabilities found. All scans clean.

Tool	Result
govulncheck	No vulnerabilities in call graph
golangci-lint (gosec, gocritic)	0 issues
trivy	0 vulnerabilities
grype	0 vulnerabilities
gitleaks	No secrets detected
semgrep	2 info (false positives)
hadolint	DL3008 in builder stage (discarded)

No network listener, no HTTP server, no exposed ports — the trigger socket is a local unix socket (/tmp/fclones-wrapper.sock), owner-only (0600), reachable only from inside the container. The FCLONES_ACTION env var is validated against an allowlist and dangerous flags (--command, --transform, --in-place, --no-copy) are blocked by default to prevent command injection via env vars. Set FCLONES_ALLOW_UNSAFE=true to disable the guardrails if you need --transform for content-aware deduplication. Runs as nonroot on a distroless base image with no shell.

Details for advanced users: Arguments are passed to exec.Command as explicit arg lists (no shell expansion). Temp files use os.CreateTemp with unpredictable names. Captured subprocess streams are size-capped, and the scan report is decoded with a strict streaming JSON decoder (bounded memory regardless of report size; a malformed report fails the run). Runs are serialized by the daemon's single executor queue; an advisory file lock (flock on /cache/.fclones.lock) additionally guards the shared cache against scans from other containers or manual docker run invocations. Wrapper-owned fclones flags (--cache, -f/--format) are rejected in FCLONES_ARGS at startup so user args cannot break the report contract.

Dependencies

Dependency	Source
rust	Rust
golang	Go
Distroless static nonroot	Distroless
fclones	GitHub

Updated automatically via Renovate; base images are pinned by digest and the upstream fclones artifact is integrity-pinned (tarball sha256 on amd64, commit on arm64). Builds carry signed SBOMs and provenance attestations verifiable with gh attestation verify.

Credits

This project packages fclones into a container image. All credit for the core functionality goes to the upstream maintainers.

Contributing

Issues and pull requests are welcome. Please open an issue first for larger changes so the approach can be discussed before implementation.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude Opus and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

This project is licensed under the GNU General Public License v3.0.

Documentation

Overview

Package main is the fclones-wrapper binary: an interval scheduler that runs fclones against mounted directories to find, hardlink, or remove duplicate files.