rclonestore

README

rclonestore

rclonestore implements storage's Blobs primitive — content-addressed immutable byte objects — by exec'ing the external rclone binary (rcat/cat/deletefile/lsf), giving looprig's workspace store a cloud-agnostic backend across any of rclone's many local and cloud remotes without ever linking librclone (whose dependency tree would defeat the point of the extraction). It drives rclone via an argv list only — never a shell string — with -- inserted before positional path arguments so no key can be mistaken for a flag, every call bounded by a context.Context, and rclone's config (which may embed remote credentials) referenced by path only: never parsed, copied, logged, or placed in an error. It depends only on the Go standard library and github.com/looprig/storage.

Usage

s, err := rclonestore.New(rclonestore.Options{
	Remote:           "myremote",       // a named rclone remote, OR a :backend: connection string
	Prefix:           "workspaces/v1",  // optional path prefix under the remote
	PersistencePaths: []string{"/data/workspaces/v1"}, // only when a named remote persists locally
	Timeout:          30 * time.Second, // optional per-call bound
})
if err != nil {
    return err // *OptionsError | *BinaryError | *ProbeError
}
// s is a storage.Blobs: Put / Get / Delete / List.

New fails fast (fail-secure) if the configuration is invalid, the binary is missing, or the remote is unreachable — never deferring those failures to first use. Store holds no persistent connection, so its Close is a documented no-op.

Options

Field	Meaning
Remote	Required. A named rclone remote or a :backend: connection string (see below).
Prefix	Optional path fragment under the remote. Must be a safe relative path: no leading/trailing /, no empty, . or .. segment.
PersistencePaths	Optional local filesystem roots used by the backend. Inline :local: roots are discovered automatically; named remotes are never introspected. Entries are exact effective roots, so Prefix is not appended.
Binary	rclone executable; default "rclone", resolved via exec.LookPath.
ConfigPath	Optional --config path. Referenced by path only — never opened, parsed, copied, or logged (it may hold remote secrets).
Timeout	Optional bound on every rclone invocation (including the startup probe). Zero means each call is bounded only by the caller's context.

The two remote forms

Remote is validated as exactly one of two forms, and the object path is built accordingly (a hardcoded colon would break the connection-string form):

• Named remote — a config alias matching ^[A-Za-z0-9_-]+$ (e.g. myremote). The object path joins with a colon: myremote: + prefix/key → myremote:prefix/key.
• Connection string — an inline spec beginning with : and a lowercase alphanumeric backend (e.g. :local:/data, :s3,provider=Minio:/bucket). The backend spec already ends with a colon, so the path is appended with a slash instead: :local:/data + prefix/key → :local:/data/prefix/key.

The parser finds the spec-ending unquoted colon. Parameter values containing colons or commas must use rclone's single- or double-quoted form; doubling the active quote escapes it. For example, :local,token='https://user:pass@host':/data still has /data as its local path. Unterminated quoted values are rejected without echoing the credential-bearing remote. Colons after the spec delimiter belong to the path. An empty Prefix collapses cleanly.

Local persistence paths

Store implements storage's optional PathReporter capability. StoragePaths returns the canonical local roots that hold blobs, allowing workspace owners to reject unsafe overlap with their own directories.

Inline :local: remotes are detected without invoking rclone or reading configuration. Their effective root includes Prefix; for example Remote: ":local:/data" with Prefix: "workspaces/v1" reports /data/workspaces/v1. Other inline backends and ordinary named remotes report no automatic path.

A named remote can itself be configured as a local backend, but rclonestore deliberately does not inspect its config because it may contain credentials. In that case the caller must provide the exact effective root through PersistencePaths. Explicit paths are unioned with any automatically detected local root, canonicalized through the nearest existing ancestor, sorted, and deduplicated. A directory that does not exist yet is supported; a broken symlink or path that resolves to a regular file returns *PersistencePathError. Both the option slice and every returned slice are defensively copied.

Startup probe

New probes reachability with rclone lsf --max-depth 0 -- <remote-root>, bounded by Timeout. A benign not-found on the root (a fresh store whose root directory does not exist yet — first Put creates it) is treated as reachable-but-empty and succeeds, matching List's semantics. Any other failure (unknown remote, auth, network) is a *ProbeError wrapping the credential-safe *RcloneError.

Security posture

• argv exec only — never a shell string. exec.CommandContext with every argument as a discrete element; no sh -c, no interpolation of any external value.
• -- before positionals, inserted exactly once, so no key or remote path starting with - can be parsed as a flag.
• Every call is context.Context-bounded; on timeout/cancel the subprocess is killed.
• No secrets in errors or logs. rclone config may embed credentials — it is referenced by path only. Errors carry only the rclone subcommand, safe subflags, the exit code, and a bounded (~4 KiB) tail of stderr; never the config path, the remote, or any positional. OptionsError names the offending field and rule but never the offending value (a connection-string Remote can embed secrets), and ProbeError does not carry the remote.
• Never links librclone / cgo — rclone is driven as a subprocess only.

Error types

All errors are typed; classify with errors.As.

• *OptionsError — invalid Remote/Prefix/Timeout (from New, before any exec).
• *PersistencePathError — a declared or automatically derived local persistence root could not be canonicalized (wraps an underlying filesystem cause when applicable).
• *BinaryError — the rclone binary could not be resolved on PATH (wraps the exec.LookPath cause).
• *ProbeError — the startup reachability probe failed (wraps the underlying *RcloneError).
• *RcloneError — a failed rclone invocation (non-zero exit, start failure, or ctx kill).
• *PutSourceError — reading the caller's Put reader failed.
• storage's *BlobNotFoundError, *BlobConflictError, *InvalidNameError per the Blobs contract.

Testing

GOWORK=off make check                          # gofmt + vet + gosec + race unit tests
GOWORK=off go test -tags integration -race ./... # storage Blobs conformance vs. real rclone

The unit tests drive a generated fake rclone (per-test #!/bin/sh script) and never touch the network. The conformance suite (//go:build integration) runs storage's storetest.TestBlobs against a real rclone using the LOCAL backend (:local:<temp dir>) — no cloud credentials — and skips (never fails) when rclone is not on PATH. It is the harness that validates the not-found exit-code classification (3/4 plus the stderr marker), the remote-form-aware object path, and the lsf-on-a-file existence probe against real rclone.

Every Go command runs with GOWORK=off so the parent go.work at ~/code never captures this module.

Documentation

Overview

Package rclonestore implements storage.Blobs by driving the external rclone binary as a context-bounded subprocess (argv exec — never a shell string, never librclone/cgo).

Index

• type BinaryError
  • func (e *BinaryError) Error() string
  • func (e *BinaryError) Unwrap() error
• type Options
• type OptionsError
  • func (e *OptionsError) Error() string
• type PersistencePathError
  • func (e *PersistencePathError) Error() string
  • func (e *PersistencePathError) Unwrap() error
• type ProbeError
  • func (e *ProbeError) Error() string
  • func (e *ProbeError) Unwrap() error
• type PutSourceError
  • func (e *PutSourceError) Error() string
  • func (e *PutSourceError) Unwrap() error
• type RcloneError
  • func (e *RcloneError) Error() string
  • func (e *RcloneError) Unwrap() error
• type Store
  • func New(opts Options) (*Store, error)
  • func (s *Store) Close() error
  • func (b Store) Delete(ctx context.Context, key string) error
  • func (b Store) Get(ctx context.Context, key string) (io.ReadCloser, error)
  • func (b Store) List(ctx context.Context, prefix string) ([]string, error)
  • func (b Store) Put(ctx context.Context, key string, r io.Reader) error
  • func (b Store) StoragePaths() []string

Types

type BinaryError

type BinaryError struct {
	Binary string
	// contains filtered or unexported fields
}

BinaryError reports that the rclone binary could not be resolved on PATH. The binary name/path is operator-supplied configuration (no credential), so it is safe to name; the underlying exec.LookPath error is wrapped for errors.Is/As.

func (*BinaryError) Error

func (e *BinaryError) Error() string

func (*BinaryError) Unwrap

func (e *BinaryError) Unwrap() error

type Options

type Options struct {
	// Remote is required: a named rclone remote (e.g. "myremote") OR a ":backend:"
	// connection string (e.g. ":local:/data", ":s3,provider=Minio:/bucket").
	Remote string
	// Prefix is an optional path fragment under the remote. If set it must be a
	// safe relative path: no leading/trailing '/', no empty, "." or ".." segment.
	Prefix string
	// Binary is the rclone executable; default "rclone", resolved via exec.LookPath.
	Binary string
	// ConfigPath is an optional --config path. It is referenced by path only —
	// never opened, parsed, copied, or logged here (it may hold remote secrets).
	ConfigPath string
	// Timeout optionally bounds every rclone invocation (including the startup
	// probe). Zero means each call is bounded only by the caller's context.
	Timeout time.Duration
	// PersistencePaths declares local filesystem roots used by the backend. Inline
	// :local: remotes are discovered automatically, so this is primarily for named
	// remotes, which are never introspected. Entries are already-effective roots;
	// Prefix is not appended to them.
	PersistencePaths []string
}

Options configures a rclonestore Store. Only Remote is required. No field is ever logged or placed in an error verbatim: a connection-string Remote or a ConfigPath can embed remote credentials.

type OptionsError

type OptionsError struct {
	Field string
	Rule  string
}

OptionsError reports an invalid field in Options. It names the Field and the Rule violated but never the offending value: Remote (a connection string) or a ConfigPath can embed credentials, so no option value is ever surfaced.

func (*OptionsError) Error

func (e *OptionsError) Error() string

type PersistencePathError

type PersistencePathError struct {
	Path string
	Rule string
	// contains filtered or unexported fields
}

PersistencePathError reports an invalid local filesystem root declared by the caller or derived from an inline local remote. Path contains only the local filesystem portion, never a named remote or inline backend parameters.

func (*PersistencePathError) Error

func (e *PersistencePathError) Error() string

func (*PersistencePathError) Unwrap

func (e *PersistencePathError) Unwrap() error

type ProbeError

type ProbeError struct {
	// contains filtered or unexported fields
}

ProbeError reports that the startup reachability probe failed: rclone could not list the remote root and the failure was not a benign not-found. It wraps the underlying *RcloneError, which is credential-safe by construction (it excludes the config path, the remote, and every positional). The remote is deliberately NOT carried on ProbeError — a connection-string remote can embed secrets.

func (*ProbeError) Error

func (e *ProbeError) Error() string

func (*ProbeError) Unwrap

func (e *ProbeError) Unwrap() error

type PutSourceError

type PutSourceError struct {
	Key string
	// contains filtered or unexported fields
}

PutSourceError reports that reading the caller-supplied Put reader failed before the blob could be compared against an existing object or streamed to rclone. It carries the storage key (safe to log — it is a validated canonical name) and wraps the underlying read error.

func (*PutSourceError) Error

func (e *PutSourceError) Error() string

func (*PutSourceError) Unwrap

func (e *PutSourceError) Unwrap() error

type RcloneError

type RcloneError struct {
	Subcommand string
	Args       []string
	ExitCode   int
	Stderr     string
	// contains filtered or unexported fields
}

RcloneError reports a failed rclone invocation: a non-zero exit, a start failure, or a subprocess killed by its context deadline/cancellation. Callers classify with errors.As.

It deliberately carries only information that cannot embed a credential:

• Subcommand — the rclone subcommand (e.g. "rcat", "cat", "lsf", "deletefile").
• Args — the subcommand's own flags (the subflags). It NEVER contains the positional path arguments (which embed the remote name and the storage key) nor the global --config path (which points at a file that may hold remote credentials). The caller passes only non-secret subflags.
• ExitCode — the process exit status; -1 for a start failure or a signal kill.
• Stderr — a bounded tail (~4 KiB) of the process's stderr, surfaced as-is from rclone (whose diagnostics do not echo config secrets) and bounded so it cannot balloon an error or log line.

The config path, the remote, and every positional are excluded by construction, so an RcloneError is always safe to log.

func (*RcloneError) Error

func (e *RcloneError) Error() string

Error renders the subcommand, the exit status, and a sanitized (quoted) stderr tail. The tail is strconv.Quote'd so newlines or control bytes emitted by the subprocess cannot inject into a log line. No credential-bearing value is ever included.

func (*RcloneError) Unwrap

func (e *RcloneError) Unwrap() error

Unwrap returns the underlying cause so callers can classify with errors.Is / errors.As: the *exec.ExitError (or start error) on a genuine failure, or the context error (context.DeadlineExceeded / context.Canceled) when the call was killed by its deadline or cancellation.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a storage.Blobs backed by the external rclone binary. It embeds the unexported blobStore, so a Store IS-A storage.Blobs (Put/Get/Delete/List are promoted). Construct one with New.

func New

func New(opts Options) (*Store, error)

New validates opts, resolves the rclone binary, and probes that the remote is reachable before returning a Store — failing loudly (fail-secure) at construction rather than at first use:

1. Remote is validated as a named remote or a connection string (*OptionsError).
2. Prefix, if set, is validated as a safe path fragment (*OptionsError).
3. Binary (default "rclone") is resolved with exec.LookPath (*BinaryError).
4. The remote root is probed with "rclone lsf --max-depth 0 -- <root>", bounded by opts.Timeout. A not-found root is treated as reachable-but-empty (a fresh store creates it on first Put); any other failure is a *ProbeError.

func (*Store) Close

func (s *Store) Close() error

Close releases resources held by the Store. rclone is driven per-call as a short-lived subprocess and holds no persistent connection, so Close is a documented no-op that always returns nil; it exists only so callers can treat a Store uniformly with stores that do hold resources.

func (Store) Delete

func (b Store) Delete(ctx context.Context, key string) error

Delete removes the object at key via "rclone deletefile". Deleting an absent object is a success (idempotent): a not-found from rclone is classified and mapped to nil; every other failure propagates.

func (Store) Get

func (b Store) Get(ctx context.Context, key string) (io.ReadCloser, error)

Get streams the object at key back to the caller. It runs "rclone cat" to completion into an in-memory buffer and returns an independent io.ReadCloser over those bytes. Buffering (rather than piping rclone's stdout through) is what lets Get satisfy the contract's synchronous not-found: storetest expects Get itself — not a later Read — to return *storage.BlobNotFoundError, which is only knowable once rclone has exited. A missing object is classified from rclone's exit code/stderr and mapped to *storage.BlobNotFoundError.

func (Store) List

func (b Store) List(ctx context.Context, prefix string) ([]string, error)

List returns the storage keys of every object under the store's prefix whose key begins with the caller's prefix, lexicographically ascending and duplicate-free. It runs "rclone lsf --files-only -R" rooted at "<remote>:<prefix>": the emitted paths are relative to that root, so they ARE the storage keys. The caller's prefix is applied locally (it need not fall on a directory boundary) and is NOT name-validated. The result is sorted locally — rclone's ordering is not trusted. An empty store surfaces as a not-found on the root directory, which maps to an empty listing rather than an error.

func (Store) Put

func (b Store) Put(ctx context.Context, key string, r io.Reader) error

Put honors storage's content-addressed conflict contract. It first probes for an existing object:

• ABSENT (the common content-addressed case): stream r straight to "rclone rcat" with no buffering — the blob never lands in memory.
• PRESENT: read r fully, "rclone cat" the existing object, and compare bytes. Byte-identical → success/no-op (the object is NOT re-uploaded). Different → *storage.BlobConflictError with the original left untouched (no upload).

The present branch buffers both the incoming reader (io.ReadAll) and the existing object (into memory) to compare them. This is deliberately the rare path: keys are content-addressed, so a re-Put of DIFFERENT bytes under the same key is pathological, and a re-Put of IDENTICAL bytes is a cheap idempotent no-op. The probe→write sequence is a TOCTOU only under concurrent writers to the same key; the workspace store holds a single-writer lease over the key space, so no concurrent writer exists by construction.

func (Store) StoragePaths

func (b Store) StoragePaths() []string

StoragePaths returns the canonical local filesystem roots used by this blob provider. Remote backends return nil. The result is a defensive copy so callers cannot mutate the provider's construction-time view.