drivesync

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

README

drivesync

drivesync shares an encrypted folder through ordinary directories on macOS and Linux. Create a folder with a name and a key, then attach it on each node. Reads are normal local files. Replicas sync both ways. There is no FUSE layer. Pure-Go builds are supported.

The authority stores ciphertext and control-plane records. It does not receive the folder key, file paths, file contents, or file metadata. SECURITY.md describes what the server can see, what a key holder can do, and the local-disk guarantees.

Quickstart

SQLite metadata is one authority process per database file. Postgres metadata, opened with OpenPostgresMetaStore, can be shared by several authority processes. The pool needs at least three connections: one stays on LISTEN and two remain for transactions. Each process listens for the others' commits. An upload records a lease renewed while bytes flow; recovery retires only a ticket whose lease has expired. A garbage-collection pass claims a lease row (a holder id and an expiry, renewed during the pass) or skips when another process holds an unexpired lease. A frozen holder stops blocking once that lease expires. auth authenticates each HTTP request and returns a Principal{Tenant, Subject}. plans implements QuotaPolicy.

meta, err := drivesync.OpenSQLiteMetaStore("authority.sqlite")
if err != nil { return err }
defer meta.Close()
blobs, err := drivesync.OpenDirectoryBlobStore("blobs")
if err != nil { return err }
defer blobs.Close()

srv := drivesync.NewServer(meta, blobs, drivesync.ServerOptions{Quotas: plans})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- srv.Run(ctx) }()
defer func() { cancel(); <-done }()
mux.Handle("/drives/", http.StripPrefix("/drives", srv.Handler(auth)))

Run recovers interrupted uploads, expires reservations, collects garbage, and compacts tombstones. Cancel it and wait before closing the stores.

On a client node, or with srv.Client(principal) inside the authority process:

c := drivesync.NewHTTPClient("https://example.com/drives", headers)
key := drivesync.NewFolderKey()
f, err := c.CreateFolder(ctx, drivesync.FolderSpec{
    Name: "workspace", Description: "Files shared across nodes",
    Limits: drivesync.Limits{
        MaxFileBytes: 64 << 20, MaxTotalBytes: 1 << 30,
        MaxFiles: 10000, MaxRows: 50000,
    },
}, key)
if err != nil { return err }
if err := c.Grant(ctx, f.ID, bob, drivesync.Writer); err != nil { return err }
r, err := drivesync.Attach(ctx, c, f.ID, key, "./workspace", drivesync.Options{})
if err != nil { return err }
defer r.Close()
status := r.Status()

Preserve the key and distribute it to each attachment. Creation computes the key check locally. Folder ids are random 128-bit ids. Names are unique per primary owner. Grant, Revoke, SetLimits, ListFolders, GetFolder, and DeleteFolder manage the folder.

Attach's context bounds setup. The replica runs until Close. Sync forces one pass. Status reports pending bytes, conflicts, rejections, quarantine, and errors. Retry retries rejected or quarantined transfers and acknowledges an intentional mass removal.

For a new node that needs only part of a drive, attach with Options{OnDemand: true}. Sync lists remote files as empty regular files with permission bits 0000. These placeholders contain no file data. Fetch content before opening it:

entries, err := r.Entries("projects/**") // Local listing, including plaintext sizes.
result, err := r.Fetch(ctx, "projects/current", "notes/*.md")
// Check result.Failed; result.Unmatched reports patterns matching nothing.

Patterns select exact paths, whole folders, or globs (*, ?, [], **). No patterns, or the root pattern ".", selects everything. A * stays within one path component; ** can span folders. Fetch authenticates bytes before replacing each placeholder and reports fetched paths, bytes, and files already present. Downloaded files stay downloaded and receive subsequent remote updates. Status.Placeholders and PlaceholderBytes describe what remains unfetched. Changing a placeholder's permissions requests its content on the next sync. Moving a placeholder fetches its content before committing the move; copying an empty placeholder is skipped. Reattaching with OnDemand: false fetches all remaining placeholders. This API does not intercept ordinary file opens; applications that need transparent reads can call Fetch from their mount layer.

On-demand replicas expand folders breadth first. MaxFolderPlaceholders (default 1,000 direct entries) and MaxPlaceholders (default 20,000 entries in total) bound local materialization. Larger folders collapse to an existing directory: their children remain in the local index, with no filesystem entry until fetched. Entries lists these remote-only paths and Fetch creates only the requested contents and their parents. Removing a collapsed directory locally deletes its remote subtree, subject to the usual mass-removal guard. New local writes at unseen remote paths are preserved as conflict copies.

Set both GuideName and Guide to put a local guide at the root and in each collapsed folder. It contains your text and an inventory of unfetched entries, up to 1,000 lines, with folder file counts and plaintext sizes. Guides refresh after sync and fetch, and disappear when their folder expands or on-demand mode is disabled. GuideName is reserved at every level, including when on-demand mode is disabled: remote entries with that name are ignored locally. Guides never upload, count toward limits, or produce conflict copies.

New placeholders publish in batches of up to 1,000 entries, with one directory barrier per affected parent before the index commits. An interrupted batch can leave empty placeholders without index entries; the next sync adopts those whose paths match authenticated incoming rows.

Replica state stores directory prefixes once and uses compact binary entries with raw hashes and blob ids. Existing local indexes migrate automatically on attach, in one transaction. Authority rows, encrypted metadata and the wire format remain compatible with earlier stations.

Options.Downloads bounds parallel transfers (zero selects eight, at most 256). Transfers stage ciphertext in bounded batches, with at most 512 MiB queued on disk. Larger individual files stream directly through authentication. Publication remains serial, with the usual conflict preservation and fsync guarantees.

Authorities with a BlobURLStore can serve fresh replicas a compressed packed file list with one direct download, followed by Changes to catch up. Set ServerOptions.SnapshotRows to the minimum live row count (zero selects 1,000; negative disables snapshots). Snapshots contain encrypted metadata, are derived data outside folder quota, and are removed with the folder. The blob store must accept and sign the reserved blob key snapshot, alongside normal hex blob IDs. They rebuild when missing or when more than the larger of 1,000 rows and 10% of live rows changed. Older authorities and failed snapshots use Changes.

A small CLI exercises the same API with one local principal. Set DRIVESYNC_TOKEN. Pass the key in DRIVESYNC_KEY or -key-file, never in argv.

go run ./cmd/drivesync serve -data ./drivesync-data
go run ./cmd/drivesync create -name shared
go run ./cmd/drivesync attach -folder ID -key-file ./folder.key -dir ./shared -state ./replica-state
go run ./cmd/drivesync status -state ./replica-state

Limits and quotas

Limits bounds sealed bytes, live entries, and retained rows. MaxFileBytes is the sealed size, including encryption overhead. MaxFiles counts live entries, including directory markers. MaxRows bounds retained rows and garbage. Zero MaxRows selects 16 times MaxFiles, capped at one million. Other zero limits mean unlimited.

Sharing, including a grant in the same tenant, requires a positive byte cap, row cap, and per-file cap. That capacity is charged to the primary owner until the folder is deleted and cleaned up. Reservations are atomic, and commit checks the stored size. Rejected files stay on the local disk with a reason and a retry backoff.

Folder reports id, name, description, owner, role, limits, and usage. Usage is sealed bytes, live entries, and reserved bytes.

Platform support

macOS and Linux are supported, including CGO_ENABLED=0. macOS watches the tree with FSEvents through purego. Linux uses inotify. Both debounce for 150 ms and rescan periodically. kqueue is not used. Other operating systems are not supported.

How it works

A replica scans its directory, encrypts new bytes, and uploads ciphertext. Peers download into a staging file, authenticate it, then publish with fsync and rename. Renames commit as one create/delete pair. A losing local write is kept as a conflict copy. Matching authenticated content is adopted in place, so an existing tree can be attached without uploading those bytes again.

Path ids, metadata, and content use separate keys. Chunks are bound to the folder, blob, path, index, and a final flag. Case collisions keep both contents under distinct local names. Only the owner execute bit syncs. Group and other permission bits stay local.

Default state lives in os.UserCacheDir()/drivesync/<folder>/<root-hash> with private permissions. It holds plaintext paths, so keep it outside other shared folders. Options.StateDir must be dedicated to that attachment. A nonempty directory without the matching state marker is refused. .drivesync* names are reserved. .drivesyncignore accepts globs, root-anchored / patterns, **, comments, and directory patterns, matched under case-folded NFC. Negation and escapes are reported as unsupported.

.git/ is ignored by default. Sync the working tree, and let git carry history.

The included blob stores are an in-memory store and a durable local directory. BlobStore.Put must publish only after a clean EOF. An embedder can add an object-store backend without taking an SDK dependency. Metadata is SQLite or Postgres. OpenPostgresMetaStore needs a pool of at least three connections. QuotaPolicy is the other extension point.

A blob store can also implement BlobURLStore.DownloadURL to return a URL for one sealed blob. The authority authorizes each blob before issuing a URL with a five-minute lifetime. Replicas download directly using an HTTP client with no authority headers, falling back to the authority when signing or direct delivery fails. Existing authorities without URL support continue to work. Treat signed URLs as secrets; revocation cannot recall URLs already issued before they expire.

Fresh databases are required. This version does not migrate older schemas, and it does not provide historical recovery, key rotation, or chunk deduplication.

License

The code is licensed under Apache-2.0. Copyright 2026 Parham Negahdar. See LICENSE.

Unicode path handling is checked against the Unicode Consortium's NormalizationTest data, which remains under the Unicode License. See UNICODE_LICENSE.txt. The corpus is internal/engine/testdata/NormalizationTest.txt.gz.

Documentation

Overview

Package drivesync shares encrypted folders through ordinary local directories. Create a folder with a name and a key, then attach it wherever files are needed.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrDenied    = engine.ErrDenied
	ErrBusy      = engine.ErrBusy
	ErrExpired   = engine.ErrExpired
	ErrQuota     = engine.ErrQuota
	ErrConflict  = engine.ErrConflict
	ErrInvalid   = engine.ErrInvalid
	ErrKey       = engine.ErrKey
	ErrIntegrity = engine.ErrIntegrity
	ErrWaitLimit = engine.ErrWaitLimit
)

Functions

This section is empty.

Types

type Authenticator 

type Authenticator func(*http.Request) (Principal, error)

type BlobStore 

type BlobStore interface {
	Put(context.Context, string, string, io.Reader) (int64, error)
	Open(context.Context, string, string) (io.ReadCloser, error)
	Size(context.Context, string, string) (int64, error)
	Delete(context.Context, string, string) error
}

BlobStore holds immutable sealed blobs. Keys are authority-generated IDs. Put streams bytes, Size reports actual stored bytes, and Delete is idempotent (os.ErrNotExist is also accepted). Implementations must honor cancellation. Put must publish only after clean EOF; a reader error must discard staging. Besides hexadecimal blob IDs, stores must accept the reserved key "snapshot" for a folder's replaceable packed file list. The authority deletes it before replacement; it is derived data and is excluded from quota.

type BlobURLStore added in v0.4.0 

type BlobURLStore interface {
	DownloadURL(ctx context.Context, folder, id string, ttl time.Duration) (string, error)
}

BlobURLStore is an optional BlobStore extension. When the authority's store implements it, an authorized download answers with a URL valid for ttl that the replica fetches directly, so blob bytes skip the authority. The URL is a bearer capability for one sealed blob: it must name only that blob, expire, and never appear in errors or logs. Implementations also sign the reserved key "snapshot" for packed file lists.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client manages folders through HTTP or an authenticated in-process authority. Its replication and encryption protocol is private to the library.

func NewHTTPClient 

func NewHTTPClient(url string, headers http.Header) *Client

NewHTTPClient connects to an authority. Use HTTPS outside local tests.

func (*Client) CreateFolder 

func (c *Client) CreateFolder(ctx context.Context, spec FolderSpec, key FolderKey) (Folder, error)

CreateFolder computes the key check locally; the key is never sent to the authority. Keep it securely and supply the same key to every attachment.

func (*Client) DeleteFolder 

func (c *Client) DeleteFolder(ctx context.Context, id string) error

func (*Client) GetFolder 

func (c *Client) GetFolder(ctx context.Context, id string) (Folder, error)

func (*Client) Grant 

func (c *Client) Grant(ctx context.Context, id string, p Principal, role Role) error

func (*Client) ListFolders 

func (c *Client) ListFolders(ctx context.Context) ([]Folder, error)

func (*Client) Revoke 

func (c *Client) Revoke(ctx context.Context, id string, p Principal) error

func (*Client) SetLimits 

func (c *Client) SetLimits(ctx context.Context, id string, limits Limits) error

type DirectoryBlobStore 

type DirectoryBlobStore struct {
	// contains filtered or unexported fields
}

DirectoryBlobStore streams sealed bytes to a durable local directory.

func OpenDirectoryBlobStore 

func OpenDirectoryBlobStore(path string) (*DirectoryBlobStore, error)

func (*DirectoryBlobStore) Close 

func (b *DirectoryBlobStore) Close() error

func (*DirectoryBlobStore) Delete 

func (b *DirectoryBlobStore) Delete(ctx context.Context, folder, id string) error

func (*DirectoryBlobStore) Open 

func (b *DirectoryBlobStore) Open(ctx context.Context, folder, id string) (io.ReadCloser, error)

func (*DirectoryBlobStore) Put 

func (b *DirectoryBlobStore) Put(ctx context.Context, folder, id string, r io.Reader) (int64, error)

func (*DirectoryBlobStore) Size 

func (b *DirectoryBlobStore) Size(ctx context.Context, folder, id string) (int64, error)

type Entry added in v0.4.0 

type Entry struct {
	Path        string
	Size        int64
	Directory   bool
	Placeholder bool
}

Entry is one path a replica tracks.

type FetchResult added in v0.4.0 

type FetchResult struct {
	// Fetched lists the placeholders this call downloaded; Bytes is their plaintext size.
	Fetched []string
	Bytes   int64
	// Present counts matched files that were already downloaded.
	Present int
	// Unmatched lists patterns that matched no tracked file.
	Unmatched []string
	Failed    []Rejection
}

FetchResult reports one Fetch.

type Folder 

type Folder struct {
	ID                string
	Name, Description string
	Owner             Principal
	Role              Role
	Limits            Limits
	Usage             Usage
}

type FolderKey 

type FolderKey [32]byte

func NewFolderKey 

func NewFolderKey() FolderKey

func ParseFolderKey 

func ParseFolderKey(s string) (FolderKey, error)

ParseFolderKey imports a 64-character hex encoding of a random 256-bit key. Passphrases and hashes of passphrases are not suitable folder keys.

func (FolderKey) String added in v0.3.0 

func (k FolderKey) String() string

String is the 64-character hex form ParseFolderKey accepts.

type FolderSpec 

type FolderSpec struct {
	Name, Description string
	Limits            Limits
}

type LimitError 

type LimitError struct {
	Limit              string
	Maximum, Requested int64
	// contains filtered or unexported fields
}

LimitError identifies a folder limit that rejected an operation.

func (*LimitError) Error 

func (e *LimitError) Error() string

func (*LimitError) Unwrap 

func (e *LimitError) Unwrap() error

type Limits 

type Limits struct{ MaxFileBytes, MaxTotalBytes, MaxFiles, MaxRows int64 }

Limits bounds sealed bytes, live entries and retained rows. Zero is unlimited except MaxRows, which selects a bounded safety budget. Sharing requires explicit positive byte, row and file caps. Bytes include encryption and metadata overhead.

type MemoryBlobStore 

type MemoryBlobStore struct {
	// contains filtered or unexported fields
}

MemoryBlobStore is useful for tests and small in-process deployments.

func NewMemoryBlobStore 

func NewMemoryBlobStore() *MemoryBlobStore

func (*MemoryBlobStore) Delete 

func (b *MemoryBlobStore) Delete(ctx context.Context, folder, id string) error

func (*MemoryBlobStore) Open 

func (b *MemoryBlobStore) Open(ctx context.Context, folder, id string) (io.ReadCloser, error)

func (*MemoryBlobStore) Put 

func (b *MemoryBlobStore) Put(ctx context.Context, folder, id string, r io.Reader) (int64, error)

func (*MemoryBlobStore) Size 

func (b *MemoryBlobStore) Size(ctx context.Context, folder, id string) (int64, error)

type MetaStore 

type MetaStore struct {
	// contains filtered or unexported fields
}

MetaStore is the authority metadata store. Metadata storage is concrete because a custom transaction interface would expose private replication/accounting records. BlobStore and QuotaPolicy remain the pluggable extension points.

func OpenPostgresMetaStore added in v0.2.0 

func OpenPostgresMetaStore(ctx context.Context, db *sql.DB, schema string) (*MetaStore, error)

OpenPostgresMetaStore uses db, which the caller owns and must open with the pgx stdlib driver (driver name "pgx"). schema is created when missing; an empty schema name uses "drivesync". Close does not close db. Several authority processes may share one schema. db must allow at least three connections: one stays on LISTEN and two remain for transactions. A limit of one or two is rejected; unlimited (zero) is accepted. Garbage collection claims a lease row in the schema (a holder id and an expiry, renewed during the pass) and does not hold a pooled connection. A frozen holder stops blocking after the lease expires.

func OpenSQLiteMetaStore 

func OpenSQLiteMetaStore(path string) (*MetaStore, error)

func (*MetaStore) Close 

func (m *MetaStore) Close() error

type Options 

type Options struct {
	Name, StateDir                          string
	Debounce, RescanInterval, RetryInterval time.Duration
	Ignore                                  []string
	Manual                                  bool
	// Moved tells Attach this directory is the same root renamed on the same
	// volume. A matching marker token and inode keep the replica index.
	Moved bool
	// OnDemand lists every remote file but downloads one only when Fetch asks
	// for it. Until then the file is an empty placeholder with no permissions,
	// so opening it fails instead of reading wrong bytes. Downloaded files stay
	// downloaded and sync both ways.
	OnDemand bool
	// Downloads bounds concurrent blob downloads; zero selects 8.
	Downloads int
	// MaxPlaceholders bounds the placeholders an on-demand replica creates,
	// breadth first; zero selects 20,000. MaxFolderPlaceholders bounds one
	// folder's direct placeholders; zero selects 1,000. A folder past either
	// limit is collapsed: its contents stay listed in the index and its guide,
	// and Fetch still downloads them.
	MaxPlaceholders, MaxFolderPlaceholders int
	// GuideName and Guide, when both are set on an on-demand replica, write a
	// local-only file named GuideName at the root and in each collapsed folder:
	// Guide followed by that folder's entries not downloaded here. It never syncs.
	GuideName, Guide string
}

Options configures a local replica. StateDir must be dedicated and outside all synced trees; its default is a folder/root directory in the user's cache. Manual disables automatic sync; Sync remains available explicitly.

type Principal 

type Principal struct{ Tenant, Subject string }

Principal is an opaque identity supplied by the embedding authenticator.

type Quota 

type Quota struct{ MaxTotalBytes, MaxFolders, MaxFileBytes, MaxFiles int64 }

Quota is an owner's plan. MaxFiles bounds retained rows across their folders.

type QuotaFunc 

type QuotaFunc func(context.Context, Principal) (Quota, error)

func (QuotaFunc) Quota 

func (f QuotaFunc) Quota(ctx context.Context, p Principal) (Quota, error)

type QuotaPolicy 

type QuotaPolicy interface {
	Quota(context.Context, Principal) (Quota, error)
}

type Rejection 

type Rejection struct{ Path, Reason string }

type Replica 

type Replica struct {
	// contains filtered or unexported fields
}

Replica keeps an encrypted folder in an ordinary local directory.

func Attach 

func Attach(ctx context.Context, c *Client, id string, key FolderKey, dir string, opts Options) (*Replica, error)

Attach uses ctx for setup. The replica remains active until Close.

func (*Replica) Close 

func (r *Replica) Close() error

func (*Replica) Entries added in v0.4.0 

func (r *Replica) Entries(patterns ...string) ([]Entry, error)

Entries lists tracked paths from local state, without the network. A pattern is an exact path, a folder (its whole subtree), or a glob with * ? [] and **. No patterns lists everything. Results are sorted by path.

func (*Replica) Fetch added in v0.4.0 

func (r *Replica) Fetch(ctx context.Context, patterns ...string) (FetchResult, error)

Fetch downloads the on-demand placeholders matching patterns in parallel and returns once they are ordinary files. Downloaded files stay downloaded.

func (*Replica) Retry 

func (r *Replica) Retry()

Retry clears rejection/quarantine backoff and acknowledges intentional mass removal for the next sync. It never overrides a missing or changed root identity.

func (*Replica) Status 

func (r *Replica) Status() Status

func (*Replica) Sync 

func (r *Replica) Sync(ctx context.Context) error

type Role 

type Role string
const (
	Owner  Role = "owner"
	Writer Role = "writer"
	Reader Role = "reader"
)

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server authorizes every operation independently of transport authentication.

func NewServer 

func NewServer(meta *MetaStore, blobs BlobStore, opts ServerOptions) *Server

func (*Server) Client 

func (s *Server) Client(p Principal) *Client

Client trusts p, which the embedding app must authenticate before calling.

func (*Server) Handler 

func (s *Server) Handler(auth Authenticator) http.Handler

func (*Server) Run 

func (s *Server) Run(ctx context.Context) error

Run recovers interrupted uploads and maintains expired reservations, garbage and tombstones until ctx is cancelled. SQLite metadata is one authority process per database file. Postgres metadata can be shared: an upload lease is renewed while bytes flow, and recovery retires only an expired lease. Cancel and join Run before closing the stores.

type ServerOptions 

type ServerOptions struct {
	Quotas                                   QuotaPolicy
	ReservationTTL, TombstoneTTL, GCInterval time.Duration
	Clock                                    func() time.Time
	// MaxWaitsPerPrincipal caps concurrent long-polls for one principal.
	// Zero keeps the default of 256. Embedders that share one principal
	// across many clients set this higher.
	MaxWaitsPerPrincipal int
	// SnapshotRows is the live-row count at which a folder whose blob store
	// signs URLs offers new replicas a packed snapshot. Zero selects 1,000;
	// negative disables snapshots.
	SnapshotRows int
}

ServerOptions configures authority policy and maintenance before serving. Zero TTLs select defaults: five-minute reservations and 30-day tombstones. A negative TombstoneTTL disables compaction. GCInterval defaults to one second.

type Status 

type Status struct {
	PendingUpBytes, PendingDownBytes int64
	Conflicts                        []string
	Rejected                         []Rejection
	Quarantined                      []string
	LastSync                         time.Time
	Errors                           []string
	// Placeholders counts on-demand files not downloaded here, and
	// PlaceholderBytes their plaintext size.
	Placeholders, PlaceholderBytes int64
}

type Usage 

type Usage struct{ Bytes, Files, Reserved int64 }

Source Files

View all Source files

Directories

Path Synopsis
cmd
drivesync command
drivesync is a small manual-testing CLI.
 drivesync is a small manual-testing CLI.
internal
embedpg
Package embedpg starts one embedded Postgres per process for tests.
 Package embedpg starts one embedded Postgres per process for tests.
engine
Package engine implements the private encrypted replication protocol.
 Package engine implements the private encrypted replication protocol.
testkit
Package testkit opens the metadata store selected by DRIVESYNC_STORE.
 Package testkit opens the metadata store selected by DRIVESYNC_STORE.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.