drivesync

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 6, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

README

drivesync

drivesync shares an encrypted folder through ordinary directories on macOS and Linux. Create a folder with a name and a key, then attach it on each node. Reads are normal local files. Replicas sync both ways. There is no FUSE layer. Pure-Go builds are supported.

The authority stores ciphertext and control-plane records. It does not receive the folder key, file paths, file contents, or file metadata. SECURITY.md describes what the server can see, what a key holder can do, and the local-disk guarantees.

Quickstart

SQLite metadata is one authority process per database file. Postgres metadata, opened with OpenPostgresMetaStore, can be shared by several authority processes. The pool needs at least three connections: one stays on LISTEN and two remain for transactions. Each process listens for the others' commits. An upload records a lease renewed while bytes flow; recovery retires only a ticket whose lease has expired. A garbage-collection pass claims a lease row (a holder id and an expiry, renewed during the pass) or skips when another process holds an unexpired lease. A frozen holder stops blocking once that lease expires. auth authenticates each HTTP request and returns a Principal{Tenant, Subject}. plans implements QuotaPolicy.

meta, err := drivesync.OpenSQLiteMetaStore("authority.sqlite")
if err != nil { return err }
defer meta.Close()
blobs, err := drivesync.OpenDirectoryBlobStore("blobs")
if err != nil { return err }
defer blobs.Close()

srv := drivesync.NewServer(meta, blobs, drivesync.ServerOptions{Quotas: plans})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- srv.Run(ctx) }()
defer func() { cancel(); <-done }()
mux.Handle("/drives/", http.StripPrefix("/drives", srv.Handler(auth)))

Run recovers interrupted uploads, expires reservations, collects garbage, and compacts tombstones. Cancel it and wait before closing the stores.

On a client node, or with srv.Client(principal) inside the authority process:

c := drivesync.NewHTTPClient("https://example.com/drives", headers)
key := drivesync.NewFolderKey()
f, err := c.CreateFolder(ctx, drivesync.FolderSpec{
    Name: "workspace", Description: "Files shared across nodes",
    Limits: drivesync.Limits{
        MaxFileBytes: 64 << 20, MaxTotalBytes: 1 << 30,
        MaxFiles: 10000, MaxRows: 50000,
    },
}, key)
if err != nil { return err }
if err := c.Grant(ctx, f.ID, bob, drivesync.Writer); err != nil { return err }
r, err := drivesync.Attach(ctx, c, f.ID, key, "./workspace", drivesync.Options{})
if err != nil { return err }
defer r.Close()
status := r.Status()

Preserve the key and distribute it to each attachment. Creation computes the key check locally. Folder ids are random 128-bit ids. Names are unique per primary owner. Grant, Revoke, SetLimits, ListFolders, GetFolder, and DeleteFolder manage the folder.

Attach's context bounds setup. The replica runs until Close. Sync forces one pass. Status reports pending bytes, conflicts, rejections, quarantine, and errors. Retry retries rejected or quarantined transfers and acknowledges an intentional mass removal.

A small CLI exercises the same API with one local principal. Set DRIVESYNC_TOKEN. Pass the key in DRIVESYNC_KEY or -key-file, never in argv.

go run ./cmd/drivesync serve -data ./drivesync-data
go run ./cmd/drivesync create -name shared
go run ./cmd/drivesync attach -folder ID -key-file ./folder.key -dir ./shared -state ./replica-state
go run ./cmd/drivesync status -state ./replica-state

Limits and quotas

Limits bounds sealed bytes, live entries, and retained rows. MaxFileBytes is the sealed size, including encryption overhead. MaxFiles counts live entries, including directory markers. MaxRows bounds retained rows and garbage. Zero MaxRows selects 16 times MaxFiles, capped at one million. Other zero limits mean unlimited.

Sharing, including a grant in the same tenant, requires a positive byte cap, row cap, and per-file cap. That capacity is charged to the primary owner until the folder is deleted and cleaned up. Reservations are atomic, and commit checks the stored size. Rejected files stay on the local disk with a reason and a retry backoff.

Folder reports id, name, description, owner, role, limits, and usage. Usage is sealed bytes, live entries, and reserved bytes.

Platform support

macOS and Linux are supported, including CGO_ENABLED=0. macOS watches the tree with FSEvents through purego. Linux uses inotify. Both debounce for 150 ms and rescan periodically. kqueue is not used. Other operating systems are not supported.

How it works

A replica scans its directory, encrypts new bytes, and uploads ciphertext. Peers download into a staging file, authenticate it, then publish with fsync and rename. Renames commit as one create/delete pair. A losing local write is kept as a conflict copy. Matching authenticated content is adopted in place, so an existing tree can be attached without uploading those bytes again.

Path ids, metadata, and content use separate keys. Chunks are bound to the folder, blob, path, index, and a final flag. Case collisions keep both contents under distinct local names. Only the owner execute bit syncs. Group and other permission bits stay local.

Default state lives in os.UserCacheDir()/drivesync/<folder>/<root-hash> with private permissions. It holds plaintext paths, so keep it outside other shared folders. Options.StateDir must be dedicated to that attachment. A nonempty directory without the matching state marker is refused. .drivesync* names are reserved. .drivesyncignore accepts globs, root-anchored / patterns, **, comments, and directory patterns, matched under case-folded NFC. Negation and escapes are reported as unsupported.

.git/ is ignored by default. Sync the working tree, and let git carry history.

The included blob stores are an in-memory store and a durable local directory. BlobStore.Put must publish only after a clean EOF. An embedder can add an object-store backend without taking an SDK dependency. Metadata is SQLite or Postgres. OpenPostgresMetaStore needs a pool of at least three connections. QuotaPolicy is the other extension point.

Fresh databases are required. This version does not migrate older schemas, and it does not provide historical recovery, key rotation, or chunk deduplication.

License

The code is licensed under Apache-2.0. Copyright 2026 Parham Negahdar. See LICENSE.

Unicode path handling is checked against the Unicode Consortium's NormalizationTest data, which remains under the Unicode License. See UNICODE_LICENSE.txt. The corpus is internal/engine/testdata/NormalizationTest.txt.gz.

Documentation

Overview

Package drivesync shares encrypted folders through ordinary local directories. Create a folder with a name and a key, then attach it wherever files are needed.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrDenied    = engine.ErrDenied
	ErrBusy      = engine.ErrBusy
	ErrExpired   = engine.ErrExpired
	ErrQuota     = engine.ErrQuota
	ErrConflict  = engine.ErrConflict
	ErrInvalid   = engine.ErrInvalid
	ErrKey       = engine.ErrKey
	ErrIntegrity = engine.ErrIntegrity
	ErrWaitLimit = engine.ErrWaitLimit
)

Functions

This section is empty.

Types

type Authenticator 

type Authenticator func(*http.Request) (Principal, error)

type BlobStore 

type BlobStore interface {
	Put(context.Context, string, string, io.Reader) (int64, error)
	Open(context.Context, string, string) (io.ReadCloser, error)
	Size(context.Context, string, string) (int64, error)
	Delete(context.Context, string, string) error
}

BlobStore holds immutable sealed blobs. Keys are authority-generated IDs. Put streams bytes, Size reports actual stored bytes, and Delete is idempotent (os.ErrNotExist is also accepted). Implementations must honor cancellation. Put must publish only after clean EOF; a reader error must discard staging.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client manages folders through HTTP or an authenticated in-process authority. Its replication and encryption protocol is private to the library.

func NewHTTPClient 

func NewHTTPClient(url string, headers http.Header) *Client

NewHTTPClient connects to an authority. Use HTTPS outside local tests.

func (*Client) CreateFolder 

func (c *Client) CreateFolder(ctx context.Context, spec FolderSpec, key FolderKey) (Folder, error)

CreateFolder computes the key check locally; the key is never sent to the authority. Keep it securely and supply the same key to every attachment.

func (*Client) DeleteFolder 

func (c *Client) DeleteFolder(ctx context.Context, id string) error

func (*Client) GetFolder 

func (c *Client) GetFolder(ctx context.Context, id string) (Folder, error)

func (*Client) Grant 

func (c *Client) Grant(ctx context.Context, id string, p Principal, role Role) error

func (*Client) ListFolders 

func (c *Client) ListFolders(ctx context.Context) ([]Folder, error)

func (*Client) Revoke 

func (c *Client) Revoke(ctx context.Context, id string, p Principal) error

func (*Client) SetLimits 

func (c *Client) SetLimits(ctx context.Context, id string, limits Limits) error

type DirectoryBlobStore 

type DirectoryBlobStore struct {
	// contains filtered or unexported fields
}

DirectoryBlobStore streams sealed bytes to a durable local directory.

func OpenDirectoryBlobStore 

func OpenDirectoryBlobStore(path string) (*DirectoryBlobStore, error)

func (*DirectoryBlobStore) Close 

func (b *DirectoryBlobStore) Close() error

func (*DirectoryBlobStore) Delete 

func (b *DirectoryBlobStore) Delete(ctx context.Context, folder, id string) error

func (*DirectoryBlobStore) Open 

func (b *DirectoryBlobStore) Open(ctx context.Context, folder, id string) (io.ReadCloser, error)

func (*DirectoryBlobStore) Put 

func (b *DirectoryBlobStore) Put(ctx context.Context, folder, id string, r io.Reader) (int64, error)

func (*DirectoryBlobStore) Size 

func (b *DirectoryBlobStore) Size(ctx context.Context, folder, id string) (int64, error)

type Folder 

type Folder struct {
	ID                string
	Name, Description string
	Owner             Principal
	Role              Role
	Limits            Limits
	Usage             Usage
}

type FolderKey 

type FolderKey [32]byte

func NewFolderKey 

func NewFolderKey() FolderKey

func ParseFolderKey 

func ParseFolderKey(s string) (FolderKey, error)

ParseFolderKey imports a 64-character hex encoding of a random 256-bit key. Passphrases and hashes of passphrases are not suitable folder keys.

func (FolderKey) String added in v0.3.0 

func (k FolderKey) String() string

String is the 64-character hex form ParseFolderKey accepts.

type FolderSpec 

type FolderSpec struct {
	Name, Description string
	Limits            Limits
}

type LimitError 

type LimitError struct {
	Limit              string
	Maximum, Requested int64
	// contains filtered or unexported fields
}

LimitError identifies a folder limit that rejected an operation.

func (*LimitError) Error 

func (e *LimitError) Error() string

func (*LimitError) Unwrap 

func (e *LimitError) Unwrap() error

type Limits 

type Limits struct{ MaxFileBytes, MaxTotalBytes, MaxFiles, MaxRows int64 }

Limits bounds sealed bytes, live entries and retained rows. Zero is unlimited except MaxRows, which selects a bounded safety budget. Sharing requires explicit positive byte, row and file caps. Bytes include encryption and metadata overhead.

type MemoryBlobStore 

type MemoryBlobStore struct {
	// contains filtered or unexported fields
}

MemoryBlobStore is useful for tests and small in-process deployments.

func NewMemoryBlobStore 

func NewMemoryBlobStore() *MemoryBlobStore

func (*MemoryBlobStore) Delete 

func (b *MemoryBlobStore) Delete(ctx context.Context, folder, id string) error

func (*MemoryBlobStore) Open 

func (b *MemoryBlobStore) Open(ctx context.Context, folder, id string) (io.ReadCloser, error)

func (*MemoryBlobStore) Put 

func (b *MemoryBlobStore) Put(ctx context.Context, folder, id string, r io.Reader) (int64, error)

func (*MemoryBlobStore) Size 

func (b *MemoryBlobStore) Size(ctx context.Context, folder, id string) (int64, error)

type MetaStore 

type MetaStore struct {
	// contains filtered or unexported fields
}

MetaStore is the authority metadata store. Metadata storage is concrete because a custom transaction interface would expose private replication/accounting records. BlobStore and QuotaPolicy remain the pluggable extension points.

func OpenPostgresMetaStore added in v0.2.0 

func OpenPostgresMetaStore(ctx context.Context, db *sql.DB, schema string) (*MetaStore, error)

OpenPostgresMetaStore uses db, which the caller owns and must open with the pgx stdlib driver (driver name "pgx"). schema is created when missing; an empty schema name uses "drivesync". Close does not close db. Several authority processes may share one schema. db must allow at least three connections: one stays on LISTEN and two remain for transactions. A limit of one or two is rejected; unlimited (zero) is accepted. Garbage collection claims a lease row in the schema (a holder id and an expiry, renewed during the pass) and does not hold a pooled connection. A frozen holder stops blocking after the lease expires.

func OpenSQLiteMetaStore 

func OpenSQLiteMetaStore(path string) (*MetaStore, error)

func (*MetaStore) Close 

func (m *MetaStore) Close() error

type Options 

type Options struct {
	Name, StateDir                          string
	Debounce, RescanInterval, RetryInterval time.Duration
	Ignore                                  []string
	Manual                                  bool
	// Moved tells Attach this directory is the same root renamed on the same
	// volume. A matching marker token and inode keep the replica index.
	Moved bool
}

Options configures a local replica. StateDir must be dedicated and outside all synced trees; its default is a folder/root directory in the user's cache. Manual disables automatic sync; Sync remains available explicitly.

type Principal 

type Principal struct{ Tenant, Subject string }

Principal is an opaque identity supplied by the embedding authenticator.

type Quota 

type Quota struct{ MaxTotalBytes, MaxFolders, MaxFileBytes, MaxFiles int64 }

Quota is an owner's plan. MaxFiles bounds retained rows across their folders.

type QuotaFunc 

type QuotaFunc func(context.Context, Principal) (Quota, error)

func (QuotaFunc) Quota 

func (f QuotaFunc) Quota(ctx context.Context, p Principal) (Quota, error)

type QuotaPolicy 

type QuotaPolicy interface {
	Quota(context.Context, Principal) (Quota, error)
}

type Rejection 

type Rejection struct{ Path, Reason string }

type Replica 

type Replica struct {
	// contains filtered or unexported fields
}

Replica keeps an encrypted folder in an ordinary local directory.

func Attach 

func Attach(ctx context.Context, c *Client, id string, key FolderKey, dir string, opts Options) (*Replica, error)

Attach uses ctx for setup. The replica remains active until Close.

func (*Replica) Close 

func (r *Replica) Close() error

func (*Replica) Retry 

func (r *Replica) Retry()

Retry clears rejection/quarantine backoff and acknowledges intentional mass removal for the next sync. It never overrides a missing or changed root identity.

func (*Replica) Status 

func (r *Replica) Status() Status

func (*Replica) Sync 

func (r *Replica) Sync(ctx context.Context) error

type Role 

type Role string
const (
	Owner  Role = "owner"
	Writer Role = "writer"
	Reader Role = "reader"
)

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server authorizes every operation independently of transport authentication.

func NewServer 

func NewServer(meta *MetaStore, blobs BlobStore, opts ServerOptions) *Server

func (*Server) Client 

func (s *Server) Client(p Principal) *Client

Client trusts p, which the embedding app must authenticate before calling.

func (*Server) Handler 

func (s *Server) Handler(auth Authenticator) http.Handler

func (*Server) Run 

func (s *Server) Run(ctx context.Context) error

Run recovers interrupted uploads and maintains expired reservations, garbage and tombstones until ctx is cancelled. SQLite metadata is one authority process per database file. Postgres metadata can be shared: an upload lease is renewed while bytes flow, and recovery retires only an expired lease. Cancel and join Run before closing the stores.

type ServerOptions 

type ServerOptions struct {
	Quotas                                   QuotaPolicy
	ReservationTTL, TombstoneTTL, GCInterval time.Duration
	Clock                                    func() time.Time
	// MaxWaitsPerPrincipal caps concurrent long-polls for one principal.
	// Zero keeps the default of 256. Embedders that share one principal
	// across many clients set this higher.
	MaxWaitsPerPrincipal int
}

ServerOptions configures authority policy and maintenance before serving. Zero TTLs select defaults: five-minute reservations and 30-day tombstones. A negative TombstoneTTL disables compaction. GCInterval defaults to one second.

type Status 

type Status struct {
	PendingUpBytes, PendingDownBytes int64
	Conflicts                        []string
	Rejected                         []Rejection
	Quarantined                      []string
	LastSync                         time.Time
	Errors                           []string
}

type Usage 

type Usage struct{ Bytes, Files, Reserved int64 }

Source Files

View all Source files

Directories

Path Synopsis
cmd
drivesync command
drivesync is a small manual-testing CLI.
 drivesync is a small manual-testing CLI.
internal
embedpg
Package embedpg starts one embedded Postgres per process for tests.
 Package embedpg starts one embedded Postgres per process for tests.
engine
Package engine implements the private encrypted replication protocol.
 Package engine implements the private encrypted replication protocol.
testkit
Package testkit opens the metadata store selected by DRIVESYNC_STORE.
 Package testkit opens the metadata store selected by DRIVESYNC_STORE.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.