README
¶
overlay
Makes a read-only filesystem writable without rewriting it on every change —
pure Go, CGO_ENABLED=0, builds for every target Go builds for.
o, err := overlay.New(base) // base is any filesystem.Filesystem
defer o.Close() // discards unsealed changes, on purpose
o.WriteFile("/notes.txt", b, 0o644) // lands in a spool; costs what it costs
o.DeleteFile("/old.log") // shadowed, not moved
o.Seal(builder) // the container is rewritten ONCE, here
Why it exists
An archive is a read-only filesystem: iso9660, squashfs, zip and tar all answer reads and refuse writes, because there is nothing to mutate in place.
Offering a writable Filesystem over one anyway means every WriteFile rewrites
the whole container — a DeleteFile on three gigabytes moving three gigabytes —
and the interface then promises something cheap that is not. The caller finds out
by watching a disk.
So changes accumulate instead. A write lands in a spool directory; the container is
rewritten once, by Seal, at a moment the caller chose and named. The interface
stops lying: mutation is cheap, and the expensive thing has its own name.
Only what changed is copied
An entry nobody touched is read from the base, and at seal time handed to the builder straight from the base. An archive of three gigabytes with one small file replaced spools that one small file.
⛔ That streaming path was unreachable for a while, and nothing noticed. Seal
asserted an anonymous interface for "can this hand out a file handle" —
structurally identical to filesystem.Opener, and therefore satisfied by nothing,
because Go interface satisfaction is by method set against a named type. Every
seal read whole files into memory and the result was correct every time. What found
it was a test that asked the base which handles it had been asked for and got
back an empty list.
Seal takes a Builder, and the size is part of it
type Builder interface {
AddDir(name string, perm os.FileMode) error
AddFile(name string, perm os.FileMode, size int64, r io.Reader) error
}
The size is given rather than discovered because tar cannot be written without it: an entry's length goes in its header, before its bytes. A builder handed only a reader would have to spool the whole entry to learn how long it is — copying a 1.2 GiB video to a temporary file to count it, when the caller knew the answer — which is the cost this whole layer exists to avoid. A format that does not need the size ignores it.
r delivers exactly size bytes. A builder may rely on that; a caller that breaks
it produces an archive whose header disagrees with its data, which is the one
failure every reader reports as corruption.
Entries arrive in path order, directories before what is inside them, so a builder that must create a parent before its children can rely on that rather than sorting for itself.
Implementations in the family: iso9660.Builder, sevenzip's writer, and the tar,
zip, ar, cpio and rar builders in
go-filesystems/unarchive.
Close discards, Seal commits
Unsealed changes live in the spool and nowhere else. Close throws them away, on
purpose and loudly:
- a
Closethat sealed in passing would commit a rewrite nobody asked for; - a
Closethat silently kept the spool would leave gigabytes in a temporary directory for the next reboot to collect.
WithSpool(dir) puts the working copies somewhere the caller owns, and Close
then leaves that directory alone.
Untrusted base, bounded spool
The base is an archive or an image somebody handed over, so Seal does not trust
its directory listings. WithLimits(overlay.Limits{...}) sets the bounds; a bound
that is hit returns an error wrapping ErrTooLarge, before the memory is spent:
| Bound | Default | Why |
|---|---|---|
entries Seal walks in the base (MaxEntries) |
4 Mi | every entry is held in memory (about 200 bytes) until the seal ends |
directory nesting (MaxDepth) |
2048 | PATH_MAX (4096) holds at most 2048 one-letter components |
live working copies in the spool (MaxSpool) |
unlimited | set it to cap the disk a writer can take |
A base that lists a directory inside itself (hard-linked directories in a
crafted image) used to make Seal recurse and grow without end: it had not
returned after 200 s. . and .. in a listing are not entries and are skipped;
an empty name or a name containing / is refused. WriteFile used to leave the
copy it replaced in the spool, and DeleteFile / DeleteDir the copy of what
they deleted: 100 rewrites of one 1 MiB file held 100 MiB until Close. They
are now released, and the bytes counted equal the live copies (FuzzSpoolAccounting
checks that; FuzzSealBase drives Seal over hostile listings).
What the upper layer does not do
| symbolic links | ReadLink asks the base; nothing in the upper layer makes links |
| in-place mutation of the base | never — the base is read, and only read |
Rename |
a read and two writes, which is what it costs here |
Licence
BSD-3-Clause.
Documentation
¶
Overview ¶
Package overlay makes a read-only filesystem writable without rewriting it on every change.
The problem it solves ¶
An archive is a read-only filesystem: iso9660, squashfs, zip and tar all answer reads and refuse writes, because there is nothing to mutate in place. Offering a writable Filesystem over one anyway means every WriteFile rewrites the whole container -- a DeleteFile on three gigabytes moving three gigabytes -- and the interface then promises something cheap that is not. The caller finds out by watching a disk.
So changes are accumulated instead. A write lands in a spool directory and costs what that write costs; the container is rewritten ONCE, by Overlay.Seal, at a moment the caller chose and named. The interface stops lying: mutation is cheap, and the expensive thing has its own name.
What is copied ¶
Only what changed. An entry nobody touched is read from the base and, at seal time, handed to the builder straight from the base -- so an archive of three gigabytes with one small file replaced spools that one small file.
Losing changes ¶
Unsealed changes live in the spool and nowhere else. Overlay.Close discards them, on purpose and loudly: a Close that sealed in passing would commit a rewrite nobody asked for, and a Close that silently kept a spool would leave gigabytes in a temporary directory for the next reboot to collect. Seal is the commit.
Index ¶
- Constants
- Variables
- type Builder
- type Limits
- type Option
- type Overlay
- func (o *Overlay) Close() error
- func (o *Overlay) DeleteDir(p string) error
- func (o *Overlay) DeleteFile(p string) error
- func (o *Overlay) ListDir(dir string) ([]filesystem.DirEntry, error)
- func (o *Overlay) MkDir(p string, perm os.FileMode) error
- func (o *Overlay) OpenFile(p string) (filesystem.File, error)
- func (o *Overlay) ReadFile(p string) ([]byte, error)
- func (o *Overlay) ReadLink(p string) (string, error)
- func (o *Overlay) Rename(from, to string) error
- func (o *Overlay) Seal(b Builder) error
- func (o *Overlay) SealToFile(target string, mk func(io.WriteSeeker) (Builder, error)) error
- func (o *Overlay) Spool() string
- func (o *Overlay) Stat(p string) (filesystem.Stat, error)
- func (o *Overlay) WriteFile(p string, data []byte, perm os.FileMode) error
Constants ¶
const ( // DefaultMaxEntries is the most entries Seal walks in the base. Every entry // is held in memory (about 200 bytes) until the seal ends, and a base that // lists a directory inside itself yields entries without end. DefaultMaxEntries = 4 << 20 // DefaultMaxDepth is the deepest directory nesting Seal descends: PATH_MAX // (4096 on Linux) holds at most 2048 one-letter components. DefaultMaxDepth = 2048 )
Defaults of Limits.
Variables ¶
var ( // ErrNotFound is returned for a path that is in neither layer, or that a // deletion has shadowed. ErrNotFound = errors.New("overlay: path not found") // ErrNotDirectory is returned when a directory operation names a file. ErrNotDirectory = errors.New("overlay: not a directory") // ErrIsDirectory is returned when a file operation names a directory. ErrIsDirectory = errors.New("overlay: is a directory") // ErrClosed is returned once the overlay has been closed. ErrClosed = errors.New("overlay: closed") // ErrExists is returned by MkDir for a path already taken. ErrExists = errors.New("overlay: already exists") // ErrTooLarge is returned, wrapped with the figure that was exceeded, when a // bound of [Limits] is hit: a base that lists more entries or nests deeper // than the limits allow, or a write that would take the spool past // MaxSpool. The work stops before the memory or disk is spent. ErrTooLarge = errors.New("overlay: limit exceeded") )
Errors this package returns.
Functions ¶
This section is empty.
Types ¶
type Builder ¶
type Builder interface {
// AddDir records a directory. A format with no directory entries may
// ignore it.
AddDir(name string, perm os.FileMode) error
// AddFile records a file of exactly size bytes and reads them from r.
//
// The size is given rather than discovered because tar cannot be written
// without it: an entry's length goes in its header, BEFORE its bytes. A
// builder handed only a reader would have to spool the whole entry to learn
// how long it is -- copying a 1.2 GiB video to a temporary file to count it,
// when the caller knew the answer -- which is the cost this whole layer
// exists to avoid. A format that does not need the size ignores it.
//
// r delivers exactly size bytes. A builder may rely on that; a caller that
// breaks it produces an archive whose header disagrees with its data, which
// is the one failure every reader reports as corruption.
AddFile(name string, perm os.FileMode, size int64, r io.Reader) error
}
Builder is what a format offers for authoring one: iso9660's Builder, a zip writer, this org's sevenzip writer. Seal drives it and knows nothing else about the format.
type Limits ¶ added in v0.4.0
type Limits struct {
// MaxEntries bounds the entries Seal walks in the base. Default
// DefaultMaxEntries.
MaxEntries int
// MaxDepth bounds the directory nesting Seal descends. Default
// DefaultMaxDepth.
MaxDepth int
// MaxSpool bounds the bytes of live working copies in the spool; a write
// that would exceed it fails with ErrTooLarge before touching the disk. Zero
// is unlimited: the spool is as large as the changes the caller makes.
MaxSpool int64
}
Limits bounds what an Overlay spends on its base and its spool. A zero field takes its default, except MaxSpool, where zero means unlimited.
type Option ¶
type Option func(*Overlay)
Option configures an Overlay.
func WithLimits ¶ added in v0.4.0
WithLimits sets the bounds. See Limits.
type Overlay ¶
type Overlay struct {
// contains filtered or unexported fields
}
Overlay is a read-only filesystem plus the changes made to it.
func New ¶
func New(base filesystem.Filesystem, opts ...Option) (*Overlay, error)
New wraps base. Nothing is read from it until something asks.
func (*Overlay) DeleteFile ¶
DeleteFile shadows a path. The base keeps it; the overlay stops answering for it, and Seal leaves it out.
func (*Overlay) ListDir ¶
func (o *Overlay) ListDir(dir string) ([]filesystem.DirEntry, error)
ListDir merges the two layers: what the base holds, minus what is shadowed, plus what was added.
func (*Overlay) MkDir ¶
MkDir records a directory. Nothing is created in the base, which has no idea this happened until Seal.
func (*Overlay) OpenFile ¶
func (o *Overlay) OpenFile(p string) (filesystem.File, error)
OpenFile hands out a handle. A working copy is opened from the spool; an untouched entry is asked of the base, if the base offers handles at all.
func (*Overlay) ReadFile ¶
ReadFile reads the working copy when there is one, and the base otherwise.
func (*Overlay) Rename ¶
Rename is a read and two writes, which is what it costs here: the point of the spool is that those are cheap.
func (*Overlay) Seal ¶
Seal hands the merged view to a builder, once.
Entries are offered in path order, directories before what is inside them, because a builder that has to create a parent before its children can then rely on that rather than sorting for itself.
An entry nobody touched is streamed from the BASE. Only what changed was ever copied, which is what keeps a small edit to a large archive small: the spool holds the edit, and the rest goes straight through.
func (*Overlay) SealToFile ¶
SealToFile seals into target, and does it by RENAME.
The new archive is written beside the target and moved over it at the end, so an interrupted seal -- a full disk, a signal, a power cut -- leaves the original exactly as it was. Sealing in place would destroy the archive in order to write it, which turns every interruption into data loss, and the window is the whole rewrite.
func (*Overlay) Spool ¶
Spool is where working copies are kept, so a caller can say how much room the changes are taking.
Source Files