Documentation
¶
Overview ¶
Package blobfs is the root package of blobfs, a SQL-backed tree of directories and file metadata over an object store the library does not import: the entity types, the root's id (RootID), the status vocabulary and its transitions, key construction and filename sanitizing, name normalization and validation, the KeyValidator interface a store satisfies, and the errors the persistence package maps violations onto.
The package is Go only; its one dependency outside the standard library is golang.org/x/text, for Unicode normalization. A consumer that keeps its own persistence takes only this package. A file's key, <id>/<sanitized name>, is opaque: nothing in blobfs parses it.
docs/concepts.md explains the model and docs/features.md states every rule, the schema included.
Index ¶
- Constants
- Variables
- func CanTransition(from, to Status) bool
- func NewID() string
- func NewKey(store KeyValidator, id, name string) (string, error)
- func NormalizeName(name string) string
- func ParseID(id string) (string, error)
- func SanitizeFilename(name string) string
- func Transition(from, to Status) error
- func ValidateName(name string) error
- type DeletingError
- type Directory
- type DirectoryStatus
- type File
- type IDError
- type KeyError
- type KeyValidator
- type NameError
- type Object
- type Status
- type TransitionError
- type ViolationError
Constants ¶
const ( // ConstraintPrimaryKeyDirectory is the primary key of blobfs_directory. // A violation is ErrIDTaken: a caller-supplied id another directory // already carries. ConstraintPrimaryKeyDirectory = "blobfs_pk_directory" // ConstraintPrimaryKeyFile is the primary key of blobfs_file. A // violation is ErrIDTaken: a caller-supplied id another file already // carries. ConstraintPrimaryKeyFile = "blobfs_pk_file" // ConstraintUniqueDirectoryRoot is the partial unique index that allows // one directory row with no parent: the one root per install. A // violation is ErrRootDirectory. ConstraintUniqueDirectoryRoot = "blobfs_uq_directory_root" // ConstraintUniqueDirectoryParentName is the unique constraint on // blobfs_directory (parent_id, name). A violation is ErrNameTaken. A // name a deleting row holds is refused before the constraint is reached. ConstraintUniqueDirectoryParentName = "blobfs_uq_directory_parent_name" // ConstraintUniqueFileDirectoryName is the unique constraint on // blobfs_file (directory_id, name). A violation is ErrNameTaken. A name // a deleting row holds is refused before the constraint is reached. ConstraintUniqueFileDirectoryName = "blobfs_uq_file_directory_name" // ConstraintForeignKeyDirectoryParent is the foreign key from // blobfs_directory.parent_id to blobfs_directory.id. On an insert or a // move a violation is ErrNotFound; on a delete it means the directory // still has child directories. ConstraintForeignKeyDirectoryParent = "blobfs_fk_directory_parent" // ConstraintForeignKeyFileDirectory is the foreign key from // blobfs_file.directory_id to blobfs_directory.id. On an insert or a // move a violation is ErrNotFound; on a delete it means the directory // still has files. ConstraintForeignKeyFileDirectory = "blobfs_fk_file_directory" )
The names of the constraints and unique indexes blobfs's DDL declares and package data maps to sentinel errors. Constraint names are public API: a violation reaches a consumer as ViolationError.Constraint, with the sqlate.ConstraintError beneath it, and the scheme is blobfs_<kind>_<table>_<detail>. The constants live here rather than in an engine package because package data imports this package and must not import an engine package. The postgres engine package's tests check that every constant names a constraint or an index in its DDL.
const MaxNameLength = 255
MaxNameLength is the longest name a directory or file may carry, counted in runes. It is the per-component limit the common file systems share (ext4, NTFS, and APFS all stop at 255), so a name copied from a local disk always fits. A key built from the longest name is 292 runes, within Azure Blob Storage's limit of 1024 characters. S3 counts its limit of 1024 in bytes of UTF-8, and a name of 255 four-byte runes makes a key of 1057 bytes, so the longest names can exceed it: the store's validator then refuses the key, and the write is a KeyError before any SQL.
const RootID = "00000000-0000-0000-0000-000000000000"
RootID is the id of the one root directory of an install: the nil UUID, seeded by the directory migration. Every path starts at the root, and a consumer that needs the root reads it by this id instead of searching for the row with no parent.
Variables ¶
var ( // ErrNotFound reports a directory or file that does not exist, including // a parent or directory id that an insert or move referenced. ErrNotFound = errors.New("blobfs: not found") // ErrNameTaken reports a name already held in the target directory by a // row of the same table that is not deleting. A deleting row also holds // its name until it is removed, but package data refuses a create or a // move onto that name with the row's DeletingError, which does not match // ErrNameTaken: the listings hide the deleting row, and its delete is // the refusal. ErrNameTaken = errors.New("blobfs: name taken") // ErrInvalidName reports a name ValidateName refused. A NameError // carries the reason. ErrInvalidName = errors.New("blobfs: invalid name") // ErrInvalidPath reports a path package data refused: one that starts // with a slash, or one with a segment ValidateName refuses, in which // case the error also matches ErrInvalidName. A path is relative, a/b // below the directory it starts from; a spelling from the root, /a/b, // is a consumer's own input syntax, which it strips before resolving // from RootID. ErrInvalidPath = errors.New("blobfs: invalid path") // ErrRootDirectory reports an operation refused because it targets the // root: creating a second row with no parent, or deleting, moving, or // renaming the root. There is exactly one root per install, seeded by // the schema with the id RootID. ErrRootDirectory = errors.New("blobfs: the root directory") // ErrInvalidKey reports a key the store refused. A KeyError carries the // key and the store's reason. ErrInvalidKey = errors.New("blobfs: invalid key") // ErrInvalidID reports a caller-supplied row id that ParseID refused: // text that is not a UUID, or the nil UUID, which is RootID. An IDError // carries the reason. ErrInvalidID = errors.New("blobfs: invalid id") // ErrIDTaken reports a caller-supplied row id that a row of the same // table already carries: the primary key refused the insert. ErrIDTaken = errors.New("blobfs: id taken") // ErrNotEmpty reports a directory delete refused because the directory // still has child directories or files. ErrNotEmpty = errors.New("blobfs: directory not empty") // ErrInvalidTransition reports a status change the transition table // does not allow. A TransitionError carries the two statuses. ErrInvalidTransition = errors.New("blobfs: invalid status transition") // ErrDeleting reports a mutation refused because the row is deleting: // a move or rename, or a status change out of deleting. It also reports // a mutation refused because a directory it reaches is deleting, since a // branch marked for removal takes nothing in and lets nothing out: a // create or an ensure under a deleting directory, a move into one, and a // move of a directory or file whose parent is deleting. A listing of a // deleting directory reports it too, unless the listing asked to include // deleting rows. Package data reports these as a DeletingError, which // says whose delete refused the mutation, whenever it can read the rows // that decide it. ErrDeleting = errors.New("blobfs: row is deleting") // ErrNotDeleting reports a purge, the last step of the two-phase // delete, refused because the row exists and is not deleting: its // delete has not begun. The caller runs the delete's first step first. ErrNotDeleting = errors.New("blobfs: the file is not deleting") // ErrReferenced reports a delete refused by a foreign key blobfs does // not own: a consumer's row still references the file or directory. // The sqlate.ConstraintError stays reachable, so the consumer matches // the constraint's name against its own and classifies further. ErrReferenced = errors.New("blobfs: the row is referenced by a consumer's row") // ErrCycle reports a directory move whose new parent sits inside the // directory's own subtree, the directory itself included. ErrCycle = errors.New("blobfs: move would create a cycle") )
The sentinels package data maps its outcomes onto. A database violation of one of blobfs's own documented constraints becomes one of these, carried by a ViolationError; a violation of a consumer's constraint is left unclassified.
Functions ¶
func CanTransition ¶
CanTransition reports whether a row may move from one status to another. A change the table does not list is refused, including every change out of deleting and every change between the same non-deleting status.
func NewID ¶
func NewID() string
NewID mints a new row id: a version 7 UUID in its canonical string form, so ids sort in insertion order and bind to a uuid column as text.
func NewKey ¶
func NewKey(store KeyValidator, id, name string) (string, error)
NewKey builds a file's key from its id and display name and validates it against the store. The key is id, a slash, and SanitizeFilename(name). A refusal is a KeyError carrying the store's reason.
func NormalizeName ¶
NormalizeName returns name in Unicode normalization form C. Uniqueness is an exact match on the stored name, so package data normalizes every name before an insert or a rename: a composed and a decomposed spelling of the same name then collide instead of coexisting.
func ParseID ¶
ParseID checks a row id a caller supplies in place of a minted one and returns it in the canonical form NewID mints, so the row and the key built from its id carry the same text whatever form the caller wrote. Any form uuid.Parse accepts is taken. The nil UUID is refused, because it is RootID and belongs to the seeded root. A refusal is an IDError.
func SanitizeFilename ¶
SanitizeFilename turns a display name into the filename segment of a key. The segment is frozen when the pending row is inserted, is never kept in step with a later rename, and is never parsed back out of the key. It exists so an operator browsing the raw container can read it.
The result satisfies the key rules of the providers blobfs targets: each invalid UTF-8 sequence, slash, backslash, and control character becomes an underscore, trailing dots and whitespace are removed, and a name that nothing survives of becomes "file".
func Transition ¶
Transition returns nil when the change from one status to another is allowed and a TransitionError otherwise.
func ValidateName ¶
ValidateName reports whether name may be a directory or file name. A name is non-empty, valid UTF-8, at most MaxNameLength runes, contains no slash and no control character, and is neither "." nor "..". The rejection is a NameError. Validate the normalized form: the check counts runes, and normalization can change the count.
Types ¶
type DeletingError ¶ added in v0.3.0
DeletingError reports a mutation refused with ErrDeleting and says whose delete refused it. Directory is false only for a file whose own delete began, its row deleting while its directory is active, and ID then names the file. Otherwise Directory is true and ID names the deleting directory: the file's own directory, for a file deleting because its branch was marked or whose directory is gone, or the directory the mutation reached. The deleting row is either the one the mutation acts on or the one that holds the name a create or a move asked for; the persistence package wraps the second kind of refusal in a message that names the row as the name's holder. Err is the refusal's cause when it has one, such as the TransitionError of a completion refused from deleting, and is nil otherwise. It matches ErrDeleting under errors.Is, and Unwrap yields Err, so errors.Is and errors.As reach the cause and its sentinels.
func (*DeletingError) Error ¶ added in v0.3.0
func (e *DeletingError) Error() string
func (*DeletingError) Is ¶ added in v0.3.0
func (e *DeletingError) Is(target error) bool
Is reports whether target is ErrDeleting.
func (*DeletingError) Unwrap ¶ added in v0.3.0
func (e *DeletingError) Unwrap() error
Unwrap returns the cause, or nil when there is none.
type Directory ¶
type Directory struct {
ID string `json:"id"`
ParentID *string `json:"parent_id"`
Name string `json:"name"`
Status DirectoryStatus `json:"status"`
Version int64 `json:"version"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
Directory is one node of the directory hierarchy, a row of blobfs_directory. The root has a nil ParentID and the Name "/", and it is the only row with either; every other directory has a ParentID and a Name that ValidateName accepts, which never contains a slash. Status is active until the delete of a branch marks the directory deleting. Version is the concurrency token the guarded commands check. The json tags are the scan and binding contract: the columns carry the same names, and a nil pointer binds or scans as NULL.
type DirectoryStatus ¶ added in v0.2.0
type DirectoryStatus string
DirectoryStatus is a directory row's place in the delete of a branch: every directory is active until the branch it sits in is marked deleting, and a mark is never undone. Its underlying type is string, so database/sql binds and scans it as the status column's text without a Valuer or Scanner, as Status does.
const ( // DirectoryStatusActive marks a directory that accepts children, files, // and moves: every directory, the root included, until a mark. DirectoryStatusActive DirectoryStatus = "active" // DirectoryStatusDeleting marks a directory whose branch is being // removed. It accepts no new child and no new file, and neither it nor // anything in it moves; the rows keep their names until they are // removed. DirectoryStatusDeleting DirectoryStatus = "deleting" )
func (DirectoryStatus) Mutable ¶ added in v0.2.0
func (s DirectoryStatus) Mutable() bool
Mutable reports whether a directory in status s accepts a create or a move beneath it, and a move of itself or of anything in it. A deleting directory refuses them all, so nothing enters or leaves a branch whose objects are being removed.
func (DirectoryStatus) String ¶ added in v0.2.0
func (s DirectoryStatus) String() string
func (DirectoryStatus) Valid ¶ added in v0.2.0
func (s DirectoryStatus) Valid() bool
Valid reports whether s is one of the two directory statuses.
type File ¶
type File struct {
ID string `json:"id"`
DirectoryID string `json:"directory_id"`
Name string `json:"name"`
Status Status `json:"status"`
Key string `json:"key"`
Size *int64 `json:"size"`
ContentType string `json:"content_type"`
ETag *string `json:"etag"`
Version int64 `json:"version"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
File is one file's metadata, a row of blobfs_file. DirectoryID is never nil: every file sits in a directory. Key is the object's key in the store, built once by NewKey when the pending row is inserted and never parsed back. Size and ETag are nil until the row is available, because the store reports them only once the object exists; ContentType is the media type the consumer declares at upload, so it is known from the pending row on, and completing the write replaces it with the type the store reports. Version is the concurrency token the guarded commands check. The json tags are the scan and binding contract: the columns carry the same names.
type IDError ¶
IDError reports a row id that ParseID refused, with the reason. It matches ErrInvalidID under errors.Is.
type KeyError ¶
KeyError reports a key the store refused, with the store's reason. It matches ErrInvalidKey under errors.Is, and errors.As reaches the store's own error through Unwrap.
type KeyValidator ¶
type KeyValidator interface {
// ValidateKey returns a non-nil error that says why when the store
// refuses key.
ValidateKey(key string) error
}
KeyValidator is the one question a write asks of the object store: whether it accepts a key. A consumer wires its store's key validation to this interface at its composition root, so blobfs validates a key before it stores the key in a pending row and never imports the store's package.
The interface carries no maximum key length. The store's own validation enforces its limit, counted the way the store counts, so blobfs keeps no second length check of its own. A key of the longest name is 292 runes, within Azure Blob Storage's 1024 characters; S3 counts its 1024 in bytes of UTF-8, and a key of 255 four-byte runes is 1057 bytes, which the store's validation refuses as a KeyError before any row is written.
type NameError ¶
NameError reports a name that ValidateName refused, with the reason. It matches ErrInvalidName under errors.Is.
type Object ¶
Object is what the store reports about a stored object once a write has completed: the facts go-storage's Object carries that a file row keeps. The consumer builds one from the store's answer to its put and hands it to the step that completes the write, so the row records the size the store counted, the entity tag it assigned, and the content type it holds.
type Status ¶
type Status string
Status is a file row's place in the two-phase write and delete: the row is inserted pending before the object is written, becomes available once the object exists, and is marked deleting before the object is removed. Its underlying type is string, so database/sql binds and scans it as the status column's text without a Valuer or Scanner.
const ( // StatusPending marks a row whose object has not been written yet. A // crash after the insert leaves the row pending, where a query finds it. StatusPending Status = "pending" // StatusAvailable marks a row whose object exists in the store. StatusAvailable Status = "available" // StatusDeleting marks a row whose object is being removed. The row // keeps its name until the purge removes it. StatusDeleting Status = "deleting" )
type TransitionError ¶
TransitionError reports a refused status change. It matches ErrInvalidTransition under errors.Is, and also ErrDeleting when the row was deleting, so a caller can tell a delete in progress from any other refusal without inspecting the fields. Package data reports a completion refused from deleting as a DeletingError whose cause is the TransitionError, so errors.As reaches either.
func (*TransitionError) Error ¶
func (e *TransitionError) Error() string
func (*TransitionError) Is ¶
func (e *TransitionError) Is(target error) bool
Is reports whether e stands for target: ErrInvalidTransition always, and ErrDeleting when the refused change started from a deleting row.
type ViolationError ¶
ViolationError reports a database constraint violation that a classifier mapped to a sentinel: the sentinel it means, the name of the violated constraint, and the error the database reported as the cause. Package data builds one for each constraint blobfs owns, and a consumer builds one for its own constraints with its own sentinels. The message prints the sentinel and the constraint name and never the driver's text. Unwrap yields the sentinel and the cause, so errors.Is matches the sentinel and errors.As reaches the sqlate.ConstraintError beneath, for a caller that imports sqlate and wants the class.
func (*ViolationError) Error ¶
func (e *ViolationError) Error() string
func (*ViolationError) Unwrap ¶
func (e *ViolationError) Unwrap() []error
Unwrap returns the sentinel and the cause, so errors.Is matches the sentinel and errors.As finds the cause's types.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package data is blobfs's persistence package: its standard-tier SQL statements, the pattern namespace it publishes, and the Store, whose Directories and Files handles run the statements through sqlate on any engine sqlate has a dialect for.
|
Package data is blobfs's persistence package: its standard-tier SQL statements, the pattern namespace it publishes, and the Store, whose Directories and Files handles run the statements through sqlate on any engine sqlate has a dialect for. |
|
datatest
Package datatest is the conformance suite of the data package: the checks every data.Store must pass against a live database, whatever the engine, the returning commands' form, and the variant.
|
Package datatest is the conformance suite of the data package: the checks every data.Store must pass against a live database, whatever the engine, the returning commands' form, and the variant. |
|
postgres
module
|