backend

README

Backend implementation notes

Store Concurrency

• The backend assumes at most one backend is running per-store-directory.
• The backend generally assumes that no other process will write to the store directory. (However, it is a bit more defensive about testing this assumption: if a store object is obviously missing or corrupted, it will complain.)
• The backend assumes that if and only if a store object is present in the store directory will a corresponding row exist in the objects table.
• The inProgress lock for a store path is acquired before accessing any store object. The lock is held while importing or accessing a store object. During access, the lock is released once it has been determined that the object exists. (Code may assume that if a store object exists at this time, it is fully constructed because of the previous bullet.) During import of a store object, the lock is released once it has been written to the filesystem and the row has been written to the objects table.

Documentation

Overview

Package backend provides a zbstorerpc implementation backed by local compute resources.

Index

• Constants
• func CanSandbox() bool
• func SystemSupportsSandbox() bool
• func WithExporter(parent context.Context, e Exporter) context.Context
• type BuildUser
  • func (user BuildUser) String() string
• type Exporter
• type NARReceiver
  • func (r *NARReceiver) Cleanup(ctx context.Context)
  • func (r *NARReceiver) ReceiveNAR(trailer *zbstore.ExportTrailer)
  • func (r *NARReceiver) Write(p []byte) (n int, err error)
• type ObjectInfo
  • func NewObjectInfo(path zbstore.Path, info *zbstorerpc.ObjectInfo) *ObjectInfo
  • func (info *ObjectInfo) AppendText(dst []byte) ([]byte, error)
  • func (info *ObjectInfo) MarshalText() ([]byte, error)
  • func (info *ObjectInfo) ToExportTrailer() *zbstore.ExportTrailer
  • func (info *ObjectInfo) ToRPC() *zbstorerpc.ObjectInfo
  • func (info *ObjectInfo) UnmarshalText(src []byte) (err error)
• type Options
• type SandboxPath
• type Server
  • func NewServer(dir zbstore.Directory, dbPath string, opts *Options) *Server
  • func (s *Server) Close() error
  • func (s *Server) Delete(ctx context.Context, paths sets.Set[zbstore.Path]) error
  • func (s *Server) DeleteIncludingReferences(ctx context.Context, paths sets.Set[zbstore.Path]) error
  • func (s *Server) Export(ctx context.Context, dst io.Writer, req *zbstorerpc.ExportRequest) error
  • func (s *Server) JSONRPC(ctx context.Context, req *jsonrpc.Request) (*jsonrpc.Response, error)
  • func (s *Server) NewNARReceiver(ctx context.Context, bufCreator bytebuffer.Creator) *NARReceiver
  • func (s *Server) RecentBuildIDs(ctx context.Context, limit int) ([]string, error)
  • func (s *Server) Register(ctx context.Context, info *ObjectInfo) error

Constants

const DefaultBuildUsersGroup = "zbld"

DefaultBuildUsersGroup is the conventional name of the Unix group for the users that execute builders on behalf of the daemon.

Functions

func CanSandbox

func CanSandbox() bool

CanSandbox reports whether the current execution environment supports sandboxing.

func SystemSupportsSandbox

func SystemSupportsSandbox() bool

SystemSupportsSandbox reports whether the host operating system supports sandboxing.

func WithExporter

func WithExporter(parent context.Context, e Exporter) context.Context

WithExporter returns a copy of parent in which the given exporter is used to send back export information.

Types

type BuildUser

type BuildUser struct {
	// UID is the user ID.
	UID int
	// GID is the user's primary group ID.
	GID int
}

BuildUser is a descriptor for a Unix user.

func (BuildUser) String

func (user BuildUser) String() string

type Exporter

type Exporter interface {
	Export(header jsonrpc.Header, r io.Reader) error
}

A type that implements Exporter can receive a `nix-store --export` formatted stream.

type NARReceiver

type NARReceiver struct {
	// contains filtered or unexported fields
}

NARReceiver is a per-connection zbstore.NARReceiver.

func (*NARReceiver) Cleanup

func (r *NARReceiver) Cleanup(ctx context.Context)

Cleanup releases any resources associated with the receiver.

func (*NARReceiver) ReceiveNAR

func (r *NARReceiver) ReceiveNAR(trailer *zbstore.ExportTrailer)

func (*NARReceiver) Write

func (r *NARReceiver) Write(p []byte) (n int, err error)

type ObjectInfo

type ObjectInfo struct {
	// StorePath is the absolute path of this store object
	// (e.g. "/opt/zb/store/s66mzxpvicwk07gjbjfw9izjfa797vsw-hello-2.12.1").
	StorePath zbstore.Path
	// NARHash is the hash of the store object as an uncompressed .nar file.
	NARHash nix.Hash
	// NARSize is the size of the decompressed .nar file in bytes.
	NARSize int64
	// References is the set of store objects that this store object references.
	References sets.Sorted[zbstore.Path]
	// CA is a content-addressability assertion.
	CA zbstore.ContentAddress
}

ObjectInfo is the full metadata of an object in the backend. Conceptually, it is a tuple of a zbstore.Path and a zbstorerpc.ObjectInfo. However, it is is more suitable as an in-memory representation.

func NewObjectInfo

func NewObjectInfo(path zbstore.Path, info *zbstorerpc.ObjectInfo) *ObjectInfo

NewObjectInfo constructs a new ObjectInfo from a zbstore.Path and a zbstorerpc.ObjectInfo.

func (*ObjectInfo) AppendText

func (info *ObjectInfo) AppendText(dst []byte) ([]byte, error)

AppendText implements encoding.TextAppender by appending a condensed version of a .narinfo file to dst. Any zero values are omitted except for store path.

func (*ObjectInfo) MarshalText

func (info *ObjectInfo) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler by calling info.AppendText(nil).

func (*ObjectInfo) ToExportTrailer

func (info *ObjectInfo) ToExportTrailer() *zbstore.ExportTrailer

ToExportTrailer converts info to a *zbstore.ExportTrailer value.

func (*ObjectInfo) ToRPC

func (info *ObjectInfo) ToRPC() *zbstorerpc.ObjectInfo

ToRPC converts info to a *zbstorerpc.ObjectInfo value.

func (*ObjectInfo) UnmarshalText

func (info *ObjectInfo) UnmarshalText(src []byte) (err error)

UnmarshalText implements encoding.TextUnmarshaler by parsing a .narinfo file format. Unrecognized keys are ignored.

type Options

type Options struct {
	// RealStoreDirectory is where the store objects are located physically on disk.
	// If empty, defaults to the store directory.
	RealStoreDirectory string
	// BuildDirectory is where realizations' working directories will be placed.
	// If empty, defaults to [os.TempDir].
	BuildDirectory string
	// LogDirectory is where builder logs will be stored.
	// If empty, defaults to a directory called "log" in the same directory as the database.
	LogDirectory string
	// ContentAddressBufferCreator is used to create buffers for content addressing analysis.
	// If nil, then in-memory byte slices are used with reasonable limits.
	ContentAddressBufferCreator bytebuffer.Creator

	// DatabasePoolSize is the maximum permitted number of concurrent connections to the database.
	// If less than 1, a reasonable default is used.
	DatabasePoolSize int

	// If AllowKeepFailed is true, then the KeepFailed field in [zbstore.RealizeRequest] will be respected.
	AllowKeepFailed bool

	// If DisableSandbox is true, then builders are always run without the sandbox.
	// Otherwise, sandboxing is used whenever possible.
	DisableSandbox bool
	// SandboxPaths is a map of paths inside the sandbox
	// to paths on the host machine.
	// These paths will be made available to sandboxed builders.
	SandboxPaths map[string]SandboxPath

	// CoresPerBuild is a hint from the user to builders
	// on the number of concurrent jobs to perform.
	// If non-positive, then the number of cores detected on the machine is used.
	CoresPerBuild int

	// BuildUsers is the set of user IDs to use for builds on non-Windows systems.
	// If empty, then builds will use the current process's privileges.
	// [NewServer] will panic if multiple entries have the same user ID.
	BuildUsers []BuildUser

	// BuildContext optionally specifies a function that detaches the context for a build.
	// If BuildContext is nil, the default is [context.Background].
	BuildContext func(parent context.Context, buildID string) context.Context

	// BuildLogRetention is the length of time to retain build logs.
	// If non-positive, then build logs will be not be automatically deleted.
	BuildLogRetention time.Duration
}

Options is the set of optional parameters to NewServer.

type SandboxPath

type SandboxPath struct {
	// Path is the path on the backend's filesystem to make available at the path.
	// If empty, the SandboxPaths key is used.
	Path string
	// If AlwaysPresent is true, then the path will always be made available in the sandbox.
	// The default is to only allow the path to be used if it is declared in __buildSystemDeps.
	AlwaysPresent bool
}

A SandboxPath is the set of options for SandboxPaths in Options.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is a local store. Server implements jsonrpc.Handler and is intended to be used with jsonrpc.Serve.

func NewServer

func NewServer(dir zbstore.Directory, dbPath string, opts *Options) *Server

NewServer returns a new Server for the given store directory and database path. Callers are responsible for calling Server.Close on the returned server.

func (*Server) Close

func (s *Server) Close() error

Close releases any resources associated with the server.

func (*Server) Delete

func (s *Server) Delete(ctx context.Context, paths sets.Set[zbstore.Path]) error

Delete deletes the set of store paths. Delete will return an error if any of the named paths do not exist or there are store objects beyond those named that refer to the named store objects.

func (*Server) DeleteIncludingReferences

func (s *Server) DeleteIncludingReferences(ctx context.Context, paths sets.Set[zbstore.Path]) error

DeleteIncludingReferences is the same as *Server.Delete, but if the store objects have other store objects referring to them, then they will be deleted as well.

func (*Server) Export

func (s *Server) Export(ctx context.Context, dst io.Writer, req *zbstorerpc.ExportRequest) error

Export exports the store objects according to the request in `nix-store --export` format to dst.

func (*Server) JSONRPC

func (s *Server) JSONRPC(ctx context.Context, req *jsonrpc.Request) (*jsonrpc.Response, error)

JSONRPC implements the jsonrpc.Handler interface and serves the zbstorerpc API.

func (*Server) NewNARReceiver

func (s *Server) NewNARReceiver(ctx context.Context, bufCreator bytebuffer.Creator) *NARReceiver

NewNARReceiver returns a new NARReceiver that is attached to the server. Callers are responsible for calling NARReceiver.Cleanup after the receiver is no longer in use.

func (*Server) RecentBuildIDs

func (s *Server) RecentBuildIDs(ctx context.Context, limit int) ([]string, error)

RecentBuildIDs returns the most recent builds started or finished. It returns at most limit values.

func (*Server) Register

func (s *Server) Register(ctx context.Context, info *ObjectInfo) error

Register adds the info for a store object that is already present in the store directory to the store's database. If the store path is already in the database and the information matches what is already there, then Register is a no-op and returns nil.