core

package
v0.0.0-...-eb887e7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 10, 2024 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package core provides the internal Ferry implementation.

This portion of ferryd is responsible for the management of the repositories, and receives packages from the builders.

Index

Constants

View Source 
const (
	// DatabasePathComponent is the suffix applied to a working directory
	// for the database file itself.
	DatabasePathComponent = "ferry_db"

	// JobDbPathComponent points ot the jobs database
	JobDbPathComponent = "ferry_db_jobs"

	// IncomingPathComponent is the base for all per-repo incoming directories
	IncomingPathComponent = "incoming"

	// Version of the ferry client library
	Version = "0.0.0"
)
View Source 
const (
	// DatabaseBucketPool is the identifier for the pool main bucket
	DatabaseBucketPool = "pool"

	// DatabaseBucketDeltaSkip is the identifier for the pool's "failed delta" entries
	DatabaseBucketDeltaSkip = "deltaSkip"

	// PoolPathComponent is the storage directory for all of our main files
	PoolPathComponent = "pool"

	// PoolSchemaVersion is the current schema version for a PoolEntry
	PoolSchemaVersion = "1.0"
)
View Source 
const (
	// RepoPathComponent is the base for all repository directories
	RepoPathComponent = "repo"

	// AssetPathComponent is where we'll find extra files like distribution.xml
	AssetPathComponent = "assets"

	// DeltaPathComponent is a temporary tree for creating delta packages
	DeltaPathComponent = "deltaBuilds"

	// DeltaStagePathComponent is where we put temporary deltas until merged
	DeltaStagePathComponent = "deltaStaging"

	// DatabaseBucketRepo is the name for the main repo toplevel bucket
	DatabaseBucketRepo = "repo"

	// DatabaseBucketPackage is the path to the subbucket within a repo bucket
	DatabaseBucketPackage = "package"

	// RepoSchemaVersion is the current schema version for a RepoEntry
	RepoSchemaVersion = "1.0"
)
View Source 
const FerrydDir = "/etc/ferryd"
View Source 
const (
	// TransitManifestSuffix is the extension that a valid transit manifest must have
	TransitManifestSuffix = ".tram"
)

Variables

View Source 
var (
	// ErrInvalidHeader will be returned when the [manifest] section is malformed
	ErrInvalidHeader = errors.New("Manifest contains an invalid header")

	// ErrMissingTarget will be returned when the target is not present
	ErrMissingTarget = errors.New("Manifest contains no target")

	// ErrMissingPayload will be returned when the [[file]]s are missing
	ErrMissingPayload = errors.New("Manifest does not contain a payload")

	// ErrInvalidPayload will be returned when the payload is in some way invalid
	ErrInvalidPayload = errors.New("Manifest contains an invalid payload")

	// ErrIllegalUpload is returned when someone is a spanner and tries uploading an unsupported file
	ErrIllegalUpload = errors.New("The manifest file is NOT an eopkg")
)

Functions

func CopyFile 

func CopyFile(source, dest string) error

CopyFile will copy the file and permissions to the new target

func FileSha1sum 

func FileSha1sum(path string) (string, error)

FileSha1sum is a quick wrapper to grab the sha1sum for the given file

func FileSha256sum 

func FileSha256sum(path string) (string, error)

FileSha256sum is a quick wrapper to grab the sha256sum for the given file

func LinkOrCopyFile 

func LinkOrCopyFile(source, dest string, forceCopy bool) error

LinkOrCopyFile is a helper which will initially try to hard link, however if we hit an error (because we tried a cross-filesystem hardlink) we'll try to copy instead.

func PathExists 

func PathExists(path string) bool

PathExists is a trivial helper to figure out if a path exists or not

func ProduceDelta 

func ProduceDelta(tmpDir, oldPackage, newPackage, targetPath string) error

ProduceDelta will attempt to batch the delta production between the two listed file paths and then copy it into the final targetPath

func RemovePackageParents 

func RemovePackageParents(path string) error

RemovePackageParents will try to remove the leading components of a package file, only if they are empty.

func WriteSha1sum 

func WriteSha1sum(inpPath, outPath string) error

WriteSha1sum will take the sha1sum of the input path and then dump it to the given output path

func WriteSha256sum 

func WriteSha256sum(inpPath, outPath string) error

WriteSha256sum will take the sha256sum of the input path and then dump it to the given output path

Types

type Component 

type Component interface {

	// Initialise the component on the initial transaction
	Init(ctx *Context, db libdb.Database) error

	// Close will request the component stops any ongoing operations and cleanup
	Close()
}

A Component of ferryd has special considerations to bootstrap itself during ferryd start, and clean up during ferryd shutdown.

type Context 

type Context struct {
	BaseDir   string // Base directory of operations
	DbPath    string // Path to the main database file
	JobDbPath string // Path to the job database file
}

The Context is shared between all of the components of ferryd to provide working directories and such.

func NewContext 

func NewContext(root string) (*Context, error)

NewContext will construct a context from the given base directory for all file path functions

type DeltaInformation 

type DeltaInformation struct {
	FromRelease int    // The source release for this delta
	FromID      string // ID for the source package
	ToRelease   int    // The target release for this delta
	ToID        string // ID for the target package
}

DeltaInformation is included in pool entries if they're actually a delta package and not a normal package

type DeltaSkipEntry 

type DeltaSkipEntry struct {
	SchemaVersion string // Version used when this skip entry was created
	Name          string
	Delta         DeltaInformation
}

A DeltaSkipEntry is used to record skipped deltas from some kind of generation failure

type Manager 

type Manager struct {
	IncomingPath string // Incoming directory
	// contains filtered or unexported fields
}

A Manager is the the singleton responsible for slip management

func NewManager 

func NewManager(path string) (*Manager, error)

NewManager will attempt to instaniate a manager for the given path, which will yield an error if the database cannot be opened for access.

func (*Manager) AddDelta 

func (m *Manager) AddDelta(repoID, deltaPath string, mapping *DeltaInformation) error

AddDelta will attempt to include the delta package specified by deltaPath into the target repository

func (*Manager) AddPackages 

func (m *Manager) AddPackages(repoID string, packages []string, anal bool) error

AddPackages will attempt to add the named packages to the repository

func (*Manager) CloneRepo 

func (m *Manager) CloneRepo(repoID, newClone string, fullClone bool) error

CloneRepo will initially construct a new repository, and then ask that it copy itself from an existing repo

If fullClone is set, all packages are copied. Otherwise only the tip for each package is taken.

func (*Manager) Close 

func (m *Manager) Close()

Close will close and clean up any associated resources, such as the underlying database.

func (*Manager) CopySource 

func (m *Manager) CopySource(repoID, target, sourceID string, release int) error

CopySource will ask the repo to copy all matching source==release packages

func (*Manager) CreateDelta 

func (m *Manager) CreateDelta(repoID string, oldPkg, newPkg *libeopkg.MetaPackage) (string, error)

CreateDelta will attempt to create a new delta package between the old and new IDs

func (*Manager) CreateRepo 

func (m *Manager) CreateRepo(id string) error

CreateRepo will request the creation of a new repository

func (*Manager) DeleteRepo 

func (m *Manager) DeleteRepo(id string) error

DeleteRepo exposes the API for repository deletion

func (*Manager) GetDeltaFailed 

func (m *Manager) GetDeltaFailed(deltaID string) bool

GetDeltaFailed will determine via the pool transaction whether a delta has previously failed.

func (*Manager) GetPackageNames 

func (m *Manager) GetPackageNames(repoID string) ([]string, error)

GetPackageNames will attempt to load all package names for the given repository.

func (*Manager) GetPackages 

func (m *Manager) GetPackages(repoID, pkgName string) ([]*libeopkg.MetaPackage, error)

GetPackages will return a set of packages for the package name within the specified repository

func (*Manager) GetPoolEntry 

func (m *Manager) GetPoolEntry(pkgID string) (*libeopkg.MetaPackage, error)

GetPoolEntry will return the metadata for a pool entry with the given pkg ID

func (*Manager) GetPoolItems 

func (m *Manager) GetPoolItems() ([]*PoolEntry, error)

GetPoolItems will return all known pool items

func (*Manager) GetRepo 

func (m *Manager) GetRepo(id string) (*Repository, error)

GetRepo will grab the repository if it exists Note that this is a read only operation

func (*Manager) GetRepos 

func (m *Manager) GetRepos() ([]*Repository, error)

GetRepos will return all known repositories

func (*Manager) HasDelta 

func (m *Manager) HasDelta(repoID, pkgID, deltaPath string) (bool, error)

HasDelta will query the repository to determine if it already has the given delta

func (*Manager) Index 

func (m *Manager) Index(repoID string) error

Index will cause the repository's index to be reconstructed

func (*Manager) MarkDeltaFailed 

func (m *Manager) MarkDeltaFailed(deltaID string, delta *DeltaInformation) error

MarkDeltaFailed will permanently record the delta package as failing so we do not attempt to recreate it (expensive)

func (*Manager) PullRepo 

func (m *Manager) PullRepo(sourceID, targetID string) ([]string, error)

PullRepo will pull from one repo, the source ID, into the target repository

func (*Manager) RefDelta 

func (m *Manager) RefDelta(repoID, deltaID string) error

RefDelta will dupe an existing delta into the target repository

func (*Manager) RemoveSource 

func (m *Manager) RemoveSource(repoID, sourceID string, release int) error

RemoveSource will ask the repo to remove all matching source==release packages.

func (*Manager) TrimObsolete 

func (m *Manager) TrimObsolete(repoID string) error

TrimObsolete will ask the repo to remove obsolete packages

func (*Manager) TrimPackages 

func (m *Manager) TrimPackages(repoID string, maxKeep int) error

TrimPackages will ask the repo to remove excessive packages

type MmapFile 

type MmapFile struct {
	Data []byte
	// contains filtered or unexported fields
}

An MmapFile is used to easily wrap the syscall mmap() functions to be readily usable from golang. This helps heaps (pun intended) when it comes to computing the hash sum for very large files, such as the index and eopkg files, in a zero copy fashion.

func MapFile 

func MapFile(path string) (*MmapFile, error)

MapFile will attempt to mmap() the input file

func (*MmapFile) Close 

func (m *MmapFile) Close() error

Close will close the previously mmapped filed

type Pool 

type Pool struct {
	// contains filtered or unexported fields
}

A Pool is used to manage and deduplicate resources between multiple resources, and represents the real backing store for referenced eopkg files.

func (*Pool) AddDelta 

func (p *Pool) AddDelta(db libdb.Database, pkg *libeopkg.Package, mapping *DeltaInformation, copyDisk bool) (*PoolEntry, error)

AddDelta will add a delta package to the pool if doesn't exist, otherwise it will increase the refcount for the package.

This is a very loose wrapper around AddPackage, but will add some delta information too. Note that a delta package is still a package in its own right, its just installed and handled differently (lacking files, etc.)

func (*Pool) AddPackage 

func (p *Pool) AddPackage(db libdb.Database, pkg *libeopkg.Package, copy bool) (*PoolEntry, error)

AddPackage will determine where the new eopkg goes, and whether we need to actually push it on disk, or simply bump the ref count. Any file passed to us is believed to be under our ownership now.

func (*Pool) Close 

func (p *Pool) Close()

Close doesn't currently do anything

func (*Pool) GetDeltaFailed 

func (p *Pool) GetDeltaFailed(db libdb.Database, id string) bool

GetDeltaFailed will determine if generation of this delta ID has actually failed in the past, skipping a potentially expensive delta examination.

func (*Pool) GetEntry 

func (p *Pool) GetEntry(db libdb.Database, id string) (*PoolEntry, error)

GetEntry will return the package entry for the given ID

func (*Pool) GetMetaPoolPath 

func (p *Pool) GetMetaPoolPath(id string, meta *libeopkg.MetaPackage) string

GetMetaPoolPath will return the internal path for a given meta package

func (*Pool) GetPackagePoolPath 

func (p *Pool) GetPackagePoolPath(pkg *libeopkg.Package) string

GetPackagePoolPath Convenience function to grab the target for the given package within the current pool

func (*Pool) GetPoolItems 

func (p *Pool) GetPoolItems(db libdb.Database) ([]*PoolEntry, error)

GetPoolItems will return a copy of the pool entries in our database

func (*Pool) GetSkipEntry 

func (p *Pool) GetSkipEntry(db libdb.Database, id string) (*DeltaSkipEntry, error)

GetSkipEntry will return the delta-skip entry for the given ID

func (*Pool) Init 

func (p *Pool) Init(ctx *Context, db libdb.Database) error

Init will create our initial working paths and DB bucket

func (*Pool) MarkDeltaFailed 

func (p *Pool) MarkDeltaFailed(db libdb.Database, id string, delta *DeltaInformation) error

MarkDeltaFailed will insert a record indicating that it is not possible to actually produce a given delta ID

func (*Pool) RefEntry 

func (p *Pool) RefEntry(db libdb.Database, id string) error

RefEntry will include the given eopkg if it doesn't yet exist, otherwise it will simply increase the ref count by 1.

func (*Pool) UnrefEntry 

func (p *Pool) UnrefEntry(db libdb.Database, id string) error

UnrefEntry will unref a given ID from the repository. Should the refcount hit 0, the package will then be removed from the pool storage.

type PoolEntry 

type PoolEntry struct {
	SchemaVersion string                // Version used when this pool entry was created
	Name          string                // Name&ID of the pool entry
	RefCount      uint64                // How many instances of this file exist right now
	Meta          *libeopkg.MetaPackage // The eopkg metadata
	Delta         *DeltaInformation     // May actually be nil if not a delta
}

A PoolEntry is the main storage unit within ferryd. Each entry contains the full data for a given eopkg file, as well as the reference count.

When the refcount hits 0, files are then purge from the pool and freed from disk. When adding a pool item to a repository, the ref count is increased, and the file is then hard-linked into place, saving on disk storage.

type RepoEntry 

type RepoEntry struct {
	SchemaVersion string   // Version used when this repo entry was created
	Name          string   // Base package name
	Available     []string // The available packages for this package name (eopkg IDs)
	Published     string   // The "tip" version of this package (eopkg ID)
	Deltas        []string // Delta packages known for this package.
}

RepoEntry is the basic repository storage unit, and details what packages are exported in the index.

type Repository 

type Repository struct {
	ID string // Name of this repository (unique)
	// contains filtered or unexported fields
}

A Repository is a simplistic representation of an exported repository within ferryd

func (*Repository) AddDelta 

func (r *Repository) AddDelta(db libdb.Database, pool *Pool, filename string, mapping *DeltaInformation) error

AddDelta will first open and read the .delta.eopkg, before passing it back off to AddLocalDelta

func (*Repository) AddLocalDelta 

func (r *Repository) AddLocalDelta(db libdb.Database, pool *Pool, pkg *libeopkg.Package, mapping *DeltaInformation) error

AddLocalDelta will attempt to add the delta to this repository, if possible All ref'd deltas are retained, but not necessarily emitted unless they're valid for the from-to relationship.

func (*Repository) AddLocalPackage 

func (r *Repository) AddLocalPackage(db libdb.Database, pool *Pool, pkg *libeopkg.Package) error

AddLocalPackage will do the real work of adding an open & loaded eopkg to the repository

func (*Repository) AddPackage 

func (r *Repository) AddPackage(db libdb.Database, pool *Pool, filename string, anal bool) error

AddPackage will attempt to load the local package and then add it to the repository via AddLocalPackage

func (*Repository) CloneFrom 

func (r *Repository) CloneFrom(db libdb.Database, pool *Pool, sourceRepo *Repository, fullClone bool) error

CloneFrom will attempt to clone everything from the target repository into ourselves

func (*Repository) CopySourceFrom 

func (r *Repository) CopySourceFrom(db libdb.Database, pool *Pool, sourceRepo *Repository, sourceID string, release int) error

CopySourceFrom will find all records within sourceRepo that have both the specified sourceID and release number.

func (*Repository) CreateDelta 

func (r *Repository) CreateDelta(db libdb.Database, oldPkg, newPkg *libeopkg.MetaPackage) (string, error)

CreateDelta is responsible for trying to create a new delta package between oldPkg and newPkg, with newPkg being the delta *to*.

This function may fail to produce a delta because they're incompatible packages, or because a delta between the two packages would be pointless (i.e. they're either identical or 100% the same.)

Lastly, this function will move the delta out of the build area into the staging area if it successfully produces a delta. This does not mark a delta attempt as "pointless", nor does it actually *include* the delta package within the repository.

func (*Repository) GetEntry 

func (r *Repository) GetEntry(db libdb.Database, id string) (*RepoEntry, error)

GetEntry will return the package entry for the given ID

func (*Repository) GetPackageNames 

func (r *Repository) GetPackageNames(db libdb.Database) ([]string, error)

GetPackageNames will traverse the buckets and find all package names as stored within the DB. This doesn't account for obsolete names, which should in fact be removed from the repo entirely.

func (*Repository) GetPackages 

func (r *Repository) GetPackages(db libdb.Database, pool *Pool, pkgName string) ([]*libeopkg.MetaPackage, error)

GetPackages will return all package objects for a given name

func (*Repository) HasDelta 

func (r *Repository) HasDelta(db libdb.Database, pkgName, deltaPath string) (bool, error)

HasDelta will work out if we actually have a delta already

func (*Repository) Index 

func (r *Repository) Index(db libdb.Database, pool *Pool) error

Index will attempt to write the eopkg index out to disk This only requires a read-only database view

func (*Repository) PullFrom 

func (r *Repository) PullFrom(db libdb.Database, pool *Pool, sourceRepo *Repository) ([]string, error)

PullFrom will iterate the source repositories contents, looking for any packages we can pull into ourselves.

If a package is missing from our own indexes (i.e. no key) we'll pull that package. If a package is present in our own indexes, but the package in sourceRepo's published field is actually _newer_ than ours, we'll pull that guy in too.

Note this isn't "pull" in the git sense, as we'll not perform any removals or attempt to sync the states completely. Pull is typically used on a clone from a volatile target, i.e. pulling from unstable into stable. As many rebuilds might happen in unstable, we actually only want to bring in the *tip* from the new sources.

Over time, a clone will drift in content from the source. In terms of upgrade paths for users, they'll only follow *one* repository, i.e. the stable rolling snapshot channel. To be able to provide usable and effective delta paths for those users, we really need to form deltas per repository between their effective tip and "old" items, otherwise with a git-like sync the users might never see any useful delta packages for the fast moving items.

Drift can be corrected by nuking a repository and performing a full clone from the source to have identical mirrors again. This should be performed rarely and only during periods of maintenance due to this method violating atomic indexes.

func (*Repository) RefDelta 

func (r *Repository) RefDelta(db libdb.Database, pool *Pool, deltaID string) error

RefDelta will take the existing delta from the pool and insert it into our own repository

func (*Repository) RefPackage 

func (r *Repository) RefPackage(db libdb.Database, pool *Pool, pkgID string) error

RefPackage will dupe a package from the pool into our own storage

func (*Repository) RemoveSource 

func (r *Repository) RemoveSource(db libdb.Database, pool *Pool, sourceID string, release int) error

RemoveSource will remove all packages that have a matching source name and release number. This allows us to remove multiple packages from a single upload/set in one go.

Distributions tend to split packages across a common identifier/release and this method will allow us to remove "bad actors" from the index.

func (*Repository) TrimObsolete 

func (r *Repository) TrimObsolete(db libdb.Database, pool *Pool) error

TrimObsolete isn't very straight forward as it has to account for some janky behaviour in eopkg.

Effectively - any name explicitly marked as obsolete is something we need to remove from the repository. However, we also need to apply certain modifications to ensure child packages (-dbginfo) are also nuked along with them.

func (*Repository) TrimPackages 

func (r *Repository) TrimPackages(db libdb.Database, pool *Pool, maxKeep int) error

TrimPackages will trim back the packages in each package entry to a maximum amount of packages, which helps to combat the issue of rapidly inserting many builds into a repo, i.e. removing old backversions

func (*Repository) UnrefPackage 

func (r *Repository) UnrefPackage(db libdb.Database, pool *Pool, pkgID string) error

UnrefPackage will remove a package from our storage, and potentially remove the entire RepoEntry for the package if none are left.

Additionally, we'll locate stray deltas which lead either TO or FROM the given package as they'll now be useless to anyone.

type RepositoryManager 

type RepositoryManager struct {
	// contains filtered or unexported fields
}

The RepositoryManager maintains all repos within ferryd which are in turn linked to the main pool

func (*RepositoryManager) Close 

func (r *RepositoryManager) Close()

Close doesn't currently do anything

func (*RepositoryManager) CreateRepo 

func (r *RepositoryManager) CreateRepo(db libdb.Database, id string) (*Repository, error)

CreateRepo will create a new repository (bucket) within the top level repo bucket.

func (*RepositoryManager) DeleteRepo 

func (r *RepositoryManager) DeleteRepo(db libdb.Database, pool *Pool, id string) error

DeleteRepo is not yet implemented

func (*RepositoryManager) GetRepo 

func (r *RepositoryManager) GetRepo(db libdb.Database, id string) (*Repository, error)

GetRepo will attempt to get the named repo if it exists, otherwise return an error. This is a transactional helper to make the API simpler

func (*RepositoryManager) GetRepos 

func (r *RepositoryManager) GetRepos(db libdb.Database) ([]*Repository, error)

GetRepos will return a copy of the repositores in our database

func (*RepositoryManager) Init 

func (r *RepositoryManager) Init(ctx *Context, db libdb.Database) error

Init will create our initial working paths and DB bucket

type TransitManifest 

type TransitManifest struct {

	// Every .tram file has a [manifest] header - this will never change and is
	// version agnostic.
	Manifest TransitManifestHeader `toml:"manifest"`

	// A list of files that accompanied this .tram upload
	File []TransitManifestFile `toml:"file"`
	// contains filtered or unexported fields
}

A TransitManifest is provided by build servers to validate the upload of packages into the incoming directory.

This is to ensure all uploads are intentional, complete and verifiable.

func NewTransitManifest 

func NewTransitManifest(path string) (*TransitManifest, error)

NewTransitManifest will attempt to load the transit manifest from the named path and perform *basic* validation.

func (*TransitManifest) GetPaths 

func (t *TransitManifest) GetPaths() []string

GetPaths will return the package paths as a slice of strings

func (*TransitManifest) ID 

func (t *TransitManifest) ID() string

ID will return the unique ID for the transit manifest file

func (*TransitManifest) ValidatePayload 

func (t *TransitManifest) ValidatePayload() error

ValidatePayload will verify the files listed in the manifest locally, ensuring that they actually exist, and that the hashes match to prevent any corrupted uploads being inadvertently imported

type TransitManifestFile 

type TransitManifestFile struct {

	// Relative filename, i.e. nano-2.7.5-68-1-x86_64.eopkg
	Path string `toml:"path"`

	// Cryptographic checksum to allow integrity checks post-upload/pre-merge
	Sha256 string `toml:"sha256"`
}

TransitManifestFile provides simple verification data for each file in the uploaded payload.

type TransitManifestHeader 

type TransitManifestHeader struct {
	// Versioning to protect against future format changes
	Version string `toml:"version"`

	// The repo that the uploader is intending to upload *to*
	Target string `toml:"target"`
}

A TransitManifestHeader is required in all .tram uploads to ensure that both the sender and recipient are talking in the same fashion.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.