README
¶
imagecore
Image list and image cache management shared by the kutti drivers (driver-vbox, driver-hyperv and driver-qemu).
The three drivers used to carry near-identical copies of this code. imagecore holds the one copy, and fixes in it once. A driver describes itself with a Config, calls New, and declares type Image = imagecore.Image, so its exported API does not change.
Using it
store := imagecore.New(imagecore.Config{
DriverName: "driver-qemu",
ConfigFile: "driver-qemu-images.json",
ImageExtension: ".qcow2",
DownloadSuffix: ".qcow2download",
SourceURL: func() string { return ImagesSourceURL },
CacheDir: cacheDir,
InUse: isImageInUse, // optional
// Unpack: ..., // optional
})
err := store.UpdateImageList()
versions := store.K8sVersions()
image, err := store.GetImage("1.35")
err = image.Fetch()
New does no I/O, and calls none of the functions in the Config. A Store keeps nothing in memory: every call reads the list file afresh, so several kutti processes see each other's changes. A Store never changes after New, and is safe for use by several goroutines. An Image is a snapshot of one entry of the list, and is not; get a fresh one for each goroutine.
Store.ImagePath(version) returns where the cached file for a version is, or would be. A driver uses it, and not its own naming code, to find the image from which to create machines.
What the drivers differ in
Config |
vbox | hyperv | qemu |
|---|---|---|---|
ImageExtension |
.ova |
.vhdx |
.qcow2 |
DownloadSuffix |
.ovadownload |
.download.zip |
.qcow2download |
Unpack |
none | unzip a single-file zip | none |
InUse |
none | PowerShell, disk parent | qemu-img info |
CacheDir |
per user | per user | shared, /var/tmp/kutti/... |
The InUse function also switches on a short-circuit: an image that is already cached with the right checksum is not copied again. Without it, an image is always copied, which is right for vbox, as it copies the OVA when a machine is created.
Behaviour
The list. The local list is ConfigFile in the workspace config directory. A missing file is an empty list, and reading never writes. UpdateImageList downloads the master list, parses and checks it in memory, and only then replaces the local list. A downloaded list that has no images is rejected like an invalid one, so a bad download cannot empty the list. An image that was Downloaded stays so if the new list has the same checksum and source URL for it. Every other image is Notdownloaded, whatever the master list says.
Bad lists. A version must match ^[A-Za-z0-9][A-Za-z0-9._+-]*$, be at most 64 characters long, and be the key its entry is stored under, because versions go into file names. One invalid entry rejects the whole list. A downloaded list that is rejected is logged at kuttilog.Error, and the local list is kept as it was. A local list that cannot be used is reported as an error, and left alone; UpdateImageList is the way to repair it, and moves the damaged file aside as <name>.<timestamp>.corrupt. A list that cannot be read for any other reason is never replaced.
Fetching. An image is downloaded into a work directory inside the cache directory, unpacked if the driver needs it, checked against the checksum in the list, and copied into the cache as kutti-<version><ImageExtension>. Work directories are removed afterwards. A killed process leaves its work directory behind, and a later fetch of the same version removes ones that are over a day old.
Locks. Fetching, importing and purging the image for one version are serialized between processes of the same user, by <DriverName>-images-<version>.lock in the config directory. A process that waits for another's fetch of the same version does nothing more if the image has become Downloaded. The list is changed only under <DriverName>-images.lock, one image at a time, and only if the entry still has the checksum and source URL of the image, so processes do not undo each other's changes. Locks are per user. For qemu, whose cache is shared by all users of a machine, two users fetching at once are not serialized against each other.
Lists that change under an image. If the list is updated after an Image was obtained, and its entry has changed, Fetch and FromFile fail at once with a message to get the image again. If the list changes during the work, the file stays in the cache, the error says so, and the image is not marked Downloaded.
Why Config is a struct
The drivers differ in data, not in behaviour. Of the eight things that vary, four are constants, two are functions that must be evaluated late, and two are optional hooks. A struct says that directly.
- Adding a setting does not break anyone. A new field in a struct that drivers fill in by name changes nothing for them. A new method on an interface breaks every implementation, in three repositories that are released separately.
- Absent is
nil.UnpackandInUseare optional, and the behaviour that depends on them is switched by whether they are set. With an interface this becomes a type assertion for optional interfaces, which hides the rule in the code. - It is a value.
Newcopies it, and it can be checked in one place. A test builds a variant by changing one field, rather than writing a new type for each case. - Behaviour that must vary is already a function.
SourceURL,CacheDir,UnpackandInUseare the places where a driver supplies code. If a driver ever needs more than that, an interface can be added next to the struct without breaking it.
Compatibility
The JSON format of the list, the source URLs, and the cache paths and file names are those of the drivers before they adopted this package. Hyperv and qemu differencing disks refer to the cached image by absolute path, so these must not change. profiles_test.go holds the per-driver tests that pin them down; a driver's migration has to pass them.
Testing
go test ./...
Most tests need nothing but the module. They use httptest servers and a temporary workspace, and replace the workspace and logging, so they must not run in parallel.
The tests of locking, and of replacing and removing files, behave differently on Windows, and should be run there too.
Documentation
¶
Overview ¶
Package imagecore manages the list of VM images, and the local cache of image files, for kutti drivers.
The vbox, hyperv and qemu drivers used to carry near-identical copies of this code. imagecore holds the one copy. A driver describes itself with a Config, passes it to New, and gets a Store. The Store owns the list of images, which is kept in a file in the workspace config directory, and the image files, which are kept in a cache directory chosen by the driver.
New does no I/O, and nothing in this package does any at import time. The first operation on a Store reads the list.
Fetching, importing and purging an image are serialized, per version, between processes of the same user. The list is changed only under its own short lock, one image at a time, so that processes do not undo each other's changes.
The Image type implements drivercore.Image. A driver declares
type Image = imagecore.Image
so that its exported API does not change.
Compatibility ¶
The JSON format of the image list, the source URLs, and the cache paths and file names are the same as those used by the drivers before they adopted this package. Machines created by older versions of a driver refer to cached image files by absolute path, so these must not change.
Index ¶
- type Config
- type Image
- func (i *Image) Deprecated() bool
- func (i *Image) Fetch() error
- func (i *Image) FetchWithProgress(progress func(current int64, total int64)) error
- func (i *Image) FromFile(filepath string) error
- func (i *Image) K8sVersion() string
- func (i *Image) MarshalJSON() ([]byte, error)
- func (i *Image) PurgeLocal() error
- func (i *Image) Status() drivercore.ImageStatus
- func (i *Image) UnmarshalJSON(b []byte) error
- type Store
- func (s *Store) GetImage(k8sversion string) (drivercore.Image, error)
- func (s *Store) ImagePath(k8sversion string) (string, error)
- func (s *Store) K8sVersions() []string
- func (s *Store) ListImages() ([]drivercore.Image, error)
- func (s *Store) UpdateImageList() error
- func (s *Store) ValidK8sVersion(k8sversion string) bool
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// DriverName identifies the driver, for example "driver-qemu". It appears
// in messages, and in the names of lock files, which are
// "<DriverName>-images.lock" for the list, and
// "<DriverName>-images-<version>.lock" for the fetch of one version.
// It may contain only letters, digits, '-' and '_'.
DriverName string
// ConfigFile is the name of the file, in the workspace config directory,
// that holds the local copy of the image list, for example
// "driver-qemu-images.json". It is also the name of the file at the source
// URL.
ConfigFile string
// ImageExtension is the extension, with its dot, of cached image files, for
// example ".qcow2". The cached file for a version is named
// "kutti-<version><ImageExtension>".
ImageExtension string
// DownloadSuffix is the ending, with its dot, of the name of the file an
// image is downloaded into before it is checked and moved into the cache,
// for example ".qcow2download". That file is named
// "kutti-k8s-<version><DownloadSuffix>".
DownloadSuffix string
// SourceURL returns the location of the master image list. It is a
// function because a driver may choose it per platform, and because tests
// may change it after the Store has been created.
SourceURL func() string
// CacheDir returns the directory that holds cached image files, creating
// it if necessary. Downloads, and files made by Unpack, are put in work
// directories inside it, which are removed when the work is done. Note
// that if this directory is shared by several users,
// as it is for the qemu driver, the locks taken by a Store, which are
// per user, do not serialize the users against each other.
CacheDir func() (string, error)
// Unpack, if not nil, is called on every image file before it is checked
// and cached, whether it was downloaded or given to Image.FromFile. It
// returns the path of the file to check and cache, and a function to call
// when that file is no longer needed, which may be nil. workdir is a
// directory in which Unpack may create files, which is removed with
// everything in it when the image has been dealt with. If Unpack is nil,
// the file is used as it is.
Unpack func(path string, workdir string) (imagepath string, cleanup func(), err error)
// InUse, if not nil, reports whether the cached image for a version is in
// use by any machine. If it is set, a cached image is not replaced with
// different content, or removed, while it is in use, and an image already
// cached with the right checksum is not copied again. If it is nil,
// images are never considered to be in use.
InUse func(k8sversion string) (bool, error)
}
Config describes how a driver stores its images. All the fields except Unpack and InUse are required.
The functions in a Config are called when they are needed, never by New. They may be called from several goroutines at once.
type Image ¶
type Image struct {
// contains filtered or unexported fields
}
Image is a VM image: a template from which machines are created. It implements drivercore.Image.
Images are created by a Store, and keep a reference to it.
func (*Image) Deprecated ¶
Deprecated returns true if the image's version of Kubernetes is deprecated. New machines should not be created from such an image.
func (*Image) Fetch ¶
Fetch downloads the image from its source URL into the local cache.
Only one process of a user fetches, imports or purges the image for a version at a time. If the image becomes Downloaded while Fetch waits its turn, because another process fetched it, Fetch does nothing more. If the image was already Downloaded when it was obtained, Fetch downloads it again, which repairs a cached file that has gone missing.
func (*Image) FetchWithProgress ¶
FetchWithProgress is like Fetch, and reports the progress of the download via the supplied callback. The callback reports current and total in bytes.
func (*Image) FromFile ¶
FromFile imports an image from a file on the local file system into the local cache. The file is unpacked, if the driver needs that, and its checksum must be the one in the image list.
If the driver can tell whether a cached image is in use by a machine, a cached image is not replaced with different content while it is, and a cached image that already has the right checksum is not copied again.
func (*Image) K8sVersion ¶
K8sVersion returns the version of Kubernetes present in the image.
func (*Image) MarshalJSON ¶
MarshalJSON returns the JSON encoding of the image.
func (*Image) PurgeLocal ¶
PurgeLocal removes the image from the local cache, and marks it Notdownloaded. It does nothing if the image list does not have the image as Downloaded, and succeeds if the cached file is already gone.
If the driver can tell whether a cached image is in use by a machine, a cached image that is in use is not removed, and PurgeLocal fails.
func (*Image) Status ¶
func (i *Image) Status() drivercore.ImageStatus
Status returns the status of the image. Status can be Downloaded, meaning the image exists in the local cache and can be used to create machines, or Notdownloaded, meaning it has to be downloaded using Fetch.
func (*Image) UnmarshalJSON ¶
UnmarshalJSON parses and restores a JSON-encoded image. If it fails, the image is not changed. Unknown fields are ignored, and missing fields are left empty.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
A Store manages the image list and the cached image files of one driver. Create one with New.
A Store keeps no copy of the image list in memory. Every operation reads the list file afresh, so that several kutti processes, each with its own Store, see each other's changes. Once created, a Store never changes, so it is safe for use by several goroutines.
func New ¶
New creates a Store for the driver described by config. It does no I/O, and calls none of the functions in config. If config is invalid, which is a programming error, the first operation on the Store returns an error that says so.
func (*Store) GetImage ¶
func (s *Store) GetImage(k8sversion string) (drivercore.Image, error)
GetImage returns the image for a version of Kubernetes. It returns a nil image and an error if the list cannot be read, or has no such image.
func (*Store) ImagePath ¶
ImagePath returns the path where the cached file for a version of Kubernetes is, or would be: "kutti-<version><ImageExtension>" in the cache directory. The file may not exist. A driver uses this to find the image from which to create machines, and in its InUse function, so that the name is worked out in one place only.
It returns an error if the version is not a valid version, so that a path can never be made from something that is not.
func (*Store) K8sVersions ¶
K8sVersions returns the Kubernetes versions in the image list, in version order: "1.9" comes before "1.10". It returns an empty slice if the list cannot be read, and logs why.
func (*Store) ListImages ¶
func (s *Store) ListImages() ([]drivercore.Image, error)
ListImages returns the images in the local image list, in version order. It returns an empty slice and an error if the list cannot be read.
func (*Store) UpdateImageList ¶
UpdateImageList downloads the master image list, and makes it the local list. An image that is Downloaded stays Downloaded if the new list has an image for the same version with the same checksum and source URL. Every other image in the new list is Notdownloaded, whatever status the master list gives it. Images that are not in the new list are dropped from the list, but their cached files are not touched.
The downloaded data is checked before anything changes. If it cannot be parsed, any entry is invalid, or it has no images at all, the whole list is rejected, an error is logged and returned, and the local list is left as it was. A list with no images is never what a master list should be, and accepting one would make every cached image Notdownloaded when the next good list arrives.
If the local list cannot be used, because its content is invalid, it is moved aside as "<name>.<timestamp>.corrupt", in the same directory, and replaced. UpdateImageList is how a damaged list is repaired.
func (*Store) ValidK8sVersion ¶
ValidK8sVersion returns true if the specified Kubernetes version is in the image list. It returns false if the list cannot be read, and logs why.
Source Files