ensure

Documentation

Overview

Package ensure contains methods and types for interacting with the 'ensure file format'.

The format is used by the CIPD client to describe the desired state of a CIPD installation. This states can be asserted with the CIPD client 'ensure' command. The state is essentially a list of packages, their versions and their installation subdirectories ("subdirs").

Format Description

The file is line-oriented. All statements fit on a single line.

A line can be blank, a comment, a setting, a package, or a directive.

Comments

A comment begins with a # and goes to the end of the line. It is ignored. Example:

# This is a comment. It has no effect.

Settings

A setting looks like `$name value`. Settings are global and can only be set once per file. The following settings are allowed:

• `$ServiceURL <url>` is the url for the CIPD service. It can be used in lieu of the -service-url command line parameter.
• `$VerifiedPlatform <platform> [ <platform>]*` allows the manifest to specify a list of ${platform} expansions to use for verification. Multiple $VerifiedPlatform directives can be specified, and will accumulate.
• `$ParanoidMode NotParanoid|CheckPresence|CheckIntegrity` controls how paranoid CIPD should be when checking the state of already installed packages. `NotParanoid` (default) indicates that CIPD should trust that no one is messing with the installation root directory. `CheckPresence` indicates CIPD should verify all files that are supposed to be already installed are indeed present (this incurs additional overhead). And finally `CheckIntegrity` will verify that files weren't modified since they were installed (this incurs even more overhead).
• `$ResolvedVersions <filename>` is a path (either relative to the ensure file or absolute) to a file that contains "frozen" instance IDs (hashes) of all packages for all verified platforms, resolved at the time the "frozen" file was generated by `cipd ensure-file-resolve`. The CIPD client will use these instance IDs (instead of contacting the backend) when installing packages in `cipd ensure`. Using this file allows to cryptographically bind the intended state of the deployment to a git commit, since all hashes of all packages become part of the git commit object. Use this if you want to reduce trust in the CIPD backend and root in the git server instead.
• `$OverrideInstallMode copy` forces all packages in this ensure-file to be installed with copy mode, regardless of what the package manifests list. `symlink` mode only works on mac/linux, and causes more problems than it solves. We recommend that all ensure files have this setting, and in the future this will become automatically set. See crbug.com/1329641 for additional discussion.

Package Definitions

A package line looks like `<package_template> <version>`. Package templates are CIPD package names, with optional expansion parameters `${os}`, `${arch}`, and `${platform}`. These placeholders can appear anywhere in the package template except for the first letter. All other characters in the template are taken verbatim.

${os} will expand to one of the following, based on the value of this client's runtime.GOOS value:

• windows
• mac
• linux

${arch} will expand to one of the following, based on the value of this client's runtime.GOARCH value:

• 386
• amd64
• armv6l

Since these two often appear together, the convenience placeholder `${platform}` expands to the equivalent of `${os}-${arch}`.

All of these parameters also support the syntax ${var=possible,values}. What this means is that the package line will be expanded if, and only if, var equals one of the possible values. If that var does not match a possible value, the line is ignored. This allows you to do, e.g.:

path/to/package/${os=windows}  windows_release
path/to/package/${os=linux}    linux_release
# no version for mac

path/to/posix/tool/${os=mac,linux}  some_tag:value
# no version for windows

Directives

A directive looks like `@name value`. Directives are 'sticky' and apply until the next same-name directive. The following directive names are allowed:

• `@Subdir` allows you to change the subdirectory that subsequent packages are installed to.
• The subdir value is always relative to the root of the CIPD installation (the directory containing the .cipd folder).
• The value of Subdir before any @Subdir directives is "", or the root of the CIPD installation.
• You can reset the directory back to root by doing `@Subdir` by itself, without a value.
• @Subdir directives also support expansion parameters `${os}`, `${arch}` and `${platform}`. Using a subdir expansion like `${param=val}` will cause that subdirectory, and any packages in it, to only exist if the param matches one of the values.

Example

Here is an example ensure file which demonstrates all the various features.

# This is an ensure file!
$ServiceURL https://chrome-infra-packages.appspot.com/
$ParanoidMode CheckPresence
$ResolvedVersions cipd_lock.versions

# This is the CIPD client itself
infra/tools/cipd/${os}-${arch}  latest

@Subdir python
python/wheels/pip                     version:8.1.2
# use the convenience placeholder
python/wheels/coverage/${platform}    version:4.1

@Subdir infra/support
infra/some/other/package deadbeefdeadbeefdeadbeefdeadbeefdeadbeef

# only exists on windows machines
@Subdir support/${os=windows}-${arch}
some/support/package  latest
some/other/support/package  latest

# Always exists, but the directory changes based on the os
@Subdir platform/${os}
a/platform/package latest

Index

• type File
  • func LoadEnsureFile(path string) (*File, error)
  • func ParseFile(r io.Reader) (*File, error)
  • func (f *File) Clone() *File
  • func (f *File) Resolve(rslv VersionResolver, expander template.Expander) (*ResolvedFile, error)
  • func (f *File) Serialize(w io.Writer) error
• type PackageDef
  • func (p *PackageDef) Expand(expander template.Expander) (pkg string, err error)
  • func (p PackageDef) String() string
• type PackageSlice
  • func (ps PackageSlice) Len() int
  • func (ps PackageSlice) Less(i, j int) bool
  • func (ps PackageSlice) Swap(i, j int)
• type ResolvedFile
  • func (f *ResolvedFile) Serialize(w io.Writer) error
• type VersionResolver
• type VersionsFile
  • func ParseVersionsFile(r io.Reader) (VersionsFile, error)
  • func (v VersionsFile) AddVersion(pkg, ver, iid string) error
  • func (v VersionsFile) Equal(a VersionsFile) bool
  • func (v VersionsFile) ResolveVersion(pkg, ver string) (common.Pin, error)
  • func (v VersionsFile) Serialize(w io.Writer) error

Types

type File

type File struct {
	ServiceURL          string
	ParanoidMode        deployer.ParanoidMode
	ResolvedVersions    string
	OverrideInstallMode pkg.InstallMode

	PackagesBySubdir map[string]PackageSlice
	VerifyPlatforms  []template.Platform
}

File is an in-process representation of the 'ensure file' format.

func LoadEnsureFile

func LoadEnsureFile(path string) (*File, error)

LoadEnsureFile loads the ensure file from the given path or stdin (if 'path' is "-").

If the ensure file has $ResolvedVersions directive, the returned File will have ResolvedVersions field set to an absolute path to the resolved versions file. Its presence or correctness is not checked.

func ParseFile

func ParseFile(r io.Reader) (*File, error)

ParseFile parses an ensure file from the given reader. See the package docs for the format of this file.

This file will contain unresolved template strings for package names as well as unpinned package versions. Use File.Resolve() to obtain resolved+pinned versions of these.

func (*File) Clone

func (f *File) Clone() *File

Clone returns a deep copy of the File.

func (*File) Resolve

func (f *File) Resolve(rslv VersionResolver, expander template.Expander) (*ResolvedFile, error)

Resolve takes the current unresolved File and expands all package templates using the provided expander (usually template.DefaultExpander()), and also resolves all versions with the provided VersionResolver, calling it concurrently from multiple goroutines.

Returns either a single error (if something is wrong with the ensure file), or a multi-error with all resolution errors, sorted by definition line numbers.

func (*File) Serialize

func (f *File) Serialize(w io.Writer) error

Serialize writes the File to an io.Writer in canonical order.

type PackageDef

type PackageDef struct {
	PackageTemplate   string
	UnresolvedVersion string

	// LineNo is set while parsing an ensure file by the ParseFile method. It is
	// used by File.Resolve to give additional context if an error occurs.
	LineNo int
}

PackageDef defines a package line parsed out of an ensure file.

func (*PackageDef) Expand

func (p *PackageDef) Expand(expander template.Expander) (pkg string, err error)

Expand expands the package name template and checks that resulting package name and version are syntactically correct.

May return template.ErrSkipTemplate is this package definition should be skipped given the current expansion variables values.

func (PackageDef) String

func (p PackageDef) String() string

type PackageSlice

type PackageSlice []PackageDef

PackageSlice is a sortable slice of PackageDef

func (PackageSlice) Len

func (ps PackageSlice) Len() int

func (PackageSlice) Less

func (ps PackageSlice) Less(i, j int) bool

func (PackageSlice) Swap

func (ps PackageSlice) Swap(i, j int)

type ResolvedFile

type ResolvedFile struct {
	ServiceURL          string
	ParanoidMode        deployer.ParanoidMode
	OverrideInstallMode pkg.InstallMode

	PackagesBySubdir common.PinSliceBySubdir
}

ResolvedFile only contains valid, fully-resolved information and is the result of calling File.Resolve.

func (*ResolvedFile) Serialize

func (f *ResolvedFile) Serialize(w io.Writer) error

Serialize writes the ResolvedFile to an io.Writer in canonical order.

type VersionResolver

type VersionResolver func(pkg, vers string) (common.Pin, error)

VersionResolver transforms a {PackageName, Version} tuple (corresponding to the given `def`) into a resolved pin.

• `pkg` is guaranteed to pass common.ValidatePackageName
• `vers` is guaranteed to pass common.ValidateInstanceVersion

VersionResolver should expect to be called concurrently from multiple goroutines.

type VersionsFile

type VersionsFile map[unresolvedVer]string

VersionsFile contains a mapping "(package name, version) -> instance ID" used to resolve versions (instead of backend calls) for ensure files that have $ResolvedVersions directive.

In serialized form it is represented as a set of triples separated by one or more new lines (there must be no new lines inside the triple):

""" # Comments are allowed, they are skipped (not even considered as '\n'). <package name> <version> <resolved instance ID>

<package name> <version> <resolved instance ID> """

Leading and trailing whitespace on a line is ignored.

In the canonical serialization triples are ordered by (package, version).

VersionsFile is safe for read-only concurrent use. Concurrent modifications should be protected by a lock. This is just a map in disguise.

func ParseVersionsFile

func ParseVersionsFile(r io.Reader) (VersionsFile, error)

ParseVersionsFile parses previously serialized versions file.

func (VersionsFile) AddVersion

func (v VersionsFile) AddVersion(pkg, ver, iid string) error

AddVersion adds (or overrides) an instance ID mapped to the given version.

Returns an error if any of the arguments is invalid.

If 'ver' is already an instance ID, just checks if it is equal to 'iid' and silently doesn't modify the map.

func (VersionsFile) Equal

func (v VersionsFile) Equal(a VersionsFile) bool

Equal returns true if version files have same entries.

func (VersionsFile) ResolveVersion

func (v VersionsFile) ResolveVersion(pkg, ver string) (common.Pin, error)

ResolveVersion returns a pin matching the given version or an error if such version is not in the map.

If 'ver' is already an instance ID, returns it right away.

func (VersionsFile) Serialize

func (v VersionsFile) Serialize(w io.Writer) error

Serialize writes the VersionsFile to an io.Writer in canonical order.