cipd

package
v0.0.0-...-678bb0e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 8, 2017 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package cipd implements client side of Chrome Infra Package Deployer.

Binary package file format (in free form representation): 

<binary package> := <zipped data>
<zipped data> := DeterministicZip(<all input files> + <manifest json>)
<manifest json> := File{
  name: ".cipdpkg/manifest.json",
  data: JSON({
    "FormatVersion": "1",
    "PackageName": <name of the package>
  }),
}
DeterministicZip = zip archive with deterministic ordering of files and stripped timestamps

Main package data (<zipped data> above) is deterministic, meaning its content depends only on inputs used to built it (byte to byte): contents and names of all files added to the package (plus 'executable' file mode bit) and a package name (and all other data in the manifest).

Binary package data MUST NOT depend on a timestamp, hostname of machine that built it, revision of the source code it was built from, etc. All that information will be distributed as a separate metadata packet associated with the package when it gets uploaded to the server.

TODO: expand more when there's server-side package data model (labels and stuff).

Index

Constants

View Source 
const (
	// GrantRole is used in PackageACLChange to request a role to be granted.
	GrantRole PackageACLChangeAction = "GRANT"
	// RevokeRole is used in PackageACLChange to request a role to be revoked.
	RevokeRole PackageACLChangeAction = "REVOKE"

	// CASFinalizationTimeout is how long to wait for CAS service to finalize the upload.
	CASFinalizationTimeout = 5 * time.Minute
	// SetRefTimeout is how long to wait for an instance to be processed when setting a ref.
	SetRefTimeout = 3 * time.Minute
	// TagAttachTimeout is how long to wait for an instance to be processed when attaching tags.
	TagAttachTimeout = 3 * time.Minute
)
View Source 
const (
	EnvCacheDir            = "CIPD_CACHE_DIR"
	EnvHTTPUserAgentPrefix = "CIPD_HTTP_USER_AGENT_PREFIX"
)

Environment variable definitions

Variables

View Source 
var (
	// ErrFinalizationTimeout is returned if CAS service can not finalize upload fast enough.
	ErrFinalizationTimeout = errors.New("timeout while waiting for CAS service to finalize the upload", transient.Tag)
	// ErrBadUpload is returned when a package file is uploaded, but servers asks us to upload it again.
	ErrBadUpload = errors.New("package file is uploaded, but servers asks us to upload it again", transient.Tag)
	// ErrBadUploadSession is returned by UploadToCAS if provided UploadSession is not valid.
	ErrBadUploadSession = errors.New("uploadURL must be set if UploadSessionID is used")
	// ErrUploadSessionDied is returned by UploadToCAS if upload session suddenly disappeared.
	ErrUploadSessionDied = errors.New("upload session is unexpectedly missing", transient.Tag)
	// ErrNoUploadSessionID is returned by UploadToCAS if server didn't provide upload session ID.
	ErrNoUploadSessionID = errors.New("server didn't provide upload session ID")
	// ErrSetRefTimeout is returned when service refuses to move a ref for a long time.
	ErrSetRefTimeout = errors.New("timeout while moving a ref", transient.Tag)
	// ErrAttachTagsTimeout is returned when service refuses to accept tags for a long time.
	ErrAttachTagsTimeout = errors.New("timeout while attaching tags", transient.Tag)
	// ErrDownloadError is returned by FetchInstance on download errors.
	ErrDownloadError = errors.New("failed to download the package file after multiple attempts", transient.Tag)
	// ErrUploadError is returned by RegisterInstance and UploadToCAS on upload errors.
	ErrUploadError = errors.New("failed to upload the package file after multiple attempts", transient.Tag)
	// ErrAccessDenined is returned by calls talking to backend on 401 or 403 HTTP errors.
	ErrAccessDenined = errors.New("access denied (not authenticated or not enough permissions)")
	// ErrBackendInaccessible is returned by calls talking to backed if it doesn't response.
	ErrBackendInaccessible = errors.New("request to the backend failed after multiple attempts", transient.Tag)
	// ErrEnsurePackagesFailed is returned by EnsurePackages if something is not right.
	ErrEnsurePackagesFailed = errors.New("failed to update packages, see the log")
	// ErrPackageNotFound is returned by DeletePackage if the package doesn't exist.
	ErrPackageNotFound = errors.New("no such package")
)
View Source 
var (
	// UserAgent is HTTP user agent string for CIPD client.
	UserAgent = "cipd 1.7.0"
)

Functions

This section is empty.

Types

type ActionError 

type ActionError struct {
	Action string     `json:"action"`
	Pin    common.Pin `json:"pin"`
	Error  JSONError  `json:"error,omitempty"`
}

ActionError holds an error that happened when installing or removing the pin.

type ActionMap 

type ActionMap map[string]*Actions

ActionMap is a map of subdir to the Actions which will occur within it.

func (ActionMap) Log 

func (am ActionMap) Log(ctx context.Context)

Log prints the pending action to the logger installed in ctx.

func (ActionMap) LoopOrdered 

func (am ActionMap) LoopOrdered(cb func(subdir string, actions *Actions))

LoopOrdered loops over the ActionMap in sorted order (by subdir).

type Actions 

type Actions struct {
	ToInstall common.PinSlice `json:"to_install,omitempty"` // pins to be installed
	ToUpdate  []UpdatedPin    `json:"to_update,omitempty"`  // pins to be replaced
	ToRemove  common.PinSlice `json:"to_remove,omitempty"`  // pins to be removed
	Errors    []ActionError   `json:"errors,omitempty"`     // all individual errors
}

Actions is returned by EnsurePackages.

It lists pins that were attempted to be installed, updated or removed, as well as all errors.

func (*Actions) Empty 

func (a *Actions) Empty() bool

Empty is true if there are no actions specified.

type Client 

type Client interface {
	// BeginBatch makes the client enter into a "batch mode".
	//
	// In this mode various cleanup and cache updates, usually performed right
	// away, are deferred until 'EndBatch' call.
	//
	// This is an optimization. Use it if you plan to call a bunch of Client
	// methods in a short amount of time (parallel or sequentially).
	//
	// Batches can be nested.
	BeginBatch(ctx context.Context)

	// EndBatch ends a batch started with BeginBatch.
	//
	// EndBatch does various delayed maintenance tasks (like cache updates, trash
	// cleanup and so on). This is best-effort operations, and thus this method
	// doesn't return an errors.
	//
	// See also BeginBatch doc for more details.
	EndBatch(ctx context.Context)

	// FetchACL returns a list of PackageACL objects (parent paths first).
	//
	// Together they define the access control list for the given package subpath.
	FetchACL(ctx context.Context, packagePath string) ([]PackageACL, error)

	// ModifyACL applies a set of PackageACLChanges to a package path.
	ModifyACL(ctx context.Context, packagePath string, changes []PackageACLChange) error

	// UploadToCAS uploads package data blob to Content Addressed Store.
	//
	// Does nothing if it is already there. The data is addressed by SHA1 hash
	// (also known as package's InstanceID). It can be used as a standalone
	// function (if 'session' is nil) or as a part of more high level upload
	// process (in that case upload session can be opened elsewhere and its
	// properties passed here via 'session' argument).
	//
	// Returns nil on successful upload.
	UploadToCAS(ctx context.Context, sha1 string, data io.ReadSeeker, session *UploadSession, timeout time.Duration) error

	// ResolveVersion converts an instance ID, a tag or a ref into a concrete Pin.
	ResolveVersion(ctx context.Context, packageName, version string) (common.Pin, error)

	// MaybeUpdateClient will update `destination` to `targetVersion` if
	// `currentHash` doesn't match version's executable hash.
	//
	// This update is done from the "infra/tools/cipd/${os}-${arch}" package.
	MaybeUpdateClient(ctx context.Context, fs local.FileSystem, targetVersion, currentHash, destination string) error

	// RegisterInstance makes the package instance available for clients.
	//
	// It uploads the instance to the storage and registers it in the package
	// repository.
	RegisterInstance(ctx context.Context, instance local.PackageInstance, timeout time.Duration) error

	// DeletePackage removes the package (all its instances) from the backend.
	//
	// It will delete all package instances, all tags and refs. There's no undo.
	DeletePackage(ctx context.Context, packageName string) error

	// SetRefWhenReady moves a ref to point to a package instance.
	SetRefWhenReady(ctx context.Context, ref string, pin common.Pin) error

	// AttachTagsWhenReady attaches tags to an instance.
	AttachTagsWhenReady(ctx context.Context, pin common.Pin, tags []string) error

	// FetchInstanceInfo returns general information about the instance.
	FetchInstanceInfo(ctx context.Context, pin common.Pin) (InstanceInfo, error)

	// FetchInstanceTags returns information about tags attached to the instance.
	//
	// The returned list is sorted by tag key and creation timestamp (newest
	// first). If 'tags' is empty, fetches all attached tags, otherwise only
	// ones specified.
	FetchInstanceTags(ctx context.Context, pin common.Pin, tags []string) ([]TagInfo, error)

	// FetchInstanceRefs returns information about refs pointing to the instance.
	//
	// The returned list is sorted by modification timestamp (newest first). If
	// 'refs' is empty, fetches all refs, otherwise only ones specified.
	FetchInstanceRefs(ctx context.Context, pin common.Pin, refs []string) ([]RefInfo, error)

	// FetchInstance downloads a package instance file from the repository.
	//
	// It verifies that the package hash matches pin.InstanceID.
	//
	// It returns an InstanceFile pointing to the raw package data. The caller
	// must close it when done.
	FetchInstance(ctx context.Context, pin common.Pin) (local.InstanceFile, error)

	// FetchInstanceTo downloads a package instance file into the given writer.
	//
	// This is roughly the same as getting a reader with 'FetchInstance' and
	// copying its data into the writer, except this call skips unnecessary temp
	// files if the client is not using cache.
	//
	// It verifies that the package hash matches pin.InstanceID, but does it while
	// writing to 'output', so expect to discard all data there if FetchInstanceTo
	// returns an error.
	FetchInstanceTo(ctx context.Context, pin common.Pin, output io.WriteSeeker) error

	// FetchAndDeployInstance fetches the package instance and deploys it.
	//
	// Deploys to the given subdir under the site root (see ClientOptions.Root).
	// It doesn't check whether the instance is already deployed.
	FetchAndDeployInstance(ctx context.Context, subdir string, pin common.Pin) error

	// ListPackages returns a list of strings of package names.
	ListPackages(ctx context.Context, path string, recursive, showHidden bool) ([]string, error)

	// SearchInstances finds all instances with given tag and optionally name.
	//
	// Returns their concrete Pins.
	SearchInstances(ctx context.Context, tag, packageName string) (common.PinSlice, error)

	// EnsurePackages installs, removes and updates packages in the site root.
	//
	// Given a description of what packages (and versions) should be installed it
	// will do all necessary actions to bring the state of the site root to the
	// desired one.
	//
	// If dryRun is true, will just check for changes and return them in Actions
	// struct, but won't actually perform them.
	//
	// If the update was only partially applied, returns both Actions and error.
	EnsurePackages(ctx context.Context, pkgs common.PinSliceBySubdir, dryRun bool) (ActionMap, error)

	// IncrementCounter adds delta to the counter's value and updates its last
	// updated timestamp.
	//
	// delta must be 0 or 1.
	IncrementCounter(ctx context.Context, pin common.Pin, counterName string, delta int) error

	// ReadCounter returns the current value of the counter.
	ReadCounter(ctx context.Context, pin common.Pin, counterName string) (Counter, error)
}

Client provides high-level CIPD client interface. Thread safe.

func NewClient 

func NewClient(opts ClientOptions) (Client, error)

NewClient initializes CIPD client object.

type ClientOptions 

type ClientOptions struct {
	// ServiceURL is root URL of the backend service.
	//
	// Default is ServiceURL const.
	ServiceURL string

	// Root is a site root directory.
	//
	// It is a directory where packages will be installed to. It also hosts
	// .cipd/* directory that tracks internal state of installed packages and
	// keeps various cache files. 'Root' can be an empty string if the client is
	// not going to be used to deploy or remove local packages.
	Root string

	// CacheDir is a directory for shared cache.
	//
	// If empty, instances are not cached and tags are cached inside the site
	// root. If both Root and CacheDir are empty, tag cache is disabled.
	CacheDir string

	// AnonymousClient is http.Client that doesn't attach authentication headers.
	//
	// Will be used when talking to the Google Storage. We use signed URLs that do
	// not require additional authentication.
	//
	// Default is http.DefaultClient.
	AnonymousClient *http.Client

	// AuthenticatedClient is http.Client that attaches authentication headers.
	//
	// Will be used when talking to the backend.
	//
	// Default is same as AnonymousClient (it will probably not work for most
	// packages, since the backend won't authorize an anonymous access).
	AuthenticatedClient *http.Client

	// UserAgent is put into User-Agent HTTP header with each request.
	//
	// Default is UserAgent const.
	UserAgent string
}

ClientOptions is passed to NewClient factory function.

func (*ClientOptions) LoadFromEnv 

func (opts *ClientOptions) LoadFromEnv(getEnv func(string) string) error

LoadFromEnv loads supplied default values from an environment into opts.

The supplied getEnv function is used to access named enviornment variables, and should return an empty string if the enviornment variable is not defined.

type Counter 

type Counter struct {
	// Name is the counter's name.
	Name string `json:"name"`
	// Value is the counter's value.
	Value int64 `json:"value"`
	// CreatedTS is the first time the counter was written.
	CreatedTS UnixTime `json:"created_ts"`
	// UpdatedTS is the most recent time the counter was written.
	UpdatedTS UnixTime `json:"updated_ts"`
}

Counter is returned by ReadCounter.

type InstanceInfo 

type InstanceInfo struct {
	// Pin identifies package instance.
	Pin common.Pin `json:"pin"`
	// RegisteredBy is identify of whoever uploaded this instance.
	RegisteredBy string `json:"registered_by"`
	// RegisteredTs is when the instance was registered.
	RegisteredTs UnixTime `json:"registered_ts"`
}

InstanceInfo is returned by FetchInstanceInfo.

type JSONError 

type JSONError struct {
	// contains filtered or unexported fields
}

JSONError is wrapper around Error that serializes it as string.

func (JSONError) MarshalJSON 

func (e JSONError) MarshalJSON() ([]byte, error)

MarshalJSON is used by JSON encoder.

type PackageACL 

type PackageACL struct {
	// PackagePath is a package subpath this ACL is defined for.
	PackagePath string `json:"package_path"`
	// Role is a role that listed users have, e.g. 'READER', 'WRITER', ...
	Role string `json:"role"`
	// Principals list users and groups granted the role.
	Principals []string `json:"principals"`
	// ModifiedBy specifies who modified the list the last time.
	ModifiedBy string `json:"modified_by"`
	// ModifiedTs is a timestamp when the list was modified the last time.
	ModifiedTs UnixTime `json:"modified_ts"`
}

PackageACL is per package path per role access control list that is a part of larger overall ACL: ACL for package "a/b/c" is a union of PackageACLs for "a" "a/b" and "a/b/c".

type PackageACLChange 

type PackageACLChange struct {
	// Action defines what action to perform: GrantRole or RevokeRole.
	Action PackageACLChangeAction
	// Role to grant or revoke to a user or group.
	Role string
	// Principal is a user or a group to grant or revoke a role for.
	Principal string
}

PackageACLChange is a mutation to some package ACL.

type PackageACLChangeAction 

type PackageACLChangeAction string

PackageACLChangeAction defines a flavor of PackageACLChange.

type ReadSeekCloser 

type ReadSeekCloser interface {
	io.Reader
	io.Seeker
	io.Closer
}

ReadSeekCloser is the interface that groups Reader, Seeker and Closer.

type RefInfo 

type RefInfo struct {
	// Ref is the ref name.
	Ref string `json:"ref"`
	// ModifiedBy is identify of whoever modified this ref last time.
	ModifiedBy string `json:"modified_by"`
	// ModifiedTs is when the ref was modified last time.
	ModifiedTs UnixTime `json:"modified_ts"`
}

RefInfo is returned by FetchInstanceRefs.

type TagInfo 

type TagInfo struct {
	// Tag is actual tag name ("key:value" pair).
	Tag string `json:"tag"`
	// RegisteredBy is identify of whoever attached this tag.
	RegisteredBy string `json:"registered_by"`
	// RegisteredTs is when the tag was registered.
	RegisteredTs UnixTime `json:"registered_ts"`
}

TagInfo is returned by FetchInstanceTags.

type UnixTime 

type UnixTime time.Time

UnixTime is time.Time that serializes to unix timestamp in JSON (represented as a number of seconds since January 1, 1970 UTC).

func (UnixTime) Before 

func (t UnixTime) Before(t2 UnixTime) bool

Before is used to compare UnixTime objects.

func (UnixTime) IsZero 

func (t UnixTime) IsZero() bool

IsZero reports whether t represents the zero time instant.

func (UnixTime) MarshalJSON 

func (t UnixTime) MarshalJSON() ([]byte, error)

MarshalJSON is used by JSON encoder.

func (UnixTime) String 

func (t UnixTime) String() string

String is needed to be able to print UnixTime.

type UpdatedPin 

type UpdatedPin struct {
	From common.Pin `json:"from"`
	To   common.Pin `json:"to"`
}

UpdatedPin specifies a pair of pins: old and new version of a package.

type UploadSession 

type UploadSession struct {
	// ID identifies upload session in the backend.
	ID string
	// URL is where to upload the data to.
	URL string
}

UploadSession describes open CAS upload session.

Source Files

View all Source files

Directories

Path Synopsis
Package common defines structures and functions used by cipd/* packages.
 Package common defines structures and functions used by cipd/* packages.
Package ensure contains methods and types for interacting with the 'ensure file format'.
 Package ensure contains methods and types for interacting with the 'ensure file format'.
messages
Package messages is a generated protocol buffer package.
 Package messages is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.