common

package
Version: v0.0.0-...-2ef5af0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 9, 2022 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package common contains widely used CV utilities & types.

Index

Constants

View Source
const (
	// MaxTriggerAge limits CV operation only to most recently triggered CLs.
	//
	// In particular, this reduces Gerrit querying to CLs modified since
	// (now-MaxTriggerAge).
	MaxTriggerAge = 24 * 7 * time.Hour
)
View Source
const RunKind = "Run"

RunKind is the Datastore entity kind for Run.

Variables

View Source
var DSContentionTag = errors.BoolTag{Key: errors.NewTagKey("Datastore Contention")}

DSContentionTag when set indicates Datastore contention.

It's set on errors by parts of CV which are especially prone to DS contention to reduce noise in logs and for more effective retries.

Functions

func CLIDsAsInt64s

func CLIDsAsInt64s(ids []CLID) []int64

CLIDsAsInt64s returns proto representation of CLIDs.

func DifferenceSorted

func DifferenceSorted(a, b []int64) []int64

DifferenceSorted returns all int64s in the first slice and not the second.

Both slices must be sorted. Doesn't modify input slices.

func DistributeOffset

func DistributeOffset(pollInterval time.Duration, keyParts ...string) time.Duration

DistributeOffset deterministically chooses an offset across keys in [1..pollInterval) range aimining for uniform distribution across all keys.

A key is simply a concatenation of key parts with '\0' filler in between.

func IsDatastoreContention

func IsDatastoreContention(err error) bool

IsDatastoreContention is best-effort detection of transactions aborted due to pessimistic concurrency control of Datastore backed by Firestore.

This is fragile, because it relies on undocumented but likely rarely changed English description of an error.

func LogError

func LogError(ctx context.Context, err error, expectedErrors ...error)

LogError is errors.Log with CV-specific package filtering.

Logs entire error stack with ERROR severity by default. Logs just error with WARNING severity iff one of error (or its inner error) equal at least one of the given list of `expectedErrors` errors. This is useful if TQ handler is known to frequently fail this way.

expectedErrors must contain only unwrapped errors.

func MostSevereError

func MostSevereError(err error) error

MostSevereError returns the most severe error in order of non-transient => transient => nil.

Walks over potentially recursive errors.MultiError errors only.

Returns only singular errors or nil if input was nil.

func PB2TimeNillable

func PB2TimeNillable(pb *timestamppb.Timestamp) time.Time

PB2TimeNillable is the opposite of Time2PBNillable.

ie same as pb.AsTime() but returns zero time if pb is nil.

func TQifyError

func TQifyError(ctx context.Context, err error) error

TQifyError is shortcut for TQIfy{}.Error.

func Time2PBNillable

func Time2PBNillable(t time.Time) *timestamppb.Timestamp

Time2PBNillable is like timestamppb.New() but returns nil on zero time.

func UnionSorted

func UnionSorted(a, b []int64) []int64

UnionSorted returns sorted unique int64s from two slices.

Both slices must be sorted and unique. Doesn't modify input slices.

func UniqueSorted

func UniqueSorted(v []int64) []int64

UniqueSorted sorts & removes duplicates in place.

Returns the potentially shorter slice.

Types

type CLID

type CLID int64

CLID is a unique ID of a CL used internally in CV.

It's just 8 bytes long and is thus much shorter than ExternalID, which reduces CPU & RAM & storage costs of CL graphs for multi-CL Runs.

type CLIDs

type CLIDs []CLID

CLIDs is a convenience type to facilitate handling of a slice of CLID.

func MakeCLIDs

func MakeCLIDs(ids ...int64) CLIDs

MakeCLIDs returns CLIDs from list of clids in int64.

func (CLIDs) Contains

func (ids CLIDs) Contains(id CLID) bool

Contains returns true if CLID is inside these CLIDs.

func (*CLIDs) Dedupe

func (p *CLIDs) Dedupe()

Dedupe removes duplicates in place and sorts the slice.

Note: Does not preserve original order.

func (CLIDs) Len

func (ids CLIDs) Len() int

Len is the number of elements in the collection.

func (CLIDs) Less

func (ids CLIDs) Less(i int, j int) bool

Less reports whether the element with index i should sort before the element with index j.

func (CLIDs) Set

func (ids CLIDs) Set() CLIDsSet

Set returns a new set of CLIDs.

func (CLIDs) Swap

func (ids CLIDs) Swap(i int, j int)

Swap swaps the elements with indexes i and j.

type CLIDsSet

type CLIDsSet map[CLID]struct{}

CLIDsSet is convenience type to reduce the boilerplate.

func MakeCLIDsSet

func MakeCLIDsSet(ids ...int64) CLIDsSet

MakeCLIDsSet returns new CLIDsSet from list of clids in int64.

func (CLIDsSet) Add

func (s CLIDsSet) Add(clid CLID)

func (CLIDsSet) AddI64

func (s CLIDsSet) AddI64(id int64)

func (CLIDsSet) Del

func (s CLIDsSet) Del(id CLID)

func (CLIDsSet) DelAll

func (s CLIDsSet) DelAll(ids CLIDs)

func (CLIDsSet) DelI64

func (s CLIDsSet) DelI64(id int64)

func (CLIDsSet) Has

func (s CLIDsSet) Has(clid CLID) bool

func (CLIDsSet) HasI64

func (s CLIDsSet) HasI64(id int64) bool

func (CLIDsSet) Reset

func (s CLIDsSet) Reset(ids ...CLID)

Reset resets the set to contain just the given IDs.

func (CLIDsSet) ResetI64

func (s CLIDsSet) ResetI64(ids ...int64)

func (CLIDsSet) ToCLIDs

func (s CLIDsSet) ToCLIDs() CLIDs

type Env

type Env struct {
	// LogicalHostname is CV hostname referred to in configs.
	//
	// On GAE, this is something like "luci-change-verifier-dev.appspot.com"
	// and it is part of the HTTPAddressBase.
	//
	// Under local development, this is usually set to GAE-looking hostname, while
	// keeping HTTPAddressBase a real localhost URL.
	LogicalHostname string

	// HTTPAddressBase can be used to generate URLs to this CV service.
	//
	// Doesn't have a trailing slash.
	//
	// For example,
	//   * "https://luci-change-verifier-dev.appspot.com"
	//   * "http://localhost:8080"
	HTTPAddressBase string

	// IsGAEDev is true if this is a -dev GAE environment.
	//
	// Deprecated. Do not use in new code. It should only be used during migration
	// from CQDaemon which doesn't have equivalent -dev environment.
	IsGAEDev bool
}

Env describes where CV runs at.

type RunID

type RunID string

RunID is an unique RunID to identify a Run in CV.

RunID is string like `luciProject/inverseTS-1-hexHashDigest` consisting of 7 parts:

  1. The LUCI Project that this Run belongs to. Purpose: separates load on Datastore from different projects.
  2. `/` separator.
  3. InverseTS, defined as (`endOfTheWorld` - CreateTime) in ms precision, left-padded with zeros to 13 digits. See `Run.CreateTime` Doc. Purpose: ensures queries by default orders runs of the same project by most recent first.
  4. `-` separator.
  5. Digest version (see part 7).
  6. `-` separator.
  7. A hex digest string uniquely identifying the set of CLs involved in this Run. Purpose: ensures two simultaneously started Runs in the same project won't have the same RunID.

func FromPublicRunID

func FromPublicRunID(id string) (RunID, error)

FromPublicRunID is the inverse of RunID.PublicID().

func MakeRunID

func MakeRunID(luciProject string, createTime time.Time, digestVersion int, clsDigest []byte) RunID

func (RunID) AttemptKey

func (id RunID) AttemptKey() string

AttemptKey returns CQDaemon attempt key.

func (RunID) Inner

func (id RunID) Inner() string

Inner is the part after "<LUCIProject>/" for use in UI.

func (RunID) InverseTS

func (id RunID) InverseTS() string

InverseTS of this Run. See RunID doc.

func (RunID) LUCIProject

func (id RunID) LUCIProject() string

LUCIProject this Run belongs to.

func (RunID) PublicID

func (id RunID) PublicID() string

PublicID returns the public representation of the RunID.

The format of a public ID is `projects/$luci-project/runs/$id`, where - luci-project is the name of the LUCI project the Run belongs to - id is an opaque key unique in the LUCI project.

func (RunID) Validate

func (id RunID) Validate() (err error)

Validate returns an error if Run ID is not valid.

If validate returns nil,

  • it means all other methods on RunID will work fine instead of panicking,
  • it doesn't mean Run ID is possible to generate using the MakeRunID. This is especially relevant in CV tests, where specifying short Run IDs is useful.

type RunIDs

type RunIDs []RunID

RunIDs is a convenience type to facilitate handling of run RunIDs.

func MakeRunIDs

func MakeRunIDs(ids ...string) RunIDs

MakeRunIDs returns RunIDs from list of strings.

func (RunIDs) ContainsSorted

func (ids RunIDs) ContainsSorted(id RunID) bool

ContainsSorted returns true if ids contain the given one.

func (*RunIDs) DelSorted

func (p *RunIDs) DelSorted(id RunID) bool

DelSorted removes the given ID if it exists.

DelSorted is a pointer receiver method, because it modifies slice itself.

func (RunIDs) DifferenceSorted

func (a RunIDs) DifferenceSorted(b RunIDs) RunIDs

DifferenceSorted returns all IDs in this slice and not the other one.

Both slices must be sorted. Doesn't modify input slices.

func (RunIDs) Equal

func (ids RunIDs) Equal(other RunIDs) bool

Equal checks if two ids are equal.

func (RunIDs) Index

func (ids RunIDs) Index(target RunID) int

Index returns the index of the first instance of the provided id.

Returns -1 if the provided id isn't present.

func (*RunIDs) InsertSorted

func (p *RunIDs) InsertSorted(id RunID)

InsertSorted adds given ID if not yet exists to the list keeping list sorted.

InsertSorted is a pointer receiver method, because it modifies slice itself.

func (RunIDs) Len

func (ids RunIDs) Len() int

func (RunIDs) Less

func (ids RunIDs) Less(i, j int) bool

sort.Interface copy-pasta.

func (RunIDs) Set

func (ids RunIDs) Set() map[RunID]struct{}

Set returns a new set of run IDs.

func (RunIDs) Swap

func (ids RunIDs) Swap(i, j int)

func (RunIDs) WithoutSorted

func (ids RunIDs) WithoutSorted(exclude RunIDs) RunIDs

WithoutSorted returns a subsequence of IDs without excluded IDs.

Both this and the excluded slices must be sorted.

If this and excluded IDs are disjoint, return this slice. Otherwise, returns a copy without excluded IDs.

type TQIfy

type TQIfy struct {
	// KnownRetry are expected errors which will result in HTTP 429 and retries.
	//
	// Retries may not happen if task queue configuration prevents it, e.g.
	// because task has exhausted its retry quota.
	//
	// KnownRetry and KnownIgnore should not match the same error, but if this
	// happens, Retry takes effect and KnownIgnore is ignored to avoid accidental
	// loss of tasks.
	//
	// Must contain only leaf errors, i.e. no annotated or MultiError objects.
	KnownRetry []error
	// KnownRetryTags are similar to `KnowRetry`, but are the expected tags that
	// the CV error should be tagged with.
	//
	// Must not contain `transient.Tag`.
	KnownRetryTags []errors.BoolTag
	// NeverRetry instructs TQ not to retry on any unexpected error.
	//
	// Transient error will be tagged with `tq.Ignore` while non-transient error
	// will be tagged with `tq.Fatal`. See the struct doc for what each tag means.
	//
	// Recommend to use this flag when tasks are executed periodically in short
	// interval (e.g. refresh config task) where as retrying failed task is not
	// necessary.
	//
	// Mutually exclusive with `KnownRetry` and `KnownRetryTags`.
	NeverRetry bool
	// KnownIgnore are expected errors which will result in HTTP 204 and no
	// retries.
	//
	// Must contain only leaf errors, i.e. no annotated or MultiError objects.
	KnownIgnore []error
	// KnownIgnoreTags are similar to `KnownIgnore`, but are the expected tags
	// that the CV error should be tagged with.
	//
	// Must not contain `transient.Tag`.
	KnownIgnoreTags []errors.BoolTag
}

TQIfy converts CV error semantics to server/TQ, and logs error if necessary.

Usage:

func tqHandler(ctx ..., payload...) error {
  err := doStuff(ctx, ...)
  return TQIfy{}.Error(ctx, err)
}

Given that:

  • TQ lib recognizes these error kinds:
  • tq.Ignore => HTTP 204, no retries
  • tq.Fatal => HTTP 202, no retries, but treated with alertable in our monitoring configuration;
  • transient.Tag => HTTP 500, will be retried;
  • else => HTTP 429, will be retried.

OTOH, CV uses

  • transient.Tag to treat all _transient_ situations, where retry should help
  • else => permanent errors, where retries aren't helpful.

Most _transient_ situations in CV are due to expected issues such as Gerrit giving stale data. Getting HTTP 500s in this case is an unfortunate noise, which obscures other infrequent situations which are worth looking at.

func (TQIfy) Error

func (t TQIfy) Error(ctx context.Context, err error) error

type TryjobID

type TryjobID int64

TryjobID is a unique ID of a Tryjob used internally in CV.

This ID is not a Buildbucket Build ID. See also tryjob.Tryjob type.

type TryjobIDSet

type TryjobIDSet map[TryjobID]struct{}

TryjobIDSet is convenience type to reduce the boilerplate.

func (TryjobIDSet) Add

func (s TryjobIDSet) Add(tjID TryjobID)

Add adds the provided Tryjob ID to the set.

func (TryjobIDSet) Has

func (s TryjobIDSet) Has(tjID TryjobID) bool

Has returns true if the provided Tryjob ID is in the set.

Otherwise, returns false.

type TryjobIDs

type TryjobIDs []TryjobID

TryjobIDs is a convenience type to facilitate handling of a slice of TryjobID.

func MakeTryjobIDs

func MakeTryjobIDs(ids ...int64) TryjobIDs

MakeTryjobIDs returns TryjobIDs from list of TryjobID in int64.

func (TryjobIDs) Contains

func (ids TryjobIDs) Contains(id TryjobID) bool

Contains returns true if TryjobID is inside these TryjobIDs.

func (*TryjobIDs) Dedupe

func (p *TryjobIDs) Dedupe()

Dedupe removes duplicates in place and sorts the slice.

Note: Does not preserve original order.

func (TryjobIDs) Len

func (ids TryjobIDs) Len() int

Len is the number of elements in the collection.

func (TryjobIDs) Less

func (ids TryjobIDs) Less(i int, j int) bool

Less reports whether the element with index i should sort before the element with index j.

func (TryjobIDs) Set

func (ids TryjobIDs) Set() map[TryjobID]struct{}

Set returns a new set of TryjobIDs.

func (TryjobIDs) Swap

func (ids TryjobIDs) Swap(i int, j int)

Swap swaps the elements with indexes i and j.

func (TryjobIDs) ToInt64

func (ids TryjobIDs) ToInt64() []int64

ToInt64 returns a slice that contains all Tryjobs in int64 type.

Directories

Path Synopsis
Package bq handles sending rows to BigQuery.
Package bq handles sending rows to BigQuery.
Package eventbox batches incoming events for a single Datastore entity for processing.
Package eventbox batches incoming events for a single Datastore entity for processing.
dsset
Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
Package pubsub provides a generic way to batch pubsub pull notifications.
Package pubsub provides a generic way to batch pubsub pull notifications.
Package tree implements fetching tree status from Tree Status App.
Package tree implements fetching tree status from Tree Status App.
treetest
Package treetest implements fake Tree for testing in CV.
Package treetest implements fake Tree for testing in CV.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL