run

package
v0.0.0-...-51f9457 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2021 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package run implements public interface for Run Manager.

Index

Constants

View Source 
const RunCLKind = "RunCL"

RunKind is the Datastore entity kind for RunCL.

View Source 
const RunKind = "Run"

RunKind is the Datastore entity kind for Run.

Variables

View Source 
var (
	Status_name = map[int32]string{
		0:  "STATUS_UNSPECIFIED",
		1:  "PENDING",
		2:  "RUNNING",
		4:  "WAITING_FOR_SUBMISSION",
		5:  "SUBMITTING",
		64: "ENDED_MASK",
		65: "SUCCEEDED",
		66: "FAILED",
		67: "CANCELLED",
	}
	Status_value = map[string]int32{
		"STATUS_UNSPECIFIED":     0,
		"PENDING":                1,
		"RUNNING":                2,
		"WAITING_FOR_SUBMISSION": 4,
		"SUBMITTING":             5,
		"ENDED_MASK":             64,
		"SUCCEEDED":              65,
		"FAILED":                 66,
		"CANCELLED":              67,
	}
)

Enum value maps for Status.

View Source 
var File_go_chromium_org_luci_cv_internal_run_storage_proto protoreflect.FileDescriptor

Functions

func EventboxRecipient 

func EventboxRecipient(ctx context.Context, runID common.RunID) eventbox.Recipient

EventboxRecipient returns eventbox.Recipient for a given Run.

func IsEnded 

func IsEnded(status Status) bool

IsEnded returns true if the given status is final.

func NewQueryWithLUCIProject 

func NewQueryWithLUCIProject(ctx context.Context, project string) *datastore.Query

NewQueryWithLUCIProject returns a query for Run entities that belongs to the given LUCI Project.

Types

type Mode 

type Mode string

Mode dictates the behavior of this Run.

The end goal is to have arbitrary user-defined Mode names. For now, CQDaemon/LUCI CV operates with just 3 pre-defined modes, whose values are fixed based on legacy CQ BQ export. 

const (
	// DryRun triggers configured Tryjobs, but doesn't submit.
	DryRun Mode = "DRY_RUN"
	// FullRun is DryRun followed by submit.
	FullRun Mode = "FULL_RUN"
	// QuickDryRun is like DryRun but different thus allowing either different or
	// faster yet less thorough Tryjobs.
	QuickDryRun Mode = "QUICK_DRY_RUN"
)

func ModeFromBQAttempt 

func ModeFromBQAttempt(m bqpb.Mode) (Mode, error)

ModeFromBQAttempt returns Mode from CQ BQ export.

func (Mode) BQAttemptMode 

func (m Mode) BQAttemptMode() bqpb.Mode

BQAttemptMode returns corresponding value for legacy CQ BQ export.

type Notifier 

type Notifier struct {
	// TasksBinding are used to register handlers of RM implementation to avoid
	// circular dependency.
	TasksBinding eventpb.TasksBinding
}

Notifier notifies Run Manager.

func NewNotifier 

func NewNotifier(tqd *tq.Dispatcher) *Notifier

func (*Notifier) Cancel 

func (n *Notifier) Cancel(ctx context.Context, runID common.RunID) error

Cancel tells RunManager to cancel the given Run.

TODO(yiwzhang,tandrii): support reason.

func (*Notifier) CancelAt 

func (n *Notifier) CancelAt(ctx context.Context, runID common.RunID, eta time.Time) error

CancelAt tells RunManager to cancel the given Run at `eta`.

TODO(crbug/1141880): Remove this API after migration. This is only needed because CV need to delay the cancellation of a Run when waiting for CQD report finished Run when CQD is in charge.

func (*Notifier) Invoke 

func (n *Notifier) Invoke(ctx context.Context, runID common.RunID, eta time.Time) error

Invoke invokes Run Manager to process events at the provided `eta`.

If the provided `eta` is zero, invokes immediately.

func (*Notifier) NotifyCLSubmitted 

func (n *Notifier) NotifyCLSubmitted(ctx context.Context, runID common.RunID, clid common.CLID) error

NotifyCLSubmitted informs RunManager that the provided CL is submitted.

Unlike other event-sending funcs, this function only delivers the event to Run's eventbox, but does not dispatch the task. This is because it is okay to process all events of this kind together to record the submission result for each individual CLs after submission completes. Waking up RM unnecessarily may increase the contention of Run entity.

func (*Notifier) NotifyCLUpdated 

func (n *Notifier) NotifyCLUpdated(ctx context.Context, runID common.RunID, clid common.CLID, eVersion int) error

NotifyCLUpdated informs RunManager that given CL has a new version available.

func (*Notifier) NotifyCQDFinished 

func (n *Notifier) NotifyCQDFinished(ctx context.Context, runID common.RunID) error

NotifyCQDFinished tells RunManager that CQDaemon has finished the provided Run.

TODO(crbug/1224170): Remove this event after migration.

func (*Notifier) NotifyCQDVerificationCompleted 

func (n *Notifier) NotifyCQDVerificationCompleted(ctx context.Context, runID common.RunID) error

NotifyCQDVerificationCompleted tells RunManager that CQDaemon has completed verifying the provided Run.

TODO(crbug/1141880): Remove this event after migration.

func (*Notifier) NotifyReadyForSubmission 

func (n *Notifier) NotifyReadyForSubmission(ctx context.Context, runID common.RunID, eta time.Time) error

NotifyReadyForSubmission informs RunManager that the provided Run will be ready for submission at `eta`.

func (*Notifier) NotifySubmissionCompleted 

func (n *Notifier) NotifySubmissionCompleted(ctx context.Context, runID common.RunID, sc *eventpb.SubmissionCompleted, invokeRM bool) error

NotifySubmissionCompleted informs RunManager that the submission of the provided Run has completed.

func (*Notifier) PokeAfter 

func (n *Notifier) PokeAfter(ctx context.Context, runID common.RunID, after time.Duration) error

PokeAfter tells RunManager to check its own state after the given duration.

Providing a non-positive duration is equivalent to `PokeNow(...)`.

func (*Notifier) PokeAt 

func (n *Notifier) PokeAt(ctx context.Context, runID common.RunID, eta time.Time) error

PokeAt tells RunManager to check its own state at around `eta`.

Guarantees no earlier than `eta` but may not be exactly at `eta`. Providing an earlier timestamp than the current is equivalent to `PokeNow(...)`.

func (*Notifier) PokeNow 

func (n *Notifier) PokeNow(ctx context.Context, runID common.RunID) error

PokeNow tells RunManager to check its own state immediately.

It's a shorthand of `PokeAfter(ctx, runID, after)` where `after` <= 0 or `PokeAt(ctx, runID, eta)` where `eta` is an earlier timestamp.

func (*Notifier) Send 

func (n *Notifier) Send(ctx context.Context, runID common.RunID, evt *eventpb.Event, eta time.Time) error

Send sends the event to Run's eventbox and invokes RunManager at `eta`.

func (*Notifier) SendNow 

func (n *Notifier) SendNow(ctx context.Context, runID common.RunID, evt *eventpb.Event) error

SendNow sends the event to Run's eventbox and invokes RunManager immediately.

func (*Notifier) Start 

func (n *Notifier) Start(ctx context.Context, runID common.RunID) error

Start tells RunManager to start the given run.

func (*Notifier) UpdateConfig 

func (n *Notifier) UpdateConfig(ctx context.Context, runID common.RunID, hash string, eversion int64) error

UpdateConfig tells RunManager to update the given Run to new config.

type Options 

type Options struct {

	// If true, submitting the Run isn't blocked on open tree.
	//
	// If false (default), respects project configuration.
	SkipTreeChecks bool `protobuf:"varint,1,opt,name=skip_tree_checks,json=skipTreeChecks,proto3" json:"skip_tree_checks,omitempty"`
	// If true, `builders.equivalent_to{...}` sections are ignored when triggering
	// tryjobs.
	//
	// If false (default), respects project configuration.
	SkipEquivalentBuilders bool `` /* 130-byte string literal not displayed */
	// If true, no longer useful tryjobs won't be cancelled.
	//
	// If false (default), respects project configuration.
	AvoidCancellingTryjobs bool `` /* 130-byte string literal not displayed */
	// If true, no tryjobs will be triggered except "presubmit" regardless of
	// project configuration.
	//
	// "presubmit" builders are legacy which are currently configured with
	// "disable_reuse: true" in project config. To skip triggering them,
	// skip_presubmit must be set to true.
	// TODO(https://crbug.com/950074): ignore.
	//
	// If false (default), respects project configuration.
	SkipTryjobs bool `protobuf:"varint,4,opt,name=skip_tryjobs,json=skipTryjobs,proto3" json:"skip_tryjobs,omitempty"`
	// Deprecated per https://crbug.com/950074.
	// See skip_tryjobs doc.
	SkipPresubmit bool `protobuf:"varint,5,opt,name=skip_presubmit,json=skipPresubmit,proto3" json:"skip_presubmit,omitempty"`
	// contains filtered or unexported fields
}

Options are Run-specific additions on top of LUCI project config.

func ExtractOptions 

func ExtractOptions(clDescription string) *Options

ExtractOptions computes the Run Options from 1 CL description.

func MergeOptions 

func MergeOptions(a, b *Options) *Options

MergeOptions merges two Run Options.

Does not modify the passed object, but may return either of them.

func (*Options) Descriptor deprecated 

func (*Options) Descriptor() ([]byte, []int)

Deprecated: Use Options.ProtoReflect.Descriptor instead.

func (*Options) GetAvoidCancellingTryjobs 

func (x *Options) GetAvoidCancellingTryjobs() bool

func (*Options) GetSkipEquivalentBuilders 

func (x *Options) GetSkipEquivalentBuilders() bool

func (*Options) GetSkipPresubmit 

func (x *Options) GetSkipPresubmit() bool

func (*Options) GetSkipTreeChecks 

func (x *Options) GetSkipTreeChecks() bool

func (*Options) GetSkipTryjobs 

func (x *Options) GetSkipTryjobs() bool

func (*Options) ProtoMessage 

func (*Options) ProtoMessage()

func (*Options) ProtoReflect 

func (x *Options) ProtoReflect() protoreflect.Message

func (*Options) Reset 

func (x *Options) Reset()

func (*Options) String 

func (x *Options) String() string

type Run 

type Run struct {

	// ID is the RunID generated at triggering time.
	//
	// See doc for type `common.RunID` about the format.
	ID common.RunID `gae:"$id"`
	// CreationOperationID is a string used to de-dup Run creation attempts.
	CreationOperationID string `gae:",noindex"`
	// Mode dictates the behavior of this Run.
	Mode Mode `gae:",noindex"`
	// Status describes the status of this Run.
	Status Status
	// EVersion is the entity version.
	//
	// It increments by one upon every successful modification.
	EVersion int `gae:",noindex"`
	// CreateTime is the timestamp when this Run was created.
	//
	// For API triggered Run, the CreateTime is when CV processes the request.
	// For non-API triggered Run, the CreateTime is the timestamp of the last
	// vote on a Gerrit CL that triggers this Run.
	CreateTime time.Time `gae:",noindex"`
	// StartTime is the timestamp when this Run was started.
	StartTime time.Time `gae:",noindex"`
	// UpdateTime is the timestamp when this entity was last updated.
	UpdateTime time.Time `gae:",noindex"`
	// EndTime is the timestamp when this Run has completed.
	EndTime time.Time `gae:",noindex"`
	// Owner is the identity of the owner of this Run.
	//
	// Currently, it is the same as owner of the CL. If `combine_cls` is
	// enabled for the ConfigGroup used by this Run, the owner is the CL which
	// has the latest triggering timestamp.
	Owner identity.Identity `gae:",noindex"`
	// ConfigGroupID is ID of the ConfigGroup that is used by this Run.
	//
	// RunManager may update the ConfigGroup in the middle of the Run if it is
	// notified that a new version of Config has been imported into CV.
	ConfigGroupID prjcfg.ConfigGroupID `gae:",noindex"`
	// CLs are IDs of all CLs involved in this Run.
	CLs common.CLIDs `gae:",noindex"`
	// Options are Run-specific additions on top of LUCI project config.
	Options *Options
	// Submission is the current state of Run Submission.
	//
	// If set, Submission is in progress or has completed.
	Submission *Submission

	// LatestCLsRefresh is the latest time when Run Manager scheduled async
	// refresh of CLs.
	LatestCLsRefresh time.Time `gae:",noindex"`

	// CQAttemptKey is what CQDaemon exports to BigQuery as Attempt's key.
	//
	// In CQDaemon's source, it's equivalent to GerritAttempt.attempt_key_hash.
	CQDAttemptKey string
	// FinalizedByCQD is true iff the Run was finalized by CQDaemon, which
	// includes submitting CLs and/or removing CQ votes and sending BQ row.
	FinalizedByCQD bool `gae:",noindex"`
	// contains filtered or unexported fields
}

Run is an entity that contains high-level information about a CV Run.

In production, Run entities are created only by the "runcreator" package.

Detailed information about CLs and Tryjobs are stored in its child entities.

func (*Run) Mutate 

func (r *Run) Mutate(ctx context.Context, mut func(*Run) (updated bool)) (updated bool)

Mutate mutates the Run by executing `mut`.

It ensures EVersion and UpdateTime are correctly updated if `mut` has changed the Run.

type RunCL 

type RunCL struct {
	ID         common.CLID           `gae:"$id"`
	Run        *datastore.Key        `gae:"$parent"`
	ExternalID changelist.ExternalID `gae:",noindex"`
	Detail     *changelist.Snapshot
	Trigger    *Trigger

	// IndexedID is a copy of ID to get an index on just the CLID,
	// as the primary automatic index is on (Run(parent), ID).
	IndexedID common.CLID
	// contains filtered or unexported fields
}

RunCL is an immutable snapshot of a CL at the time of the Run start.

func LoadRunCLs 

func LoadRunCLs(ctx context.Context, runID common.RunID, clids common.CLIDs) ([]*RunCL, error)

LoadRunCLs loads `RunCL` entities of the provided cls in the Run.

type RunOwner 

type RunOwner struct {

	// ID is the user identity.
	ID identity.Identity `gae:"$id"`
	// ActiveRuns are all Runs triggered by this user that are active.
	ActiveRuns common.RunIDs `gae:",noindex"`
	// PendingRuns are all Runs triggered by this user that are
	// yet-to-be-launched (i.e. quota doesn't permit).
	PendingRuns common.RunIDs `gae:",noindex"`
	// contains filtered or unexported fields
}

RunOwner keeps tracks of all open (active or pending) Runs for a user.

type Status 

type Status int32

Status describes the status of a CV Run. 

const (
	// Unspecified status.
	Status_STATUS_UNSPECIFIED Status = 0
	// Run is pending to start.
	//
	// It is either because Run Manager hasn't processed the StartEvent yet or
	// the RunOwner has exhausted all the quota and waiting for new quota to
	// be available.
	Status_PENDING Status = 1
	// Run is running.
	Status_RUNNING Status = 2
	// Run is waiting for submission.
	//
	// Run is in this status if one of the following scenario is true:
	//   1. Tree is closed at the time Run attempts to submit.
	//   2. There is another Run in the same LUCI Project that is currently
	//      submitting.
	//   3. The submission is rate limited according to the submit option in
	//      Project Config.
	//
	// This status is cancellable.
	Status_WAITING_FOR_SUBMISSION Status = 4
	// Run is submitting.
	//
	// A Run can't be cancelled while submitting. A Run may transition from
	// this status to either `WAITING_FOR_SUBMISSION` status or a non-cancelled
	// terminal status.
	Status_SUBMITTING Status = 5
	// ENDED_MASK can be used as a bitmask to check if a Run has ended.
	// This MUST NOT be used as the status of a Run.
	Status_ENDED_MASK Status = 64
	// Run ends successfully.
	Status_SUCCEEDED Status = 65
	// Run ends unsuccessfully.
	Status_FAILED Status = 66
	// Run is cancelled.
	Status_CANCELLED Status = 67
)

func (Status) Descriptor 

func (Status) Descriptor() protoreflect.EnumDescriptor

func (Status) Enum 

func (x Status) Enum() *Status

func (Status) EnumDescriptor deprecated 

func (Status) EnumDescriptor() ([]byte, []int)

Deprecated: Use Status.Descriptor instead.

func (Status) Number 

func (x Status) Number() protoreflect.EnumNumber

func (Status) String 

func (x Status) String() string

func (Status) Type 

func (Status) Type() protoreflect.EnumType

type Submission 

type Submission struct {

	// The deadline of this submission.
	//
	// If the deadline is not set or has already expired, a RunManager task
	// can claim the exclusive privilege by setting the deadline to a future
	// timestamp (generally, end of task deadline).
	Deadline *timestamppb.Timestamp `protobuf:"bytes,1,opt,name=deadline,proto3" json:"deadline,omitempty"`
	// ID of the task that executes this submission.
	TaskId string `protobuf:"bytes,2,opt,name=task_id,json=taskId,proto3" json:"task_id,omitempty"`
	// IDs of all CLs that should be submitted in this submission.
	//
	// Must be ordered in submission order.
	Cls []int64 `protobuf:"varint,3,rep,packed,name=cls,proto3" json:"cls,omitempty"`
	// IDs of all CLs that have been submitted successfully already.
	SubmittedCls []int64 `protobuf:"varint,4,rep,packed,name=submitted_cls,json=submittedCls,proto3" json:"submitted_cls,omitempty"`
	// IDs of all CLs that fails to submit if any.
	//
	// CLs that are neither in this list nor in the `submitted_cls` should be
	// treated as if CV has never attempted to submit them.
	//
	// This could be empty even when the entire submission fails, which would be
	// typically caused by faulty infrastructure (e.g. Task Queue not executing
	// a Run Manager task before the whole submission timeout is reached).
	FailedCls []int64 `protobuf:"varint,5,rep,packed,name=failed_cls,json=failedCls,proto3" json:"failed_cls,omitempty"`
	// If True, Tree is currently in open state.
	TreeOpen bool `protobuf:"varint,10,opt,name=tree_open,json=treeOpen,proto3" json:"tree_open,omitempty"`
	// The timestamp when the Tree status was last fetched.
	LastTreeCheckTime *timestamppb.Timestamp `protobuf:"bytes,11,opt,name=last_tree_check_time,json=lastTreeCheckTime,proto3" json:"last_tree_check_time,omitempty"`
	// contains filtered or unexported fields
}

Submission describes the current state of Run submission.

func (*Submission) Descriptor deprecated 

func (*Submission) Descriptor() ([]byte, []int)

Deprecated: Use Submission.ProtoReflect.Descriptor instead.

func (*Submission) GetCls 

func (x *Submission) GetCls() []int64

func (*Submission) GetDeadline 

func (x *Submission) GetDeadline() *timestamppb.Timestamp

func (*Submission) GetFailedCls 

func (x *Submission) GetFailedCls() []int64

func (*Submission) GetLastTreeCheckTime 

func (x *Submission) GetLastTreeCheckTime() *timestamppb.Timestamp

func (*Submission) GetSubmittedCls 

func (x *Submission) GetSubmittedCls() []int64

func (*Submission) GetTaskId 

func (x *Submission) GetTaskId() string

func (*Submission) GetTreeOpen 

func (x *Submission) GetTreeOpen() bool

func (*Submission) ProtoMessage 

func (*Submission) ProtoMessage()

func (*Submission) ProtoReflect 

func (x *Submission) ProtoReflect() protoreflect.Message

func (*Submission) Reset 

func (x *Submission) Reset()

func (*Submission) String 

func (x *Submission) String() string

type Trigger 

type Trigger struct {
	Time *timestamppb.Timestamp `protobuf:"bytes,1,opt,name=time,proto3" json:"time,omitempty"`
	// Mode is string value of run.Mode.
	Mode string `protobuf:"bytes,2,opt,name=mode,proto3" json:"mode,omitempty"`
	// Additional label is recorded in case applicable ConfigGroup had additional
	// modes, e.g. for QUICK_DRY_RUN mode.
	AdditionalLabel string `protobuf:"bytes,5,opt,name=additional_label,json=additionalLabel,proto3" json:"additional_label,omitempty"`
	// Triggering user email if known.
	//
	// Gerrit doesn't guarantee that every user has set their preferred email,
	// but LUCI ACLs are based entirely on user emails. Thus, Runs with the email
	// unset are quickly aborted by CQDaemon.
	//
	// TODO(tandrii): once CQDaemon is deleted, require email to start a Run,
	// and remove Gerrit-specific gerrit_account_id.
	Email string `protobuf:"bytes,3,opt,name=email,proto3" json:"email,omitempty"`
	// Gerrit account ID. Always known.
	GerritAccountId int64 `protobuf:"varint,4,opt,name=gerrit_account_id,json=gerritAccountId,proto3" json:"gerrit_account_id,omitempty"`
	// contains filtered or unexported fields
}

Trigger describes who/how CV was triggered on a specific CL.

func (*Trigger) Descriptor deprecated 

func (*Trigger) Descriptor() ([]byte, []int)

Deprecated: Use Trigger.ProtoReflect.Descriptor instead.

func (*Trigger) GetAdditionalLabel 

func (x *Trigger) GetAdditionalLabel() string

func (*Trigger) GetEmail 

func (x *Trigger) GetEmail() string

func (*Trigger) GetGerritAccountId 

func (x *Trigger) GetGerritAccountId() int64

func (*Trigger) GetMode 

func (x *Trigger) GetMode() string

func (*Trigger) GetTime 

func (x *Trigger) GetTime() *timestamppb.Timestamp

func (*Trigger) ProtoMessage 

func (*Trigger) ProtoMessage()

func (*Trigger) ProtoReflect 

func (x *Trigger) ProtoReflect() protoreflect.Message

func (*Trigger) Reset 

func (x *Trigger) Reset()

func (*Trigger) String 

func (x *Trigger) String() string

Source Files

View all Source files

Directories

Path Synopsis
Package bq provides functionality for sending finished Run rows to BigQuery.
 Package bq provides functionality for sending finished Run rows to BigQuery.
Package eventpb stores protos for event processing of RunManager.
 Package eventpb stores protos for event processing of RunManager.
Package impl implements a RunManager.
 Package impl implements a RunManager.
handler
Package handler implements handlers that handles run events.
 Package handler implements handlers that handles run events.
state
Package state defines the model for a Run state.
 Package state defines the model for a Run state.
submit
Package submit contains utilities for Run submission.
 Package submit contains utilities for Run submission.
Package runcreator creates new Runs.
 Package runcreator creates new Runs.
Package runtest implements tests for working with Run Manager.
 Package runtest implements tests for working with Run Manager.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.