model

package
v0.0.0-...-d60a78d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2023 License: Apache-2.0 Imports: 20 Imported by: 4

Documentation

Overview

Package model contains all of DM's datastore models.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ActivateExecution 

func ActivateExecution(c context.Context, auth *dm.Execution_Auth, actToken []byte) (a *Attempt, e *Execution, err error)

ActivateExecution validates that the execution is unactivated and that the activation token matches and then sets the token to the new value.

It's OK to retry this. Subsequent invocations with the same Token will recognize this case and not return an error.

func AttemptIDFromKey 

func AttemptIDFromKey(k *ds.Key) *dm.Attempt_ID

AttemptIDFromKey makes a AttemptID from the given datastore.Key. It panics if the Key does not point to a Attempt.

func AttemptKeyFromID 

func AttemptKeyFromID(c context.Context, aid *dm.Attempt_ID) *ds.Key

AttemptKeyFromID makes a datastore.Key given the AttemptID.

func AuthenticateExecution 

func AuthenticateExecution(c context.Context, auth *dm.Execution_Auth) (a *Attempt, e *Execution, err error)

AuthenticateExecution verifies that the Attempt is executing, and that evkey matches the execution key of the current Execution for this Attempt.

As a bonus, it will return the loaded Attempt and Execution.

func ExecutionIDFromKey 

func ExecutionIDFromKey(k *ds.Key) *dm.Execution_ID

ExecutionIDFromKey makes a ExecutionID from the given datastore.Key. It panics if the Key does not point to a Execution.

func ExecutionKeyFromID 

func ExecutionKeyFromID(c context.Context, eid *dm.Execution_ID) *ds.Key

ExecutionKeyFromID makes a datastore.Key given the ExecutionID.

func FwdDepKeysFromList 

func FwdDepKeysFromList(c context.Context, base *dm.Attempt_ID, list *dm.AttemptList) []*ds.Key

FwdDepKeysFromList makes a list of datastore.Key's that correspond to all of the FwdDeps expressed by the <base, list> pair.

func InvalidateExecution 

func InvalidateExecution(c context.Context, auth *dm.Execution_Auth) (a *Attempt, e *Execution, err error)

InvalidateExecution verifies that the execution key is valid, and then revokes the execution key.

As a bonus, it will return the loaded Attempt and Execution.

func MakeRandomToken 

func MakeRandomToken(c context.Context, l uint32) []byte

MakeRandomToken creates a cryptographically random byte slice of the specified length. It panics if the specified length cannot be read in full.

func QueryAttemptsForQuest 

func QueryAttemptsForQuest(c context.Context, qid string) *ds.Query

QueryAttemptsForQuest returns all Attempt objects that exist for this Quest.

func QuestIDFromKey 

func QuestIDFromKey(k *ds.Key) string

QuestIDFromKey makes a QuestID from the given datastore.Key. It panics if the Key does not point to a Quest.

func QuestKeyFromID 

func QuestKeyFromID(c context.Context, qid string) *ds.Key

QuestKeyFromID makes a datastore.Key given the QuestID.

Types

type Attempt 

type Attempt struct {
	ID dm.Attempt_ID `gae:"$id"`

	Created  time.Time
	Modified time.Time

	State      dm.Attempt_State
	RetryState AttemptRetryState

	// IsAbnormal is true iff State==ABNORMAL_FINISHED, used for walk_graph.
	IsAbnormal bool

	// A lazily-updated boolean to reflect that this Attempt is expired for
	// queries.
	IsExpired bool

	// Contains either data (State==FINISHED) or abnormal_finish (State==ABNORMAL_FINISHED)
	//
	// Does not contain the `data.object` field (which is in the AttemptResult,1 object)
	Result dm.Result `gae:",noindex"`

	// CurExecution is the maximum Execution ID for this Attempt so far. Execution
	// IDs are contiguous from [1, CurExecution]. If the State is not currently
	// Executing, then CurExecution represents the execution that JUST finished
	// (or 0 if no Executions have been made yet).
	CurExecution uint32

	// LastSuccessfulExecution is the execution ID of the last successful
	// execution, or 0 if no such execution occured yet.
	LastSuccessfulExecution uint32

	// DepMap is valid only while Attempt is in a State of EXECUTING or WAITING.
	//
	// The size of this field is inspected to deteremine what the next state after
	// EXECUTING is. If the size == 0, it means the Attempt should move to the
	// FINISHED state. Otherwise it means that the Attempt should move to the
	// WAITING state.
	//
	// A bit field value of 0 means that the dep is currently waiting, and a bit
	// value of 1 means that the coresponding dep is satisfined. The Attempt can
	// be unblocked from WAITING back to SCHEDULING when all bits are set to 1.
	DepMap bf.BitField `gae:",noindex" json:"-"`
}

Attempt is the datastore model for a DM Attempt. It has no parent key, but it may have the following children entities:

  • FwdDep
  • AttemptResult

Additionally, every Attempt has an associated BackDepGroup whose ID equals the ID of this Attempt.

func AttemptFromID 

func AttemptFromID(aid *dm.Attempt_ID) *Attempt

AttemptFromID produces an empty Attempt model from the AttemptID.

func MakeAttempt 

func MakeAttempt(c context.Context, aid *dm.Attempt_ID) *Attempt

MakeAttempt is a convenience function to create a new Attempt model in the NeedsExecution state.

func (*Attempt) DataProto 

func (a *Attempt) DataProto() (ret *dm.Attempt_Data)

DataProto returns an Attempt.Data message for this Attempt.

func (*Attempt) ModifyState 

func (a *Attempt) ModifyState(c context.Context, newState dm.Attempt_State) error

ModifyState changes the current state of this Attempt and updates its Modified timestamp.

func (*Attempt) ToProto 

func (a *Attempt) ToProto(withData bool) *dm.Attempt

ToProto returns a dm proto version of this Attempt.

type AttemptResult 

type AttemptResult struct {
	Attempt *datastore.Key `gae:"$parent"`

	// The sizes and expirations are denormalized across Attempt and
	// AttemptResult.
	Data dm.JsonResult `gae:",noindex"`
	// contains filtered or unexported fields
}

AttemptResult holds the raw, compressed json blob returned from the execution.

type AttemptRetryState 

type AttemptRetryState struct {
	Failed   uint32
	Expired  uint32
	TimedOut uint32
	Crashed  uint32
}

AttemptRetryState indicates the current state of the Attempt's retry counters.

func (*AttemptRetryState) Reset 

func (a *AttemptRetryState) Reset()

Reset resets all of the AttemptRetryState counters.

type BackDep 

type BackDep struct {
	// The attempt id of the attempt that's depending on this dependee.
	Depender dm.Attempt_ID `gae:"$id"`

	// The BackdepGroup for the attempt that is being depended on.
	DependeeGroup *datastore.Key `gae:"$parent"`

	// Propagated is true if the BackDepGroup has AttemptFinished, and this
	// BackDep has been processed by the mutate.RecordCompletion tumble
	// mutation. So if with two attempts A and B, A depends on B, the
	// BackDep{DependeeGroup: B, Depender: A} has Propagated as true when B is
	// finished, and a tumble Mutation has been launched to inform A of that fact.
	Propagated bool
}

BackDep represents a single backwards dependency. Its ID is the same as the Attempt that's depending on this one. See BackDepGroup for more context.

func (*BackDep) Edge 

func (b *BackDep) Edge() *FwdEdge

Edge produces a fwdedge object which points from the depending attempt to the depended-on attempt.

type BackDepGroup 

type BackDepGroup struct {
	// Dependee is the "<AttemptID>" that the deps in this group point
	// back FROM.
	Dependee dm.Attempt_ID `gae:"$id"`

	// This is a denormalized version of Attempt.State, used to allow
	// transactional additions to the BackDepGroup to stay within this Entity
	// Group when adding new back deps.
	AttemptFinished bool
}

BackDepGroup describes a group of reverse dependencies ('depended-by') between Attempts. Its ID is the same as the id of the Attempt that's being depended-on by other attempts, and it serves as the parent entity for the BackDep model. So: 

Attempt(OTHER_QUEST|2)
  FwdDep(QUEST|1)

Attempt(QUEST|1)

BackDepGroup(QUEST|1)
  BackDep(OTHER_QUEST|2)

Represents the OTHER_QUEST|2 depending on QUEST|1.

type Execution 

type Execution struct {
	ID      invertedHexUint32 `gae:"$id"`
	Attempt *ds.Key           `gae:"$parent"`

	Created  time.Time
	Modified time.Time

	// DistributorConfigName is redundant with the Quest definition, but this
	// helps avoid extra unnecessary datastore round-trips to load the Quest.
	DistributorConfigName    string
	DistributorConfigVersion string
	DistributorToken         string

	State dm.Execution_State

	// IsAbnormal is true iff State==ABNORMAL_FINISHED. Used for walk_graph.
	IsAbnormal bool

	// A lazily-updated boolean to reflect that this Execution is expired for
	// queries.
	IsExpired bool

	// Contains either data (State==FINISHED) or abnormal_finish (State==ABNORMAL_FINISHED)
	Result dm.Result `gae:",noindex"`

	// These are DM's internal mechanism for performing timeout actions on
	// Executions.
	//
	// The TimeTo* variables are copied from the quest description.
	//
	// The Timeout is only active when the Execution is in a non-terminal state.
	TimeToStart time.Duration `gae:",noindex"` // timeouts.start
	TimeToRun   time.Duration `gae:",noindex"` // timeouts.run
	TimeToStop  time.Duration `gae:",noindex"` // pollTimeout || timeouts.stop

	// Token is a randomized nonce that's used to verify that RPCs verify from the
	// expected client (the client that's currently running the Execution). The
	// Token has 2 modes.
	//
	// When the Execution is handed to the distributor, the Token is randomly
	// generated by DM and passed to the distributor. The State of the Execution
	// starts as SCHEDULED. This token may be used by the client to "activate" the
	// Execution with the ActivateExecution rpc. At that point, the client
	// provides a new random token, the Execution State moves from SCHEDULED to
	// RUNNING, and Token assumes the new value. As long as the Execution State is
	// RUNNING, the client may continue to use that new Token value to
	// authenticate other rpc's like AddDeps and FinishAttempt.
	//
	// As soon as the Execution is in the STOPPING, ABNORMAL_FINISHED or FINISHED
	// state, this will be nil'd out.
	Token []byte `gae:",noindex"`
}

Execution represents either an ongoing execution on the Quest's specified distributor, or is a placeholder for an already-completed Execution.

func ExecutionFromID 

func ExecutionFromID(c context.Context, eid *dm.Execution_ID) *Execution

ExecutionFromID produces an empty Execution model from the ExecutionID.

func MakeExecution 

func MakeExecution(c context.Context, e *dm.Execution_ID, cfgName, cfgVers string) *Execution

MakeExecution makes a new Execution in the SCHEDULING state, with a new random Token.

func (*Execution) DataProto 

func (e *Execution) DataProto() (ret *dm.Execution_Data)

DataProto returns an Execution.Data message for this Execution.

This omits the DistributorInfo.Url portion, which must be filled in elsewhere for package cyclical import reasons.

func (*Execution) GetEID 

func (e *Execution) GetEID() *dm.Execution_ID

GetEID gets an Execution_ID for this Execution. It panics if the Execution is in an invalid state.

func (*Execution) ModifyState 

func (e *Execution) ModifyState(c context.Context, newState dm.Execution_State) error

ModifyState changes the current state of this Execution and updates its Modified timestamp.

func (*Execution) Revoke 

func (e *Execution) Revoke(c context.Context) error

Revoke will clear the Token and Put this Execution to the datastore. This action requires the Execution to be in the RUNNING state, and causes it to enter the STOPPING state.

func (*Execution) ToProto 

func (e *Execution) ToProto(includeID bool) *dm.Execution

ToProto returns a dm proto version of this Execution.

type FwdDep 

type FwdDep struct {
	// Attempt that this points from.
	Depender *ds.Key `gae:"$parent"`

	// A FwdDep's ID is the Attempt ID that it points to.
	Dependee dm.Attempt_ID `gae:"$id"`

	// This will be used to set a bit in the Attempt (WaitingDepBitmap) when the
	// Dep completes.
	BitIndex uint32

	// ForExecution indicates which Execution added this dependency. This is used
	// for validation of AckFwdDep mutations to ensure that they're operating
	// on an Attempt in the correct state, but can also be used for historical
	// analysis/display.
	ForExecution uint32
}

FwdDep describes a 'depends-on' relation between two Attempts. It has a reciprocal BackDep as well, which notes the depended-on-by relationship. So: 

Attempt(OTHER_QUEST|2)
  FwdDep(QUEST|1)

Attempt(QUEST|1)

BackDepGroup(QUEST|1)
  BackDep(OTHER_QUEST|2)

Represents the OTHER_QUEST|2 depending on QUEST|1.

func FwdDepsFromList 

func FwdDepsFromList(c context.Context, base *dm.Attempt_ID, list *dm.AttemptList) []*FwdDep

FwdDepsFromList creates a slice of *FwdDep given an originating base Attempt_ID, and a list of dependency Attempts.

func (*FwdDep) Edge 

func (f *FwdDep) Edge() *FwdEdge

Edge produces a edge object which points 'forwards' from the depending attempt to the depended-on attempt.

type FwdEdge 

type FwdEdge struct {
	From *dm.Attempt_ID
	To   *dm.Attempt_ID
}

FwdEdge represents a forward-dependency from one attempt to another. The From attempt will block until the To attempt completes.

func (*FwdEdge) Back 

func (e *FwdEdge) Back(c context.Context) (*BackDepGroup, *BackDep)

Back returns the BackDepGroup (To) and the BackDep (From) models that represent the reverse dependency for this FwdEdge.

func (*FwdEdge) Fwd 

func (e *FwdEdge) Fwd(c context.Context) (*Attempt, *FwdDep)

Fwd returns the Attempt (From) and FwdDep (To) models that this FwdEdge represents.

type Quest 

type Quest struct {
	// ID is the base64 sha256 of questPayload
	ID string `gae:"$id"`

	Desc    dm.Quest_Desc `gae:",noindex"`
	BuiltBy TemplateInfo  `gae:",noindex"`

	Created time.Time `gae:",noindex"`
}

Quest is the model for a job-to-run. Its questPayload should fully describe the job to be done.

func NewQuest 

func NewQuest(c context.Context, desc *dm.Quest_Desc) *Quest

NewQuest builds a new Quest object with a correct ID given the current contents of the Quest_Desc. It returns an error if the Desc is invalid.

Desc must already be Normalize()'d

func QuestFromID 

func QuestFromID(qid string) *Quest

QuestFromID produces an empty Quest model from the QuestID.

func (*Quest) DataProto 

func (q *Quest) DataProto() *dm.Quest_Data

DataProto gets the Quest.Data proto message for this Quest.

func (*Quest) Equals 

func (q *Quest) Equals(a *Quest) bool

Equals is true if q and a and equal.

func (*Quest) ToProto 

func (q *Quest) ToProto() *dm.Quest

ToProto converts this Quest into its display equivalent.

type TemplateInfo 

type TemplateInfo []dm.Quest_TemplateSpec

TemplateInfo is an ordered list of dm.Quest_TemplateSpec's

func (*TemplateInfo) Add 

func (ti *TemplateInfo) Add(ts ...dm.Quest_TemplateSpec)

Add adds ts to the TemplateInfo uniq'ly.

func (TemplateInfo) Equals 

func (ti TemplateInfo) Equals(other TemplateInfo) bool

Equals returns true iff this TemplateInfo is exactly same as `other`.

func (TemplateInfo) EqualsData 

func (ti TemplateInfo) EqualsData(other []*dm.Quest_TemplateSpec) bool

EqualsData returns true iff this TemplateInfo has the same content as the proto-style TemplateInfo. This assumes that `other` is sorted.

func (TemplateInfo) Len 

func (ti TemplateInfo) Len() int

func (TemplateInfo) Less 

func (ti TemplateInfo) Less(i, j int) bool

func (TemplateInfo) Swap 

func (ti TemplateInfo) Swap(i, j int)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.