Documentation

Overview

    Package model contains all of DM's datastore models.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    func ActivateExecution

    func ActivateExecution(c context.Context, auth *dm.Execution_Auth, actToken []byte) (a *Attempt, e *Execution, err error)

      ActivateExecution validates that the execution is unactivated and that the activation token matches and then sets the token to the new value.

      It's OK to retry this. Subsequent invocations with the same Token will recognize this case and not return an error.

      func AttemptIDFromKey

      func AttemptIDFromKey(k *ds.Key) *dm.Attempt_ID

        AttemptIDFromKey makes a AttemptID from the given datastore.Key. It panics if the Key does not point to a Attempt.

        func AttemptKeyFromID

        func AttemptKeyFromID(c context.Context, aid *dm.Attempt_ID) *ds.Key

          AttemptKeyFromID makes a datastore.Key given the AttemptID.

          func AuthenticateExecution

          func AuthenticateExecution(c context.Context, auth *dm.Execution_Auth) (a *Attempt, e *Execution, err error)

            AuthenticateExecution verifies that the Attempt is executing, and that evkey matches the execution key of the current Execution for this Attempt.

            As a bonus, it will return the loaded Attempt and Execution.

            func ExecutionIDFromKey

            func ExecutionIDFromKey(k *ds.Key) *dm.Execution_ID

              ExecutionIDFromKey makes a ExecutionID from the given datastore.Key. It panics if the Key does not point to a Execution.

              func ExecutionKeyFromID

              func ExecutionKeyFromID(c context.Context, eid *dm.Execution_ID) *ds.Key

                ExecutionKeyFromID makes a datastore.Key given the ExecutionID.

                func FwdDepKeysFromList

                func FwdDepKeysFromList(c context.Context, base *dm.Attempt_ID, list *dm.AttemptList) []*ds.Key

                  FwdDepKeysFromList makes a list of datastore.Key's that correspond to all of the FwdDeps expressed by the <base, list> pair.

                  func InvalidateExecution

                  func InvalidateExecution(c context.Context, auth *dm.Execution_Auth) (a *Attempt, e *Execution, err error)

                    InvalidateExecution verifies that the execution key is valid, and then revokes the execution key.

                    As a bonus, it will return the loaded Attempt and Execution.

                    func MakeRandomToken

                    func MakeRandomToken(c context.Context, l uint32) []byte

                      MakeRandomToken creates a cryptographically random byte slice of the specified length. It panics if the specified length cannot be read in full.

                      func QueryAttemptsForQuest

                      func QueryAttemptsForQuest(c context.Context, qid string) *ds.Query

                        QueryAttemptsForQuest returns all Attempt objects that exist for this Quest.

                        func QuestIDFromKey

                        func QuestIDFromKey(k *ds.Key) string

                          QuestIDFromKey makes a QuestID from the given datastore.Key. It panics if the Key does not point to a Quest.

                          func QuestKeyFromID

                          func QuestKeyFromID(c context.Context, qid string) *ds.Key

                            QuestKeyFromID makes a datastore.Key given the QuestID.

                            Types

                            type Attempt

                            type Attempt struct {
                            	ID dm.Attempt_ID `gae:"$id"`
                            
                            	Created  time.Time
                            	Modified time.Time
                            
                            	State      dm.Attempt_State
                            	RetryState AttemptRetryState
                            
                            	// IsAbnormal is true iff State==ABNORMAL_FINISHED, used for walk_graph.
                            	IsAbnormal bool
                            
                            	// A lazily-updated boolean to reflect that this Attempt is expired for
                            	// queries.
                            	IsExpired bool
                            
                            	// Contains either data (State==FINISHED) or abnormal_finish (State==ABNORMAL_FINISHED)
                            	//
                            	// Does not contain the `data.object` field (which is in the AttemptResult,1 object)
                            	Result dm.Result `gae:",noindex"`
                            
                            	// CurExecution is the maximum Execution ID for this Attempt so far. Execution
                            	// IDs are contiguous from [1, CurExecution]. If the State is not currently
                            	// Executing, then CurExecution represents the execution that JUST finished
                            	// (or 0 if no Executions have been made yet).
                            	CurExecution uint32
                            
                            	// LastSuccessfulExecution is the execution ID of the last successful
                            	// execution, or 0 if no such execution occured yet.
                            	LastSuccessfulExecution uint32
                            
                            	// DepMap is valid only while Attempt is in a State of EXECUTING or WAITING.
                            	//
                            	// The size of this field is inspected to deteremine what the next state after
                            	// EXECUTING is. If the size == 0, it means the Attempt should move to the
                            	// FINISHED state. Otherwise it means that the Attempt should move to the
                            	// WAITING state.
                            	//
                            	// A bit field value of 0 means that the dep is currently waiting, and a bit
                            	// value of 1 means that the coresponding dep is satisfined. The Attempt can
                            	// be unblocked from WAITING back to SCHEDULING when all bits are set to 1.
                            	DepMap bf.BitField `gae:",noindex" json:"-"`
                            }

                              Attempt is the datastore model for a DM Attempt. It has no parent key, but it may have the following children entities:

                              * FwdDep
                              * AttemptResult
                              

                              Additionally, every Attempt has an associated BackDepGroup whose ID equals the ID of this Attempt.

                              func AttemptFromID

                              func AttemptFromID(aid *dm.Attempt_ID) *Attempt

                                AttemptFromID produces an empty Attempt model from the AttemptID.

                                func MakeAttempt

                                func MakeAttempt(c context.Context, aid *dm.Attempt_ID) *Attempt

                                  MakeAttempt is a convenience function to create a new Attempt model in the NeedsExecution state.

                                  func (*Attempt) DataProto

                                  func (a *Attempt) DataProto() (ret *dm.Attempt_Data)

                                    DataProto returns an Attempt.Data message for this Attempt.

                                    func (*Attempt) ModifyState

                                    func (a *Attempt) ModifyState(c context.Context, newState dm.Attempt_State) error

                                      ModifyState changes the current state of this Attempt and updates its Modified timestamp.

                                      func (*Attempt) ToProto

                                      func (a *Attempt) ToProto(withData bool) *dm.Attempt

                                        ToProto returns a dm proto version of this Attempt.

                                        type AttemptResult

                                        type AttemptResult struct {
                                        	Attempt *datastore.Key `gae:"$parent"`
                                        
                                        	// The sizes and expirations are denormalized across Attempt and
                                        	// AttemptResult.
                                        	Data dm.JsonResult `gae:",noindex"`
                                        	// contains filtered or unexported fields
                                        }

                                          AttemptResult holds the raw, compressed json blob returned from the execution.

                                          type AttemptRetryState

                                          type AttemptRetryState struct {
                                          	Failed   uint32
                                          	Expired  uint32
                                          	TimedOut uint32
                                          	Crashed  uint32
                                          }

                                            AttemptRetryState indicates the current state of the Attempt's retry counters.

                                            func (*AttemptRetryState) Reset

                                            func (a *AttemptRetryState) Reset()

                                              Reset resets all of the AttemptRetryState counters.

                                              type BackDep

                                              type BackDep struct {
                                              	// The attempt id of the attempt that's depending on this dependee.
                                              	Depender dm.Attempt_ID `gae:"$id"`
                                              
                                              	// The BackdepGroup for the attempt that is being depended on.
                                              	DependeeGroup *datastore.Key `gae:"$parent"`
                                              
                                              	// Propagated is true if the BackDepGroup has AttemptFinished, and this
                                              	// BackDep has been processed by the mutate.RecordCompletion tumble
                                              	// mutation. So if with two attempts A and B, A depends on B, the
                                              	// BackDep{DependeeGroup: B, Depender: A} has Propagated as true when B is
                                              	// finished, and a tumble Mutation has been launched to inform A of that fact.
                                              	Propagated bool
                                              }

                                                BackDep represents a single backwards dependency. Its ID is the same as the Attempt that's depending on this one. See BackDepGroup for more context.

                                                func (*BackDep) Edge

                                                func (b *BackDep) Edge() *FwdEdge

                                                  Edge produces a fwdedge object which points from the depending attempt to the depended-on attempt.

                                                  type BackDepGroup

                                                  type BackDepGroup struct {
                                                  	// Dependee is the "<AttemptID>" that the deps in this group point
                                                  	// back FROM.
                                                  	Dependee dm.Attempt_ID `gae:"$id"`
                                                  
                                                  	// This is a denormalized version of Attempt.State, used to allow
                                                  	// transactional additions to the BackDepGroup to stay within this Entity
                                                  	// Group when adding new back deps.
                                                  	AttemptFinished bool
                                                  }

                                                    BackDepGroup describes a group of reverse dependencies ('depended-by') between Attempts. Its ID is the same as the id of the Attempt that's being depended-on by other attempts, and it serves as the parent entity for the BackDep model. So:

                                                    Attempt(OTHER_QUEST|2)
                                                      FwdDep(QUEST|1)
                                                    
                                                    Attempt(QUEST|1)
                                                    
                                                    BackDepGroup(QUEST|1)
                                                      BackDep(OTHER_QUEST|2)
                                                    

                                                    Represents the OTHER_QUEST|2 depending on QUEST|1.

                                                    type Execution

                                                    type Execution struct {
                                                    	ID      invertedHexUint32 `gae:"$id"`
                                                    	Attempt *ds.Key           `gae:"$parent"`
                                                    
                                                    	Created  time.Time
                                                    	Modified time.Time
                                                    
                                                    	// DistributorConfigName is redundant with the Quest definition, but this
                                                    	// helps avoid extra unnecessary datastore round-trips to load the Quest.
                                                    	DistributorConfigName    string
                                                    	DistributorConfigVersion string
                                                    	DistributorToken         string
                                                    
                                                    	State dm.Execution_State
                                                    
                                                    	// IsAbnormal is true iff State==ABNORMAL_FINISHED. Used for walk_graph.
                                                    	IsAbnormal bool
                                                    
                                                    	// A lazily-updated boolean to reflect that this Execution is expired for
                                                    	// queries.
                                                    	IsExpired bool
                                                    
                                                    	// Contains either data (State==FINISHED) or abnormal_finish (State==ABNORMAL_FINISHED)
                                                    	Result dm.Result `gae:",noindex"`
                                                    
                                                    	// These are DM's internal mechanism for performing timeout actions on
                                                    	// Executions.
                                                    	//
                                                    	// The TimeTo* variables are copied from the quest description.
                                                    	//
                                                    	// The Timeout is only active when the Execution is in a non-terminal state.
                                                    	TimeToStart time.Duration `gae:",noindex"` // timeouts.start
                                                    	TimeToRun   time.Duration `gae:",noindex"` // timeouts.run
                                                    	TimeToStop  time.Duration `gae:",noindex"` // pollTimeout || timeouts.stop
                                                    
                                                    	// Token is a randomized nonce that's used to verify that RPCs verify from the
                                                    	// expected client (the client that's currently running the Execution). The
                                                    	// Token has 2 modes.
                                                    	//
                                                    	// When the Execution is handed to the distributor, the Token is randomly
                                                    	// generated by DM and passed to the distributor. The State of the Execution
                                                    	// starts as SCHEDULED. This token may be used by the client to "activate" the
                                                    	// Execution with the ActivateExecution rpc. At that point, the client
                                                    	// provides a new random token, the Execution State moves from SCHEDULED to
                                                    	// RUNNING, and Token assumes the new value. As long as the Execution State is
                                                    	// RUNNING, the client may continue to use that new Token value to
                                                    	// authenticate other rpc's like AddDeps and FinishAttempt.
                                                    	//
                                                    	// As soon as the Execution is in the STOPPING, ABNORMAL_FINISHED or FINISHED
                                                    	// state, this will be nil'd out.
                                                    	Token []byte `gae:",noindex"`
                                                    }

                                                      Execution represents either an ongoing execution on the Quest's specified distributor, or is a placeholder for an already-completed Execution.

                                                      func ExecutionFromID

                                                      func ExecutionFromID(c context.Context, eid *dm.Execution_ID) *Execution

                                                        ExecutionFromID produces an empty Execution model from the ExecutionID.

                                                        func MakeExecution

                                                        func MakeExecution(c context.Context, e *dm.Execution_ID, cfgName, cfgVers string) *Execution

                                                          MakeExecution makes a new Execution in the SCHEDULING state, with a new random Token.

                                                          func (*Execution) DataProto

                                                          func (e *Execution) DataProto() (ret *dm.Execution_Data)

                                                            DataProto returns an Execution.Data message for this Execution.

                                                            This omits the DistributorInfo.Url portion, which must be filled in elsewhere for package cyclical import reasons.

                                                            func (*Execution) GetEID

                                                            func (e *Execution) GetEID() *dm.Execution_ID

                                                              GetEID gets an Execution_ID for this Execution. It panics if the Execution is in an invalid state.

                                                              func (*Execution) ModifyState

                                                              func (e *Execution) ModifyState(c context.Context, newState dm.Execution_State) error

                                                                ModifyState changes the current state of this Execution and updates its Modified timestamp.

                                                                func (*Execution) Revoke

                                                                func (e *Execution) Revoke(c context.Context) error

                                                                  Revoke will clear the Token and Put this Execution to the datastore. This action requires the Execution to be in the RUNNING state, and causes it to enter the STOPPING state.

                                                                  func (*Execution) ToProto

                                                                  func (e *Execution) ToProto(includeID bool) *dm.Execution

                                                                    ToProto returns a dm proto version of this Execution.

                                                                    type FwdDep

                                                                    type FwdDep struct {
                                                                    	// Attempt that this points from.
                                                                    	Depender *ds.Key `gae:"$parent"`
                                                                    
                                                                    	// A FwdDep's ID is the Attempt ID that it points to.
                                                                    	Dependee dm.Attempt_ID `gae:"$id"`
                                                                    
                                                                    	// This will be used to set a bit in the Attempt (WaitingDepBitmap) when the
                                                                    	// Dep completes.
                                                                    	BitIndex uint32
                                                                    
                                                                    	// ForExecution indicates which Execution added this dependency. This is used
                                                                    	// for validation of AckFwdDep mutations to ensure that they're operating
                                                                    	// on an Attempt in the correct state, but can also be used for historical
                                                                    	// analysis/display.
                                                                    	ForExecution uint32
                                                                    }

                                                                      FwdDep describes a 'depends-on' relation between two Attempts. It has a reciprocal BackDep as well, which notes the depended-on-by relationship. So:

                                                                      Attempt(OTHER_QUEST|2)
                                                                        FwdDep(QUEST|1)
                                                                      
                                                                      Attempt(QUEST|1)
                                                                      
                                                                      BackDepGroup(QUEST|1)
                                                                        BackDep(OTHER_QUEST|2)
                                                                      

                                                                      Represents the OTHER_QUEST|2 depending on QUEST|1.

                                                                      func FwdDepsFromList

                                                                      func FwdDepsFromList(c context.Context, base *dm.Attempt_ID, list *dm.AttemptList) []*FwdDep

                                                                        FwdDepsFromList creates a slice of *FwdDep given an originating base Attempt_ID, and a list of dependency Attempts.

                                                                        func (*FwdDep) Edge

                                                                        func (f *FwdDep) Edge() *FwdEdge

                                                                          Edge produces a edge object which points 'forwards' from the depending attempt to the depended-on attempt.

                                                                          type FwdEdge

                                                                          type FwdEdge struct {
                                                                          	From *dm.Attempt_ID
                                                                          	To   *dm.Attempt_ID
                                                                          }

                                                                            FwdEdge represents a forward-dependency from one attempt to another. The From attempt will block until the To attempt completes.

                                                                            func (*FwdEdge) Back

                                                                            func (e *FwdEdge) Back(c context.Context) (*BackDepGroup, *BackDep)

                                                                              Back returns the BackDepGroup (To) and the BackDep (From) models that represent the reverse dependency for this FwdEdge.

                                                                              func (*FwdEdge) Fwd

                                                                              func (e *FwdEdge) Fwd(c context.Context) (*Attempt, *FwdDep)

                                                                                Fwd returns the Attempt (From) and FwdDep (To) models that this FwdEdge represents.

                                                                                type Quest

                                                                                type Quest struct {
                                                                                	// ID is the base64 sha256 of questPayload
                                                                                	ID string `gae:"$id"`
                                                                                
                                                                                	Desc    dm.Quest_Desc `gae:",noindex"`
                                                                                	BuiltBy TemplateInfo  `gae:",noindex"`
                                                                                
                                                                                	Created time.Time `gae:",noindex"`
                                                                                }

                                                                                  Quest is the model for a job-to-run. Its questPayload should fully describe the job to be done.

                                                                                  func NewQuest

                                                                                  func NewQuest(c context.Context, desc *dm.Quest_Desc) *Quest

                                                                                    NewQuest builds a new Quest object with a correct ID given the current contents of the Quest_Desc. It returns an error if the Desc is invalid.

                                                                                    Desc must already be Normalize()'d

                                                                                    func QuestFromID

                                                                                    func QuestFromID(qid string) *Quest

                                                                                      QuestFromID produces an empty Quest model from the QuestID.

                                                                                      func (*Quest) DataProto

                                                                                      func (q *Quest) DataProto() *dm.Quest_Data

                                                                                        DataProto gets the Quest.Data proto message for this Quest.

                                                                                        func (*Quest) Equals

                                                                                        func (q *Quest) Equals(a *Quest) bool

                                                                                          Equals is true if q and a and equal.

                                                                                          func (*Quest) ToProto

                                                                                          func (q *Quest) ToProto() *dm.Quest

                                                                                            ToProto converts this Quest into its display equivalent.

                                                                                            type TemplateInfo

                                                                                            type TemplateInfo []dm.Quest_TemplateSpec

                                                                                              TemplateInfo is an ordered list of dm.Quest_TemplateSpec's

                                                                                              func (*TemplateInfo) Add

                                                                                              func (ti *TemplateInfo) Add(ts ...dm.Quest_TemplateSpec)

                                                                                                Add adds ts to the TemplateInfo uniq'ly.

                                                                                                func (TemplateInfo) Equals

                                                                                                func (ti TemplateInfo) Equals(other TemplateInfo) bool

                                                                                                  Equals returns true iff this TemplateInfo is exactly same as `other`.

                                                                                                  func (TemplateInfo) EqualsData

                                                                                                  func (ti TemplateInfo) EqualsData(other []*dm.Quest_TemplateSpec) bool

                                                                                                    EqualsData returns true iff this TemplateInfo has the same content as the proto-style TemplateInfo. This assumes that `other` is sorted.

                                                                                                    func (TemplateInfo) Len

                                                                                                    func (ti TemplateInfo) Len() int

                                                                                                    func (TemplateInfo) Less

                                                                                                    func (ti TemplateInfo) Less(i, j int) bool

                                                                                                    func (TemplateInfo) Swap

                                                                                                    func (ti TemplateInfo) Swap(i, j int)