dispatch

package module
v0.0.0-...-017a74b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 16, 2026 License: MIT Imports: 14 Imported by: 0

README

dispatch

Submit and supervise multi-node training jobs on the scheduler you already have, behind one contract.

A job's outcome is a coarse state plus a typed cause (preempted, node lost, out of memory, timeout, no capacity...) that is never collapsed into "failed". Capability support is reported per cluster, not per adapter, and recovery policies are data that a separate supervisor applies. Scheduler adapters live in their own modules: go/dispatch-slurm and go/dispatch-kubeflow, and are documented on this module's docs site. Part of the phpboyscout Go toolkit.

Status

In development, nothing released. Decisions are written as specs in this project's wiki, and code is built against an APPROVED spec. The supervisor was built ahead of its spec's approval, alongside the deterministic simulator that tests it; the spec was approved once the simulator passed against the real code.

Documentation

dispatch.go.phpboyscout.uk has a tutorial, how-to guides, explanation and reference. The Go API is on pkg.go.dev.

Licence

MIT. See LICENSE.

Documentation

Overview

Package dispatch submits and supervises multi-node training jobs on the scheduler you already have, behind one contract.

A job's outcome is a coarse state plus a typed cause that is never collapsed into "failed". Capability support is reported per cluster, recovery policy is data the core validates and never executes, and the job level is computed by pure functions over the record and what the backend shows: Fold, Project and Next.

Open or Compose a Cluster over an adapter's Backend and a records.Records store, then submit through its Operator; MakePlan returns the plan Submit would record, without creating anything. Package supervise carries out what Next returns, and package conformance checks an adapter.

The tutorial, guides and reference are at https://dispatch.go.phpboyscout.uk. The design is in the project wiki's specs 0001 to 0005.

Example (SuperviseAJob)

This is the tutorial "Submit and supervise a job": a job is submitted with a rule that resubmits it after preemption, a supervisor runs it, the scheduler preempts it, the supervisor recovers it, and it succeeds. 

package main

import (
	"context"
	"fmt"
	"time"

	"gitlab.com/phpboyscout/go/dispatch"
	"gitlab.com/phpboyscout/go/dispatch/memory"
	"gitlab.com/phpboyscout/go/dispatch/records/recordstest"
	"gitlab.com/phpboyscout/go/dispatch/supervise"
)

// waitFor polls the job's status until ok holds. The tutorial's scheduler is the memory adapter,
// which moves in milliseconds; a real one would be watched with dispatch.Watch.
func waitFor(ctx context.Context, o dispatch.Observer, id dispatch.JobID, ok func(dispatch.Status) bool) (dispatch.Status, error) {
	ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
	defer cancel()

	for {
		st, err := o.Get(ctx, id)
		if err != nil {
			return dispatch.Status{}, err
		}

		if ok(st) {
			return st, nil
		}

		select {
		case <-ctx.Done():
			return st, fmt.Errorf("job %s stayed %s: %w", id, st.State, ctx.Err())
		case <-time.After(5 * time.Millisecond):
		}
	}
}

func main() {
	ctx := context.Background()

	backend, err := memory.New(memory.Config{Shape: memory.ShapeSlurm, AutoRun: true})
	if err != nil {
		fmt.Println(err)

		return
	}

	store := recordstest.New()
	cluster := dispatch.Compose(backend, dispatch.Config{Store: store})

	spec := dispatch.JobSpec{
		Name:    "finetune",
		Command: []string{"torchrun", "train.py"},
		Nodes:   2,
		Recovery: dispatch.RecoveryPolicy{Rules: []dispatch.Rule{
			{On: []dispatch.CauseKind{dispatch.CausePreempted}, Action: dispatch.ActionResubmit, Budget: 2},
		}},
	}
	me := dispatch.Actor{Kind: dispatch.ActorHuman, ID: "ada"}

	plan, err := cluster.Operator().Submit(ctx, spec, me, dispatch.WithSupervisor())
	if err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println("rule 0 is enforced by the", plan.Rules[0].Enforcer)
	fmt.Println("uncovered:", plan.Uncovered)

	sup, err := supervise.New(backend, store)
	if err != nil {
		fmt.Println(err)

		return
	}

	done := make(chan error, 1)

	go func() { done <- sup.Run(ctx, plan.Job) }()

	obs := cluster.Observer()

	st, err := waitFor(ctx, obs, plan.Job, func(s dispatch.Status) bool { return s.State == dispatch.StateRunning })
	if err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println(st.State, "as submission", st.Submissions[0].N)

	if err := backend.Env().Preempt(*st.Submissions[0].Canonical); err != nil {
		fmt.Println(err)

		return
	}

	st, err = waitFor(ctx, obs, plan.Job, func(s dispatch.Status) bool {
		return s.State == dispatch.StateRunning && len(s.Submissions) == 2
	})
	if err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println(st.State, "as submission", st.Submissions[1].N, "after", st.Plan.Rules[0].Firings, "firing of rule 0")

	if err := backend.Env().Complete(*st.Submissions[1].Canonical); err != nil {
		fmt.Println(err)

		return
	}

	if err := <-done; err != nil {
		fmt.Println(err)

		return
	}

	st, err = obs.Get(ctx, plan.Job)
	if err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println(st.State, "with cause", st.Cause.Kind)

	for _, e := range st.History {
		fmt.Printf("  %s -> %s (submission %d)\n", e.From, e.To, e.Submission)
	}

}

Output:
rule 0 is enforced by the supervisor
uncovered: [node-lost out-of-memory time-limit non-zero-exit launch-failed no-capacity requeued unknown]
running as submission 1
running as submission 2 after 1 firing of rule 0
succeeded with cause none
  pending -> recovering (submission 2)
  recovering -> succeeded (submission 2)

Index

Examples

Constants

View Source 
const (
	StatePending          = model.StatePending
	StateHeld             = model.StateHeld
	StateRunning          = model.StateRunning
	StateRecovering       = model.StateRecovering
	StateSucceeded        = model.StateSucceeded
	StateFailed           = model.StateFailed
	StateCancelled        = model.StateCancelled
	StateUnknown          = model.StateUnknown
	CauseNone             = model.CauseNone
	CausePreempted        = model.CausePreempted
	CauseNodeLost         = model.CauseNodeLost
	CauseOutOfMemory      = model.CauseOutOfMemory
	CauseTimeLimit        = model.CauseTimeLimit
	CauseNonZeroExit      = model.CauseNonZeroExit
	CauseLaunchFailed     = model.CauseLaunchFailed
	CauseNoCapacity       = model.CauseNoCapacity
	CauseHeld             = model.CauseHeld
	CauseReleased         = model.CauseReleased
	CauseRequeued         = model.CauseRequeued
	CauseCancelled        = model.CauseCancelled
	CauseDuplicate        = model.CauseDuplicate
	CauseUnknown          = model.CauseUnknown
	ProvenanceNone        = model.ProvenanceNone
	ProvenanceScheduler   = model.ProvenanceScheduler
	ProvenanceApplication = model.ProvenanceApplication
	ProvenanceActor       = model.ProvenanceActor
	ActorUnset            = model.ActorUnset
	ActorHuman            = model.ActorHuman
	ActorSupervisor       = model.ActorSupervisor
	ActorAdapter          = model.ActorAdapter
	HoldUnset             = model.HoldUnset
	HoldBackend           = model.HoldBackend
	HoldRecovery          = model.HoldRecovery
	ActionUnset           = model.ActionUnset
	ActionResubmit        = model.ActionResubmit
	ActionRestart         = model.ActionRestart
	ActionRequeue         = model.ActionRequeue
	ActionHold            = model.ActionHold
	ActionEscalate        = model.ActionEscalate
	EnforcerUnset         = model.EnforcerUnset
	EnforcerBackend       = model.EnforcerBackend
	EnforcerSupervisor    = model.EnforcerSupervisor
	EnforcerNobody        = model.EnforcerNobody
	CheckpointIsolated    = model.CheckpointIsolated
	CheckpointShared      = model.CheckpointShared
	UnmatchedResidual     = model.UnmatchedResidual
	UnmatchedManual       = model.UnmatchedManual
	Unknown               = model.Unknown
	Yes                   = model.Yes
	No                    = model.No
	SourceDeclared        = model.SourceDeclared
	SourceProbed          = model.SourceProbed
	SourceCatalogue       = model.SourceCatalogue
	LevelJob              = model.LevelJob
	LevelObjectOnly       = model.LevelObjectOnly
	CleanupClear          = model.CleanupClear
	CleanupPending        = model.CleanupPending
	CleanupBlocked        = model.CleanupBlocked
	ObjectGated           = model.ObjectGated
	ObjectPending         = model.ObjectPending
	ObjectHeld            = model.ObjectHeld
	ObjectRunning         = model.ObjectRunning
	ObjectEnded           = model.ObjectEnded
	Live                  = model.Live
	EvidenceState         = model.EvidenceState
	EvidenceCause         = model.EvidenceCause
	EvidenceExitCode      = model.EvidenceExitCode
	EvidenceExecution     = model.EvidenceExecution
	ContainerImage        = model.ContainerImage
	NativeRequeue         = model.NativeRequeue
	NativeRestart         = model.NativeRestart
	HoldCapability        = model.HoldCapability
	PreTerminationSignal  = model.PreTerminationSignal
	AccountingHistory     = model.AccountingHistory
	AttemptInjection      = model.AttemptInjection
	ReadmissionGate       = model.ReadmissionGate
	WorkingDirectory      = model.WorkingDirectory
	ProcsPerNode          = model.ProcsPerNode
	OutputPath            = model.OutputPath
	OutputRead            = model.OutputRead
	OutputFollow          = model.OutputFollow
	OutputRetained        = model.OutputRetained
)

The enum and capability values of model, re-exported so consumers import one package (spec 0001 D1).

View Source 
const (
	// UnresolvedMinAge is the default for how long a submission's attempts stay unknown before
	// they escalate.
	UnresolvedMinAge = 10 * time.Minute
	// CleanupDeadline is the default for how long clean-up may stay pending before it escalates.
	CleanupDeadline = 30 * time.Minute
	// CallDeadline is how far ahead a hold-family or requeue call's deadline is set.
	CallDeadline = 30 * time.Second
)

Timing the protocol uses (spec 0004 D2, D4).

View Source 
const CessationInterval = time.Minute

CessationInterval is how often a site's cessation signal is asked again while a shared- checkpoint hold stands.

View Source 
const HoldBase = model.HoldBase

HoldBase is the offset of a Slurm hold's far-future start time (spec 0003 D9).

View Source 
const SharedHoldFor = 24 * time.Hour

SharedHoldFor is the deadline of the hold a recovery rule becomes under a Shared checkpoint (spec 0001 D16). The hold is for a person; past it, it escalates and stays held.

View Source 
const Unbounded = model.Unbounded

Unbounded is the PolledLoss window of evidence that polling never loses (spec 0003 D7).

Variables

View Source 
var (
	ErrNotFound          = model.ErrNotFound
	ErrUnsupported       = model.ErrUnsupported
	ErrInvalid           = model.ErrInvalid
	ErrRejected          = model.ErrRejected
	ErrUnauthorized      = model.ErrUnauthorized
	ErrUnreachable       = model.ErrUnreachable
	ErrReadOnly          = model.ErrReadOnly
	ErrUnavailable       = model.ErrUnavailable
	ErrOutcomeUnknown    = model.ErrOutcomeUnknown
	ErrAlreadyRegistered = model.ErrAlreadyRegistered
)

The core's sentinels (spec 0001 D12).

Functions

func Adapters 

func Adapters() []string

Adapters lists the registered adapter names, sorted.

func FindAll 

func FindAll(ctx context.Context, b Backend, jobs []JobID, states ...ObjectState) (map[JobID]Discovery, error)

FindAll returns a Discovery for every job, keeping only objects in states when any are named: in one query when b is a MultiFinder, and otherwise by one Find per job.

func Register 

func Register(name string, f Factory) error

Register makes an adapter available to Open under name. A duplicate is ErrAlreadyRegistered, never an overwrite (spec 0001 D11).

func Watch 

func Watch(ctx context.Context, o Observer, id JobID, every time.Duration) <-chan Event

Watch reports each state transition of a job, polling o every interval, until ctx ends; the channel is closed then. Transitions are the core's, from the record (spec 0001 D13). Without a store a job is never reported terminal, because an object's end is not the job's.

Watch is an observer API off the simulated path, and the one function there that starts a goroutine (spec 0005 D2 rule 1).

Types

type AbandonTarget 

type AbandonTarget = model.AbandonTarget

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Action 

type Action = model.Action

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Actor 

type Actor = model.Actor

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type ActorKind 

type ActorKind = model.ActorKind

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type AdapterConfig 

type AdapterConfig struct {
	Backend  string
	ReadOnly bool
	// Settings is the adapter's own section, validated against its schema with
	// additionalProperties false at every level.
	Settings json.RawMessage
	// Token is Config.Token: called on every use, never cached (spec 0001 D11).
	Token func(ctx context.Context) (string, error)
}

AdapterConfig is what a Factory receives: no store and no job level (spec 0003 D13).

type AttemptRec 

type AttemptRec struct {
	Number    int
	Reply     records.Reply
	Ref       *ObjectRef
	Matched   bool
	Abandoned bool
	At        time.Time
	Detail    string
}

AttemptRec is one Start call for a submission, numbered before the call (spec 0003 Notes).

func (*AttemptRec) Unresolved 

func (a *AttemptRec) Unresolved() bool

Unresolved reports whether the attempt's outcome is still unknown: no reply or an unknown one, no object matched to it, and not abandoned.

type Backend 

type Backend interface {
	Capabilities(ctx context.Context, spec *JobSpec) (map[Capability]Support, error)
	// Rules answers one RuleSupport per rule, in the policy's order.
	Rules(ctx context.Context, spec JobSpec, policy RecoveryPolicy) ([]RuleSupport, error)
	// Start creates the object with its gate closed; ErrOutcomeUnknown when it cannot tell.
	Start(ctx context.Context, req StartRequest) (ObjectRef, error)
	// Open lifts the gate; an object already past its gate is success.
	Open(ctx context.Context, ref ObjectRef) error
	// Find returns every object of the job, by its discovery name.
	Find(ctx context.Context, job JobID) (Discovery, error)
	Inspect(ctx context.Context, ref ObjectRef) (ObjectObs, error)
	// Stop stops this one object only; an object already gone is success.
	Stop(ctx context.Context, ref ObjectRef, reason string) error
	// StopJob stops every object of the job by name, and verifies the result.
	StopJob(ctx context.Context, job JobID, reason string) (StopReport, error)
	Declarations() Declarations
	Close() error
}

Backend is what an adapter implements: backend objects and their runs, never jobs or the record (spec 0003 D1). Optional capabilities are found by type assertion.

func NewBackend 

func NewBackend(ctx context.Context, name string, cfg Config) (Backend, error)

NewBackend builds the named adapter's Backend from cfg, for a host that also runs a supervisor over it and composes the Cluster itself.

type Budgets 

type Budgets = model.Budgets

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Call 

type Call struct {
	Ref      ObjectRef
	Instance uint32
	Op       records.CallOp
	Deadline time.Time
	Expires  time.Time
}

Call is a recorded hold-family, requeue or restore call: its deadline and when it counts as expired.

type CallMeta 

type CallMeta = model.CallMeta

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type CancelReceipt 

type CancelReceipt struct {
	Committed bool
	// Already is the terminal state an earlier outcome left, when the cancel could not change it.
	Already State
	// CleanupComplete is set once Find shows nothing of the job's name running, pending or
	// completing.
	CleanupComplete bool
	// Live names objects still stopping.
	Live []ObjectRef
	// CessationUnproven names nodes that were down or not responding, so the processes may live.
	CessationUnproven []string
}

CancelReceipt reports a cancel's two halves separately: the commit, which is final, and backend clean-up, which may lag (spec 0004 D4).

type Capability 

type Capability = model.Capability

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Cause 

type Cause = model.Cause

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type CauseKind 

type CauseKind = model.CauseKind

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type CessationEvidence 

type CessationEvidence = model.CessationEvidence

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type CessationSignal 

type CessationSignal interface {
	Ceased(ctx context.Context, ref ObjectRef) (CessationEvidence, error)
}

CessationSignal asks a site's own signal whether an object's processes are gone (spec 0003 D12).

type CheckpointMode 

type CheckpointMode = model.CheckpointMode

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type CleanupState 

type CleanupState = model.CleanupState

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Cluster 

type Cluster interface {
	Observer() Observer
	// Operator is nil when read-only or without a store.
	Operator() Operator
	// Stopper is nil only when read-only.
	Stopper() Stopper
	Close() error
}

Cluster is an opened adapter composed with the core (spec 0001 D6, D11).

func Compose 

func Compose(b Backend, cfg Config) Cluster

Compose builds a Cluster over an already-built Backend, for hosts and tests that construct their adapter directly.

func Open 

func Open(ctx context.Context, name string, cfg Config) (Cluster, error)

Open composes the core over the named adapter's Backend and the record (spec 0001 D11). Without a store and with ReadOnly false, the cluster has an Observer and a Stopper but no Operator: the emergency stop is the only write that needs no record.

type Completeness 

type Completeness = model.Completeness

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Config 

type Config struct {
	// Adapter is the adapter's own configuration, passed to its Factory as JSON.
	Adapter any
	// Token is called on every use; optional.
	Token func(ctx context.Context) (string, error)
	// Store nil allows read-only access only.
	Store    records.Records
	ReadOnly bool
	// Rand is the source of job identities; nil means crypto/rand (spec 0005 D2).
	Rand io.Reader
}

Config is what Open needs (spec 0001 D11).

type Declarations 

type Declarations = model.Declarations

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Desired 

type Desired int

Desired is what the user wants: run, hold or cancel (spec 0002 appendix, control). 

const (
	DesiredRun Desired = iota
	DesiredHold
	DesiredCancel
)

The desired states.

type Discovery 

type Discovery = model.Discovery

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Disposition 

type Disposition int

Disposition is a submission's standing: Open while it launches, Canonical once its handle is committed, Abandoned once a person closed it (spec 0002 appendix). 

const (
	DispositionOpen Disposition = iota
	DispositionCanonical
	DispositionAbandoned
)

The dispositions.

type Enforcer 

type Enforcer = model.Enforcer

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Escalation 

type Escalation struct {
	ID        string
	Detail    string
	At        time.Time
	Delivered time.Time
}

Escalation is one escalation, identified by its cause so it is committed once (spec 0004 D5).

type Event 

type Event struct {
	From, To   State
	Cause      Cause
	Submission int
	Level      Level
	At         time.Time
}

Event is one state transition, computed once in the core (spec 0001 D13).

type Evidence 

type Evidence = model.Evidence

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type ExecutionObs 

type ExecutionObs = model.ExecutionObs

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Extension 

type Extension = model.Extension

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Extensions 

type Extensions = model.Extensions

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Factory 

type Factory func(ctx context.Context, cfg AdapterConfig) (Backend, error)

Factory builds an adapter's Backend.

type Filter 

type Filter struct {
	// States keeps jobs in any of these states; empty keeps all.
	States []State
	// Settled includes records that have settled.
	Settled bool
}

Filter selects jobs for List.

type Hold 

type Hold = model.Hold

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type HoldKind 

type HoldKind = model.HoldKind

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type HoldMeta 

type HoldMeta = model.HoldMeta

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type HoldRec 

type HoldRec struct {
	Instance uint32
	Hold     Hold
	// Cause is what ended the execution a recovery hold suspended, in Submission.
	Cause      Cause
	Submission int
	Active     bool
	Refused    bool
	Released   time.Time
}

HoldRec is one hold instance (spec 0002 appendix, holds).

type Holder 

type Holder interface {
	Hold(ctx context.Context, ref ObjectRef, m HoldMeta) error
	// Release releases only the instance m names.
	Release(ctx context.Context, ref ObjectRef, m HoldMeta) error
}

Holder places and releases backend holds (spec 0003 D9).

type IdentityWeakener 

type IdentityWeakener interface {
	// IdentityWeakened says why spec's execution identity would be weakened, or "" when it would
	// not be. MakePlan records it in Plan.IdentityWeakened and refuses nothing on it: the host shows it.
	IdentityWeakened(ctx context.Context, spec JobSpec) (string, error)
}

IdentityWeakener is implemented by an adapter that, for some specs, lets the backend rerun a rank under the same execution identity, such as Kubeflow's site opt-in allowRankRestart. An adapter that does not implement it keeps every execution's identity its own.

type Image 

type Image = model.Image

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Job 

type Job struct {
	Header   records.Header
	Revision uint64

	Desired Desired
	// Outcome is the terminal state committed, if any; terminal states never change.
	Outcome      *Outcome
	Cancel       *records.Control
	Cleanup      CleanupState
	Settled      bool
	SettledAt    time.Time
	Submissions  []*SubmissionRec
	Objects      []*ObjectRec
	Holds        []*HoldRec
	Requeue      *records.Control
	RequeueCalls []Call
	// RestoreCalls are restores of a requeue bit left set (spec 0003 D10).
	RestoreCalls []Call
	HoldCalls    []Call
	StopJobs     []time.Time
	Escalations  []Escalation
	// RuleUse counts the submissions authorized per rule, so budgets survive any actor.
	RuleUse map[int]int
	// Resubmits counts human Resubmit intents not yet authorized as a submission.
	Resubmits int
	// Resumed is the submission whose shared-checkpoint hold a cessation answer lifted, so its
	// original rule now applies (spec 0004 D9); 0 when none.
	Resumed int
	// History is every state transition, in record order.
	History []Event
}

Job is the record folded into the job level: everything Next and Project need, computed by Fold alone so every actor sees the same job from the same revision (spec 0001 D14).

func Fold 

func Fold(rec records.View, now time.Time) (Job, error)

Fold replays a record into the job level. It performs no I/O, and now is an argument, so deadlines such as Overdue are computed without a clock inside (spec 0001 D14). A history that cannot have been written by the protocol is refused with ErrInvalid, never folded silently.

func (*Job) ActiveHold 

func (j *Job) ActiveHold() *HoldRec

ActiveHold returns the hold in force, or nil.

func (*Job) AllRequeueClear 

func (j *Job) AllRequeueClear() time.Time

AllRequeueClear is RequeueClear over every object of the job.

func (*Job) Escalated 

func (j *Job) Escalated(id string) bool

Escalated reports whether an escalation with id was committed.

func (*Job) HoldCallsClear 

func (j *Job) HoldCallsClear(ref ObjectRef) time.Time

HoldCallsClear is when every recorded hold-family call on ref has landed or expired.

func (*Job) HoldMeta 

func (j *Job) HoldMeta(instance uint32, m CallMeta) HoldMeta

HoldMeta returns the meta of a call on hold instance, carrying who placed it, until when and why from the record's hold control (spec 0003 D9). An instance the record does not hold carries only the instance and m.

func (*Job) Latest 

func (j *Job) Latest() *SubmissionRec

Latest returns the latest submission, or nil before the first.

func (*Job) Object 

func (j *Job) Object(ref ObjectRef) *ObjectRec

Object returns the recorded object with ref, or nil.

func (*Job) RequeueClear 

func (j *Job) RequeueClear(ref ObjectRef) time.Time

RequeueClear is when every recorded requeue call on ref has landed or expired: the greatest expiry recorded for it (spec 0004 D6). The zero time means none was recorded.

func (*Job) Residuals 

func (j *Job) Residuals() []model.AbandonTarget

Residuals are the unmatched attempts the host keeps sweeping after the record settles, under the Residual policy (spec 0002 appendix).

func (*Job) RestoreClear 

func (j *Job) RestoreClear(ref ObjectRef) time.Time

RestoreClear is when every recorded restore call on ref has landed or expired.

func (*Job) Submission 

func (j *Job) Submission(n int) *SubmissionRec

Submission returns submission n, or nil.

func (*Job) Terminal 

func (j *Job) Terminal() bool

Terminal reports whether an outcome was committed.

type JobID 

type JobID = model.JobID

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type JobSpec 

type JobSpec = model.JobSpec

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Level 

type Level = model.Level

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type MultiFinder 

type MultiFinder interface {
	// FindMany answers a Discovery for every job asked, keeping only objects in states when any
	// are named.
	FindMany(ctx context.Context, jobs []JobID, states ...ObjectState) (map[JobID]Discovery, error)
}

MultiFinder finds the objects of many jobs in one scheduler query, as Find does for one; FindAll uses it when the adapter has it. dunscaith's residual sweep is its consumer (dunscaith 0001 D8).

type NativeOffer 

type NativeOffer = model.NativeOffer

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type NativePolicyCompiler 

type NativePolicyCompiler interface {
	Native(ctx context.Context, spec JobSpec, policy RecoveryPolicy) ([]Support, error)
	Compile(ctx context.Context, spec JobSpec, policy RecoveryPolicy, remaining Budgets) (NativeOffer, error)
}

NativePolicyCompiler offers rules the backend would enforce itself (spec 0003 D3).

type ObjectObs 

type ObjectObs = model.ObjectObs

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type ObjectRec 

type ObjectRec struct {
	Ref        ObjectRef
	Submission int
	Attempt    int
	Foreign    bool
	Disposed   bool
	Retired    bool
	// Opening is set once an Open was decided for it; EverOpened once it was seen past its gate.
	Opening    bool
	EverOpened bool
	Ended      bool
	EndedAt    time.Time
	// VerifiedStopped is the reconciler's judgement from absence (spec 0002 appendix).
	VerifiedStopped bool
	StoppedAt       time.Time
	SeenInFind      bool
	Last            *ObjectObs
	// NodeLostSince is when the object was first seen running with a lost node.
	NodeLostSince time.Time
	// Ceased is the site cessation signal's latest answer, and CessationAt when it was recorded.
	Ceased      bool
	CessationAt time.Time
}

ObjectRec is one scheduler object ever recorded for the job (spec 0002 appendix, objects).

func (*ObjectRec) Done 

func (o *ObjectRec) Done() bool

Done reports whether the object is ended or verified stopped.

type ObjectRef 

type ObjectRef = model.ObjectRef

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type ObjectState 

type ObjectState = model.ObjectState

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Observer 

type Observer interface {
	Capabilities(ctx context.Context, spec *JobSpec) (map[Capability]Support, error)
	Get(ctx context.Context, id JobID) (Status, error)
	List(ctx context.Context, f Filter) ([]Status, error)
}

Observer reads jobs (spec 0001 D6).

type Operator 

type Operator interface {
	Submit(ctx context.Context, spec JobSpec, by Actor, opts ...SubmitOption) (Plan, error)
	// Cancel commits first, then stops; its receipt separates the two.
	Cancel(ctx context.Context, id JobID, by Actor, reason string) (CancelReceipt, error)
	Hold(ctx context.Context, id JobID, by Actor, reason string, until time.Time) error
	Release(ctx context.Context, id JobID, by Actor) error
	// Resubmit creates a new submission of the same job.
	Resubmit(ctx context.Context, id JobID, by Actor, reason string) error
	// Supersede creates a new job whose Supersedes names id.
	Supersede(ctx context.Context, id JobID, by Actor, reason string, opts ...SubmitOption) (Plan, error)
	GiveUp(ctx context.Context, id JobID, by Actor, reason string) error
	Abandon(ctx context.Context, id JobID, t AbandonTarget, by Actor, reason string) error
	// Requeue commits a revocable intent; ErrUnsupported without a Requeuer.
	Requeue(ctx context.Context, id JobID, by Actor, reason string) error
}

Operator acts on jobs; every action names its actor and is recorded (spec 0001 D6). There is no Operator without a store.

type Outcome 

type Outcome struct {
	State State
	Cause Cause
	At    time.Time
}

Outcome is a committed terminal state and its cause.

type Output 

type Output = model.Output

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type OutputReader 

type OutputReader interface {
	Output(ctx context.Context, id JobID, submission int, s Stream, follow bool) (io.ReadCloser, error)
}

OutputReader reads a submission's output where the cluster keeps it: a capability (OutputRead, OutputFollow, OutputRetained), never a Status field, found on a Cluster's Observer by type assertion. A cluster that cannot read output returns an error, which is what makes those capabilities visible on failure (spec 0001 D7, D15).

type Plan 

type Plan = model.Plan

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

func MakePlan 

func MakePlan(ctx context.Context, b Backend, job JobID, spec JobSpec, opts ...SubmitOption) (Plan, error)

MakePlan checks spec against what b supports and assigns every rule an enforcer, in spec 0001 D9's order, before anything is created. It is Submit's decision, separate so hosts can show a plan without submitting.

Example

MakePlan shows the plan Submit would record, without creating anything. Here nobody would enforce the rule, so the plan is refused until the host says it will supervise the job. 

package main

import (
	"context"
	"errors"
	"fmt"

	"gitlab.com/phpboyscout/go/dispatch"
	"gitlab.com/phpboyscout/go/dispatch/memory"
)

func main() {
	ctx := context.Background()

	backend, err := memory.New(memory.Config{Shape: memory.ShapeSlurm})
	if err != nil {
		fmt.Println(err)

		return
	}

	spec := dispatch.JobSpec{
		Command: []string{"torchrun", "train.py"},
		Nodes:   4,
		Recovery: dispatch.RecoveryPolicy{Rules: []dispatch.Rule{
			{On: []dispatch.CauseKind{dispatch.CausePreempted, dispatch.CauseNodeLost}, Action: dispatch.ActionResubmit, Budget: 3},
		}},
	}

	_, err = dispatch.MakePlan(ctx, backend, "", spec)
	fmt.Println(errors.Is(err, dispatch.ErrRejected), err)

	plan, err := dispatch.MakePlan(ctx, backend, "", spec, dispatch.WithSupervisor())
	if err != nil {
		fmt.Println(err)

		return
	}

	for _, a := range plan.Rules {
		fmt.Println(a.Index, a.Effective.Action, a.Effective.On, "enforced by the", a.Enforcer)
	}

	fmt.Println("checkpoint:", plan.Checkpoint, "unmatched:", plan.Unmatched)

}

Output:
true dispatch: rejected: nobody would enforce rule 0; supervise the job or AcceptUnenforced it
0 resubmit [preempted node-lost] enforced by the supervisor
checkpoint: isolated unmatched: residual
Example (SharedCheckpoint)

A job that declares a shared checkpoint cannot isolate its executions, so every recovery rule becomes a hold for a person, and the plan shows the substitution before anything starts. 

package main

import (
	"context"
	"fmt"

	"gitlab.com/phpboyscout/go/dispatch"
	"gitlab.com/phpboyscout/go/dispatch/memory"
)

func main() {
	ctx := context.Background()

	backend, err := memory.New(memory.Config{Shape: memory.ShapeSlurm})
	if err != nil {
		fmt.Println(err)

		return
	}

	spec := dispatch.JobSpec{
		Command:    []string{"torchrun", "train.py"},
		Nodes:      4,
		Extensions: dispatch.Extensions{dispatch.SharedCheckpoint{}},
		Recovery: dispatch.RecoveryPolicy{Rules: []dispatch.Rule{
			{On: []dispatch.CauseKind{dispatch.CausePreempted}, Action: dispatch.ActionResubmit, Budget: 3},
		}},
	}

	plan, err := dispatch.MakePlan(ctx, backend, "", spec, dispatch.WithSupervisor())
	if err != nil {
		fmt.Println(err)

		return
	}

	a := plan.Rules[0]
	fmt.Println("checkpoint:", plan.Checkpoint)
	fmt.Println("asked for:", a.Rule.Action, "budget", a.Rule.Budget)
	fmt.Println("will do:", a.Effective.Action, "for", a.Effective.For)

}

Output:
checkpoint: shared
asked for: resubmit budget 3
will do: hold for 24h0m0s

type Provenance 

type Provenance = model.Provenance

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type RecoveryPolicy 

type RecoveryPolicy = model.RecoveryPolicy

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Requeuer 

type Requeuer interface {
	Requeue(ctx context.Context, ref ObjectRef, m CallMeta) error
	// RestoreRequeueBit clears a requeue bit left set outside Requeue, by a crash between its
	// steps or by an operator; on Slurm it is `scontrol update Requeue=0`.
	RestoreRequeueBit(ctx context.Context, ref ObjectRef, m CallMeta) error
}

Requeuer re-runs the same object, under a host-checked deadline (spec 0003 D10).

type Rule 

type Rule = model.Rule

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type RuleAssignment 

type RuleAssignment = model.RuleAssignment

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type RuleSupport 

type RuleSupport = model.RuleSupport

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type SharedCheckpoint 

type SharedCheckpoint = model.SharedCheckpoint

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Source 

type Source = model.Source

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type StartRequest 

type StartRequest = model.StartRequest

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type State 

type State = model.State

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Status 

type Status struct {
	Job         JobID
	Name        string
	Level       Level
	State       State
	Cause       Cause
	Unattended  bool
	Cleanup     CleanupState
	Hold        *Hold
	Submissions []Submission
	History     []Event
	Plan        *Plan
	Supersedes  JobID
}

Status is a job as the core computes it from the record and what the backend shows (spec 0001 D4).

func Project 

func Project(j Job, d Discovery, now time.Time) Status

Project is the job as status reports it: the folded record plus what the backend shows now (spec 0001 D4, D14). It performs no I/O.

func (Status) IdentityWeakened 

func (s Status) IdentityWeakened() string

IdentityWeakened is the plan's reason the job's execution identity is weakened, or "" when it is not or there is no plan. A host shows it; the core does not act on it.

type Step 

type Step struct {
	Kind       StepKind
	Submission int
	Attempt    int
	TriggerKey string
	Rule       int
	Ref        ObjectRef
	Obs        *ObjectObs
	Outcome    State
	Cause      Cause
	Escalation string
	Detail     string
	// Hold is set for a recovery hold placed by StepPlaceHold; a backend hold names Instance.
	Hold     *Hold
	Instance uint32
	Op       records.CallOp
	// Deadline and Expires are a hold-family, requeue or restore call's, committed before it is issued.
	Deadline time.Time
	Expires  time.Time
	// FindAt is the Find a verified-stopped judgement rests on.
	FindAt time.Time
	// Wake is when a StepWait should next look.
	Wake time.Time
}

Step is one thing to do. Only the fields its kind uses are set.

func Next 

func Next(j Job, d Discovery, now time.Time) []Step

Next returns what should happen next for the job, given what the backend shows now. It is pure, so every actor computes the same steps from the same inputs (spec 0001 D14, 0004 D1). Decision steps are committed against the revision j was folded from, which was read before d's Find.

func (Step) Decision 

func (s Step) Decision() bool

Decision reports whether the step commits a reconciler decision, which carries the owner token; evidence and backend calls do not (spec 0004 D2).

type StepKind 

type StepKind int

StepKind is one of the closed set of things Next can ask for (spec 0001 D14). The supervisor executes each through spec 0004 D2 to D6. 

const (
	StepWait StepKind = iota
	// StepObserve commits what the backend showed, as evidence.
	StepObserve
	// StepAuthorize commits submission N and its budget reservation.
	StepAuthorize
	// StepStart commits the attempt, calls Start with the gate closed, and records the reply.
	StepStart
	StepCanonical
	// StepOpen commits the decision to open, then lifts the gate.
	StepOpen
	StepRetire
	StepPlaceHold
	StepReleaseHold
	StepRequeue
	StepStop
	StepStopJob
	StepVerifiedStopped
	StepOutcome
	StepEscalate
	// StepSettle commits that the settled predicate holds.
	StepSettle
	// StepCease asks the adapter's cessation signal about an object and records the answer.
	StepCease
	// StepRestoreRequeue commits a restore call's deadline, then clears a requeue bit left set
	// outside Requeue (spec 0003 D10).
	StepRestoreRequeue
)

The steps.

type StopReceipt 

type StopReceipt struct {
	Report StopReport
}

StopReceipt is the emergency stop's result. The cancellation is not recorded, and recovery may still occur (spec 0001 D6).

type StopReport 

type StopReport = model.StopReport

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Stopper 

type Stopper interface {
	EmergencyStop(ctx context.Context, id JobID, by Actor, reason string) (StopReceipt, error)
}

Stopper is the emergency stop, which needs no record (spec 0001 D6).

type Stream 

type Stream int

Stream names one of an execution's output streams (spec 0001 D15). 

const (
	Stdout Stream = iota
	Stderr
)

The streams.

type Submission 

type Submission struct {
	N          int
	Canonical  *ObjectRef
	Executions []ExecutionObs
	Abandoned  bool
}

Submission is submission N as status reports it: its canonical object, if committed, and the executions seen.

type SubmissionRec 

type SubmissionRec struct {
	N            int
	TriggerKey   string
	Rule         int
	Disposition  Disposition
	Attempts     []*AttemptRec
	Canonical    *ObjectRef
	AuthorizedAt time.Time
}

SubmissionRec is submission N: its budget reservation, its attempts and its canonical handle.

type SubmitOption 

type SubmitOption func(*SubmitOptions)

SubmitOption configures Submit and Supersede.

func AcceptUnenforced 

func AcceptUnenforced(rule int) SubmitOption

AcceptUnenforced accepts that nobody enforces rule (by its index), so a matching cause gives Failed with an escalation (spec 0001 D9 step 5).

func AcceptUnverified 

func AcceptUnverified(rule int, enforcer Enforcer) SubmitOption

AcceptUnverified accepts rule (by its index in the policy) whose Perform support is Unknown, naming who will enforce it (spec 0001 D9 step 2).

func WithCleanupDeadline 

func WithCleanupDeadline(d time.Duration) SubmitOption

WithCleanupDeadline sets how long clean-up may stay pending before it escalates (spec 0004 D4).

func WithIdempotencyKey 

func WithIdempotencyKey(k string) SubmitOption

WithIdempotencyKey makes a repeated Submit return the existing job's plan (spec 0001 D10).

func WithSubmitRetries 

func WithSubmitRetries(n int) SubmitOption

WithSubmitRetries sets how often an ambiguous submit is retried; 0 disables retries (spec 0004 D3). It is recorded with the job, so every actor uses it.

func WithSupervisor 

func WithSupervisor() SubmitOption

WithSupervisor says the host will supervise this job, so Supervisor may enforce its rules (spec 0001 D9).

func WithUnmatched 

func WithUnmatched(p UnmatchedPolicy) SubmitOption

WithUnmatched chooses how an attempt whose outcome stayed unknown affects settlement; the default is Residual (spec 0004 D2).

func WithUnresolvedAge 

func WithUnresolvedAge(d time.Duration) SubmitOption

WithUnresolvedAge sets how long a submission's attempts may stay unknown before they escalate (spec 0004 D2).

type SubmitOptions 

type SubmitOptions struct {
	Supervised       bool
	IdempotencyKey   string
	Unmatched        UnmatchedPolicy
	AcceptUnverified map[int]Enforcer
	AcceptUnenforced map[int]bool
	// SubmitRetries is how often an ambiguous submit is retried; nil means the default (2), and 0
	// disables retries (spec 0004 D3).
	SubmitRetries *int
	// CleanupDeadline and UnresolvedAge override the defaults of 30 and 10 minutes (spec 0004 D2,
	// D4).
	CleanupDeadline time.Duration
	UnresolvedAge   time.Duration
}

SubmitOptions is what the options set; adapters and hosts read it through ApplySubmitOptions.

func ApplySubmitOptions 

func ApplySubmitOptions(opts ...SubmitOption) SubmitOptions

ApplySubmitOptions folds opts into their settings.

type Support 

type Support = model.Support

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Tristate 

type Tristate = model.Tristate

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type UnmatchedPolicy 

type UnmatchedPolicy = model.UnmatchedPolicy

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type ValidationError 

type ValidationError = model.ValidationError

The value types live in model so dispatch/records can store them without a cycle (spec 0001 D1); they are re-exported here so consumers import one package.

type Watcher 

type Watcher interface {
	Watch(ctx context.Context, job JobID) (<-chan ObjectObs, error)
}

Watcher streams observations as they happen, for evidence a poll would lose (spec 0003 D7).

Source Files

View all Source files

Directories

Path Synopsis
Package checkpoint publishes and resolves training checkpoints under one checkpoint root, on a POSIX filesystem reached through an afero.Fs.
 Package checkpoint publishes and resolves training checkpoints under one checkpoint root, on a POSIX filesystem reached through an afero.Fs.
creds
Package creds resolves the S3 credentials the checkpoint-publication helper signs with, from an ordered list of credential sources (spec 0001 D16).
 Package creds resolves the S3 credentials the checkpoint-publication helper signs with, from an ordered list of credential sources (spec 0001 D16).
creds/vault
Package vault is the vault credential-source kind (spec 0001 D16): S3 credentials read from a Vault KV v2 secret with go/config-vault, through a client from go/vaultclient, after logging in with Kubernetes auth or AppRole.
 Package vault is the vault credential-source kind (spec 0001 D16): S3 credentials read from a Vault KV v2 secret with go/config-vault, through a client from go/vaultclient, after logging in with Kubernetes auth or AppRole.
proxy
Package proxy is the one rule for the HTTP proxy a checkpoint-publication client may be told to use (spec 0001 D16).
 Package proxy is the one rule for the HTTP proxy a checkpoint-publication client may be told to use (spec 0001 D16).
s3
Package s3 is the checkpoint publication log on S3-compatible object storage (spec 0001 D16).
 Package s3 is the checkpoint publication log on S3-compatible object storage (spec 0001 D16).
s3/s3fake
Package s3fake is an in-process S3 server for tests: just the calls checkpoint publication makes (a put that honours If-None-Match: *, GET, ListObjectsV2 with prefix, delimiter and continuation, HEAD, DELETE), each refused unless its SigV4 signature verifies.
 Package s3fake is an in-process S3 server for tests: just the calls checkpoint publication makes (a put that honours If-None-Match: *, GET, ListObjectsV2 with prefix, delimiter and continuation, HEAD, DELETE), each refused unless its SigV4 signature verifies.
cmd
publish command
Command publish is the checkpoint-publication helper a job script runs on the compute node (spec 0001 D16): resolve before training, watch beside it, publish a save explicitly, prune, and probe an S3 store before a job is accepted.
 Command publish is the checkpoint-publication helper a job script runs on the compute node (spec 0001 D16): resolve before training, watch beside it, publish a save explicitly, prune, and probe an S3 store before a job is accepted.
Package conformance is the backend conformance suite: what every dispatch adapter must show at object level, checked through the SPI alone (spec 0003 D11).
 Package conformance is the backend conformance suite: what every dispatch adapter must show at object level, checked through the SPI alone (spec 0003 D11).
internal
mutants
Package mutants holds the knobs that each revert one fix the model checks found, so the simulator can show it finds that bug again (spec 0005 D9).
 Package mutants holds the knobs that each revert one fix the model checks found, so the simulator can show it finds that bug again (spec 0005 D9).
proxytest
Package proxytest is a forwarding HTTP proxy for tests, which records every request it carries.
 Package proxytest is a forwarding HTTP proxy for tests, which records every request it carries.
sim
Package sim is the deterministic simulator that gates spec 0004 (spec 0005).
 Package sim is the deterministic simulator that gates spec 0004 (spec 0005).
sim/crashfs
Package crashfs is the simulator's crash-model filesystem (spec 0005 D3): an afero.Fs over afero's in-memory filesystem that refuses what POSIX refuses, makes file data and directory entries durable only when they are synced, and models a process crash and a power cut.
 Package crashfs is the simulator's crash-model filesystem (spec 0005 D3): an afero.Fs over afero's in-memory filesystem that refuses what POSIX refuses, makes file data and directory entries durable only when they are synced, and models a process crash and a power cut.
sim/lint
Package lint checks the mechanical rules of spec 0005 D2 over the code on the simulated path: rule 1 (no goroutines started), rule 3 (no global random source, no package-level channel or WaitGroup), rule 4 (map ranges sorted or annotated) and rule 6's clauses on context.Background and context.WithoutCancel.
 Package lint checks the mechanical rules of spec 0005 D2 over the code on the simulated path: rule 1 (no goroutines started), rule 3 (no global random source, no package-level channel or WaitGroup), rule 4 (map ranges sorted or annotated) and rule 6's clauses on context.Background and context.WithoutCancel.
sim/locktable
Package locktable is the simulator's lock table (spec 0005 D3): the file store's Locker, with one view per simulated process, so that the simulator decides whose locks a crash releases, can grant owner.lock to several supervisors at once, and can revoke it from a live holder.
 Package locktable is the simulator's lock table (spec 0005 D3): the file store's Locker, with one view per simulated process, so that the simulator decides whose locks a crash releases, can grant owner.lock to several supervisors at once, and can revoke it from a live holder.
Package memory is an in-memory dispatch adapter in two shapes, Slurm and Kubeflow, that implements the whole SPI, including all five optional interfaces (spec 0003 D11).
 Package memory is an in-memory dispatch adapter in two shapes, Slurm and Kubeflow, that implements the whole SPI, including all five optional interfaces (spec 0003 D11).
Package model holds go/dispatch's value types: the job vocabulary, states and causes, holds, recovery policy and plans, actors, capabilities, and the adapter SPI's value types.
 Package model holds go/dispatch's value types: the job vocabulary, states and causes, holds, recovery policy and plans, actors, capabilities, and the adapter SPI's value types.
Package records is go/dispatch's typed view of a job record held in go/ledger (spec 0002, appendix).
 Package records is go/dispatch's typed view of a job record held in go/ledger (spec 0002, appendix).
ledgerstore
Package ledgerstore is records.Records over a go/ledger store: dispatch's typed record on the store that keeps state opaque (spec 0002 appendix).
 Package ledgerstore is records.Records over a go/ledger store: dispatch's typed record on the store that keeps state opaque (spec 0002 appendix).
recordstest
Package recordstest is an in-memory records.Records with ownership, for tests and for the simulator's early stages before the go/ledger file store drives it.
 Package recordstest is an in-memory records.Records with ownership, for tests and for the simulator's early stages before the go/ledger file store drives it.
Package supervise is the supervisor: it executes what dispatch.Next says should happen next, through conditional commits and gated launches (spec 0004).
 Package supervise is the supervisor: it executes what dispatch.Next says should happen next, through conditional commits and gated launches (spec 0004).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.