openjd

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 13, 2026 License: AGPL-3.0 Imports: 17 Imported by: 0

Documentation

Overview

Package openjd implements parsing, validation, and parameter-space expansion for Open Job Description (OpenJD) job templates.

Specification version

This package targets the "jobtemplate-2023-09" specification revision, which is the version used by AWS Deadline Cloud and other OpenJD-compatible systems. The specificationVersion field in a submitted template must equal that literal string.

Usage

Parse a raw YAML or JSON submission, then validate and expand it: 

tmpl, err := openjd.Parse(rawBytes, openjd.FormatYAML)
if err != nil {
    // low-level decode error (bad YAML/JSON)
}

if errs := openjd.Validate(tmpl); len(errs) > 0 {
    for _, e := range errs {
        fmt.Printf("%s: %s\n", e.Pointer, e.Message)
    }
}

for i, step := range tmpl.Steps {
    tasks, err := openjd.ExpandParameterSpace(step.ParameterSpace)
    // each task is a map[string]string of parameter name → value
}

Format strings

OpenJD format strings use double-braces to reference job or task parameters, for example: "{{Param.FrameNumber}}" or "{{Task.Param.OutputFile}}". This package records format strings verbatim; interpolation is performed later by the worker when it executes the task.

Path-mapping rules

PATH-typed job parameters and task parameters hold filesystem or S3 paths. At execution time the worker resolves them to concrete local paths using the path-mapping JSON file written into the session working directory. The parser records PATH parameters as strings; no path resolution is performed here.

Index

Constants

View Source 
const SpecVersion = "jobtemplate-2023-09"

SpecVersion is the only specification version accepted by this package.

Variables

View Source 
var ErrInvalidTransition = errors.New("openjd: invalid state transition")

ErrInvalidTransition is returned when a requested status transition is not permitted by the task or step state machine.

Use errors.Is to test: 

err := ValidateTaskTransition(from, to)
if errors.Is(err, ErrInvalidTransition) { ... }

Functions

func CancelDependents 

func CancelDependents(ctx context.Context, st store.Store, jobID string) (int, []store.Task, error)

CancelDependents cancels every store.StepStatusPending step of jobID whose dependency graph includes a step that terminated unsuccessfully — that is, reached store.StepStatusFailed or store.StepStatusCanceled — and cancels each such step's store.TaskStatusPending tasks.

A pending step that depends on a failed or canceled step can never become ready (ResolveDependencies only promotes steps whose dependencies all reached completed), so it would otherwise strand forever and prevent the job from ever reaching a terminal state. Canceling it lets job-completion detection proceed.

It returns the number of steps newly canceled and the pending tasks that were transitioned to canceled (so the caller can fan those terminal task transitions out to subscribers). Each canceled task is stamped with store.FailureReasonUpstreamFailed in the same UPDATE, unless it already carries a more specific reason.

CancelDependents loops to a fixpoint: canceling a step makes it an unsuccessful dependency for its own dependents, so a single call propagates through transitive dependency chains. Calling it is idempotent — steps already past pending are skipped.

It is typically invoked by the scheduler immediately after any step transitions to failed or canceled.

func DeriveChunkBounds added in v0.2.0 

func DeriveChunkBounds(rows []TaskParams, ps *StepParameterSpace)

DeriveChunkBounds adds "<name>.Start" and "<name>.End" keys to every row in rows for each CHUNK[INT] parameter declared in ps. Each chunk value is a contiguous integer range encoding (e.g. "1-10", "5", "1-10:2"); Start is the smallest integer in the chunk and End the largest. It is a no-op when ps is nil or declares no CHUNK[INT] parameter, and is only meaningful (and only called) when the SQI_CHUNK_BOUNDS extension is enabled. A value that fails to parse is left without derived keys — expansion has already validated ranges.

func DeriveStorageType added in v0.2.0 

func DeriveStorageType(roots map[string]string) string

DeriveStorageType classifies a storage location from its roots: "s3" when every root is an S3 URI, "filesystem" when none are (or there are no roots), and "mixed" when the roots span both. The storage-location type is descriptive only — sqi keys real behavior off each root's scheme, not off this value.

func ExtractJobLevelLocRefs 

func ExtractJobLevelLocRefs(tmpl *JobTemplate) []string

ExtractJobLevelLocRefs returns the deduplicated set of location names referenced at job scope only — job-parameter defaults and job environments. Step references are intentionally excluded; use ExtractStepLocRefs for those.

func ExtractStepLocRefs 

func ExtractStepLocRefs(st StepTemplate) []string

ExtractStepLocRefs returns the deduplicated set of location names referenced in a single step (its script, environments, and parameter space range lists).

func ExtractTemplateLocRefs 

func ExtractTemplateLocRefs(tmpl *JobTemplate) []string

ExtractTemplateLocRefs returns the deduplicated set of location names referenced anywhere in the given job template (command args, environment variables, embedded file data, and task/job parameter defaults).

Only the location names are returned; use ExtractLocRefs when the full LocRef details are needed.

func IsS3Root added in v0.2.0 

func IsS3Root(root string) bool

IsS3Root reports whether root is an S3 object-store root — i.e. it begins with the lowercase "s3://" scheme. Uppercase "S3://" is intentionally NOT treated as S3 (it is a filesystem path), matching JoinPath's forward-slash rule.

func JoinPath 

func JoinPath(root, relPath string) string

JoinPath joins a resolved root with the relative path portion of a loc:// URI. relPath includes the leading slash (or is empty).

For S3 roots (e.g. "s3://bucket/prefix"), path separators are kept as "/". For filesystem roots, the OS-appropriate separator is used.

func ResolveDependencies 

func ResolveDependencies(ctx context.Context, st store.Store, jobID string) (int, error)

ResolveDependencies inspects every step of jobID and promotes any store.StepStatusPending step whose declared dependencies have all reached store.StepStatusCompleted to store.StepStatusReady, then transitions all of that step's store.TaskStatusPending tasks to store.TaskStatusReady.

It returns the number of steps newly promoted to ready.

Calling ResolveDependencies is idempotent: steps that are already past pending are skipped. It is typically invoked by the scheduler immediately after any step transitions to store.StepStatusCompleted.

Steps with an empty DependsOn list are always eligible. In normal operation such steps are created with store.StepStatusReady by Submitter.Submit, so they will simply be skipped here. If a no-dep step is somehow in pending state, ResolveDependencies will promote it correctly.

Note: ResolveDependencies considers only a single pass through the step list. Call it again if the first pass unblocked a step whose promotion might unblock further steps in the same job.

func ResolveLocURIs 

func ResolveLocURIs(s string, resolve func(name, relPath string) (string, error)) (string, error)

ResolveLocURIs replaces every loc:// URI in s with the concrete path returned by resolve(locationName, relPath). relPath includes the leading slash; it is empty when the URI is a bare "loc://name" reference.

resolve is called once per unique URI occurrence; if resolve returns an error the function aborts and returns the error.

func ResolveParameterSpaceParams 

func ResolveParameterSpaceParams(ps *StepParameterSpace, jobParams map[string]string) (*StepParameterSpace, ValidationErrors)

ResolveParameterSpaceParams returns a new *StepParameterSpace with every {{Param.<name>}} and {{RawParam.<name>}} reference in RangeExpr and RangeList entries substituted with the corresponding value from jobParams.

Both "Param.<name>" and "RawParam.<name>" resolve to the same bound string value — path mapping is worker-side and out of scope at submit time.

The input ps is never mutated; a fresh struct (with a new slice of TaskParamDefinition values) is always returned on success.

If ps is nil, (nil, nil) is returned immediately.

On a malformed reference or an unknown variable, a ValidationError is accumulated with a pointer of the form 

/parameterSpace/taskParameterDefinitions/<i>/range
/parameterSpace/taskParameterDefinitions/<i>/range/<j>   (RangeList entry j)

All task-parameter definitions are inspected before returning so callers receive a complete error list in one round-trip. On any error (nil, errs) is returned.

func ResolveRoot 

func ResolveRoot(locationName string, roots map[string]string, computeLocation string) (string, error)

ResolveRoot returns the concrete root path for locationRoots given a compute location name. It prefers the compute-location-specific root, falls back to "default", and returns an error if neither is present.

func ValidLocationName 

func ValidLocationName(name string) bool

ValidLocationName reports whether name is a referenceable storage-location name — i.e. one that a loc:// URI can address. Names containing whitespace, "/", or quotes are rejected because the loc:// parser cannot match them.

func ValidS3Root added in v0.2.0 

func ValidS3Root(root string) bool

ValidS3Root reports whether root is a well-formed s3://bucket[/prefix] URI: it must carry the s3:// scheme, a non-empty bucket segment, and no whitespace in that bucket segment. The optional key prefix after the first "/" is not constrained (object keys are permissive across providers).

func ValidateStepTransition 

func ValidateStepTransition(from, to store.StepStatus) error

ValidateStepTransition returns nil if transitioning a step from old to new status is permitted by the state machine, or a descriptive error wrapping ErrInvalidTransition otherwise.

func ValidateTaskTransition 

func ValidateTaskTransition(from, to store.TaskStatus) error

ValidateTaskTransition returns nil if transitioning a task from old to new status is permitted by the state machine, or a descriptive error wrapping ErrInvalidTransition otherwise.

Types

type Action 

type Action struct {
	// Command is the executable (may be a format string).
	Command string
	// Args is the optional argument list (each entry may be a format string).
	Args []string
	// TimeoutSeconds, when > 0, limits how long the action may run.
	TimeoutSeconds int
	// Cancelation describes how to stop a running action.
	Cancelation *CancelationMethod
}

Action is an executable command run as part of a step or environment script.

type AmountRequirement 

type AmountRequirement struct {
	// Name is the capability name (e.g. "amount.worker.vcpu").
	Name string
	// Min and Max bound the acceptable range.  Nil means no bound.
	// Stored as strings for lossless round-tripping of int and float values.
	Min *string
	Max *string
}

AmountRequirement is a single quantifiable host capability requirement.

type AttributeRequirement 

type AttributeRequirement struct {
	// Name is the capability name (e.g. "attr.worker.os.family").
	Name string
	// AnyOf requires the host attribute to match at least one value.
	AnyOf []string
	// AllOf requires the host attribute to match all listed values.
	AllOf []string
}

AttributeRequirement is a categorical host capability requirement.

type CancelationMethod 

type CancelationMethod struct {
	// Mode selects the cancelation strategy.
	Mode CancelationMode
	// NotifyPeriodSeconds is the delay between SIGTERM and SIGKILL when Mode is
	// [CancelModeNotifyThenTerminate].  0 means use the spec default (120s for
	// onRun actions, 30s for others).
	NotifyPeriodSeconds int
}

CancelationMethod configures how a running Action is canceled.

type CancelationMode 

type CancelationMode string

CancelationMode is the cancelation strategy for an Action. 

const (
	// CancelModeTerminate sends SIGKILL / TerminateProcess immediately.
	CancelModeTerminate CancelationMode = "TERMINATE"
	// CancelModeNotifyThenTerminate sends SIGTERM first, then SIGKILL after
	// NotifyPeriodSeconds.
	CancelModeNotifyThenTerminate CancelationMode = "NOTIFY_THEN_TERMINATE"
)

type ControlType added in v0.2.0 

type ControlType string

ControlType is the OpenJD base-spec userInterface control for a job parameter. 

const (
	// ControlLineEdit is a single-line text input.
	ControlLineEdit ControlType = "LINE_EDIT"
	// ControlMultilineEdit is a multi-line text input.
	ControlMultilineEdit ControlType = "MULTILINE_EDIT"
	// ControlDropdownList is a dropdown; requires allowedValues.
	ControlDropdownList ControlType = "DROPDOWN_LIST"
	// ControlCheckBox is a two-state checkbox; requires exactly two allowedValues.
	ControlCheckBox ControlType = "CHECK_BOX"
	// ControlChipInput is a multi-value chip/tag input.
	ControlChipInput ControlType = "CHIP_INPUT"
	// ControlHidden hides the parameter from the generated form.
	ControlHidden ControlType = "HIDDEN"
	// ControlSpinBox is a numeric spinner; valid only on INT/FLOAT.
	ControlSpinBox ControlType = "SPIN_BOX"
)

type EmbeddedFile 

type EmbeddedFile struct {
	// Name is the identifier used to reference this file.
	Name string
	// Filename is the on-disk name; if empty the worker generates one.
	Filename string
	// Data is the file content (may contain format-string references).
	Data string
	// Type is the file-type discriminator.  In jobtemplate-2023-09 the only
	// allowed value is [EmbeddedFileTypeText].  Empty means absent; validation
	// rejects it as required.
	Type EmbeddedFileType
	// Runnable, when true, sets the file's execute permission.
	Runnable bool
	// EndOfLine is "AUTO", "LF", or "CRLF".  Empty means "AUTO".
	EndOfLine string
}

EmbeddedFile is a plain-text file embedded directly in the job template. The worker materializes it to the session working directory before running the associated actions.

type EmbeddedFileType 

type EmbeddedFileType string

EmbeddedFileType is the type discriminator for an EmbeddedFile. In jobtemplate-2023-09 the only allowed value is EmbeddedFileTypeText. An empty value means the field was absent in the source document. 

const (
	// EmbeddedFileTypeText is the only supported embedded-file type in
	// jobtemplate-2023-09.  It means the file contains UTF-8 plain text.
	EmbeddedFileTypeText EmbeddedFileType = "TEXT"
)

type Environment 

type Environment struct {
	// Name is unique within its containing list.
	Name string
	// Description is optional documentation.
	Description string
	// Script defines the onEnter and onExit actions.
	Script *EnvironmentScript
	// Variables are environment variables set for all actions in the session.
	Variables map[string]string
}

Environment is an OpenJD environment block used at the job or step level. Environments are entered once per session and exited in reverse order.

type EnvironmentActions 

type EnvironmentActions struct {
	OnEnter *Action
	OnExit  *Action
}

EnvironmentActions are the lifecycle hooks for an Environment. At least one of OnEnter or OnExit must be non-nil.

type EnvironmentScript 

type EnvironmentScript struct {
	// EmbeddedFiles are plain-text files materialized before actions run.
	EmbeddedFiles []EmbeddedFile
	// Actions holds the onEnter and onExit lifecycle hooks.
	Actions EnvironmentActions
}

EnvironmentScript holds the runnable actions for an Environment.

type Extension added in v0.2.0 

type Extension struct {
	// Name is the OpenJD extension identifier, matching [A-Z_0-9]{3,128}.
	Name string
	// Origin is official (upstream OpenJD) or vendor (sqi-defined).
	Origin Origin
	// Status is the implementation status, e.g. "supported".
	Status string
	// Summary is a one-line description.
	Summary string
	// DocPath points at the extension's contribution doc, repo-relative.
	DocPath string
}

Extension is one entry in the supported-extension registry.

func LookupExtension added in v0.2.0 

func LookupExtension(name string) (Extension, bool)

LookupExtension returns the registry entry for name and whether it exists.

type Format 

type Format int

Format identifies the wire format of a raw template submission. 

const (
	// FormatYAML means the raw bytes are YAML.
	FormatYAML Format = iota
	// FormatJSON means the raw bytes are JSON.
	FormatJSON
)

type HostRequirements 

type HostRequirements struct {
	// Amounts are quantifiable requirements such as CPUs, RAM, or software slots.
	Amounts []AmountRequirement
	// Attributes are categorical requirements such as OS or CPU architecture.
	Attributes []AttributeRequirement
}

HostRequirements declares the worker capabilities required by a step.

type JobParamType 

type JobParamType string

JobParamType is the discriminator for a JobParameter. 

const (
	// JobParamTypeInt is a 64-bit integer parameter.
	JobParamTypeInt JobParamType = "INT"
	// JobParamTypeFloat is a decimal (float64) parameter.
	JobParamTypeFloat JobParamType = "FLOAT"
	// JobParamTypeString is an arbitrary UTF-8 string parameter.
	JobParamTypeString JobParamType = "STRING"
	// JobParamTypePath is a filesystem or S3 path parameter.
	JobParamTypePath JobParamType = "PATH"
)

type JobParameter 

type JobParameter struct {
	// Name is the parameter identifier; must match [A-Za-z_][A-Za-z0-9_]*.
	Name string
	// Type discriminates INT / FLOAT / STRING / PATH.
	Type JobParamType
	// Description is optional documentation shown in UI elements.
	Description string
	// Default is the value used when the submitter does not provide one.
	// Stored as a string regardless of Type to avoid lossy conversion.
	// Nil means no default (parameter is required).
	Default *string
	// AllowedValues, when non-nil, enumerates the only acceptable values.
	AllowedValues []string
	// MinValue / MaxValue constrain INT and FLOAT parameters.
	// Stored as strings for lossless round-tripping.
	MinValue *string
	MaxValue *string
	// MinLength / MaxLength constrain STRING and PATH parameters.
	MinLength *int
	MaxLength *int
	// ObjectType narrows how the PATH value is interpreted: FILE or DIRECTORY.
	// Empty means absent; the spec default is DIRECTORY.
	// Only valid for PATH parameters; validation rejects it on other types.
	ObjectType PathObjectType
	// DataFlow describes the data-flow direction of the PATH value.
	// Empty means absent; the spec default is NONE.
	// Only valid for PATH parameters; validation rejects it on other types.
	DataFlow PathDataFlow
	// UserInterface carries optional OpenJD base-spec presentation hints.
	// Nil when the parameter declares no userInterface object.
	UserInterface *ParameterUserInterface
}

JobParameter is one entry in JobTemplate.ParameterDefinitions.

type JobTemplate 

type JobTemplate struct {
	// SpecificationVersion must equal [SpecVersion].
	SpecificationVersion string

	// Name is the human-readable job name.  Required; may contain format-string
	// references such as "Render {{Param.SceneName}}".
	Name string

	// Description is an optional free-form description (no functional effect).
	Description string

	// Extensions is the optional list of named OpenJD feature extensions to
	// enable (e.g. "feature-bundle-1").
	Extensions []string

	// PathTranslation is the parsed SQI_PATH_TRANSLATION extension block, or nil
	// when the extension is not declared.
	PathTranslation *PathTranslation

	// ParameterDefinitions declares the job-level parameters that submitters
	// must or may provide.  Parameter names must be unique.
	ParameterDefinitions []JobParameter

	// JobEnvironments are environments that wrap every session in this job.
	// They are entered in order and exited in reverse order.
	JobEnvironments []Environment

	// Steps is the ordered list of step definitions.  At least one step is
	// required.
	Steps []StepTemplate
}

JobTemplate is the root document of an OpenJD submission.

func Parse 

func Parse(data []byte, f Format) (*JobTemplate, error)

Parse decodes raw YAML or JSON bytes into a JobTemplate. It returns a low-level decode error for malformed documents; call Validate afterward to check semantic correctness.

type LocRef 

type LocRef struct {
	// LocationName is the symbolic storage-location name (e.g. "nas_shows").
	LocationName string
	// RelPath is the path portion after the location name, including the
	// leading slash, or empty for a bare root reference (e.g. "loc://nas_shows").
	RelPath string
	// Raw is the full original URI string.
	Raw string
}

LocRef is a single named-location reference extracted from a template string.

func ExtractLocRefs 

func ExtractLocRefs(s string) []LocRef

ExtractLocRefs scans s for all loc:// URI occurrences and returns them in order of appearance. Duplicates are not deduplicated.

type Origin added in v0.2.0 

type Origin string

Origin distinguishes an upstream OpenJD extension from one defined by sqi. 

const (
	// OriginOfficial marks an extension defined by the upstream OpenJD spec.
	OriginOfficial Origin = "official"
	// OriginVendor marks an extension defined by sqi. Vendor extension names
	// MUST carry the SQI_ prefix so they can never collide with a future
	// official OpenJD name; if upstreamed, the prefix is dropped and the entry
	// flips to OriginOfficial (the "promotion path").
	OriginVendor Origin = "vendor"
)

type ParameterUserInterface added in v0.2.0 

type ParameterUserInterface struct {
	// Control selects the input widget.
	Control ControlType
	// Label is the human-readable field label.
	Label string
	// GroupLabel groups related fields in the generated form.
	GroupLabel string
	// Decimals sets the precision for a SPIN_BOX on a FLOAT parameter.
	Decimals *int
	// SingleStepRemoval applies to CHIP_INPUT only.
	SingleStepRemoval *bool
}

ParameterUserInterface is the OpenJD base-spec userInterface hint object on a job parameter. It is presentation metadata consumed by submission UIs; the server parses, validates, and carries it but does not act on it.

type PathDataFlow 

type PathDataFlow string

PathDataFlow is the dataFlow discriminator for a PATH JobParameter. An empty value means the field was not set; the OpenJD spec default when absent is NONE. Only valid on PATH parameters. 

const (
	// PathDataFlowNone means the path has no specific data-flow direction.
	// This is the spec default when dataFlow is absent.
	PathDataFlowNone PathDataFlow = "NONE"
	// PathDataFlowIn means the path is read as input by the job.
	PathDataFlowIn PathDataFlow = "IN"
	// PathDataFlowOut means the path is written as output by the job.
	PathDataFlowOut PathDataFlow = "OUT"
	// PathDataFlowInOut means the path is both read and written.
	PathDataFlowInOut PathDataFlow = "INOUT"
)

type PathDelivery added in v0.2.0 

type PathDelivery struct {
	Kind PathDeliveryKind
	// Pattern is the flag template for DeliveryCommandFlags (uses {src}/{dest}).
	Pattern string
	// Variable is the environment variable name for DeliveryEnvironment.
	Variable string
}

PathDelivery is one enabled delivery plus its optional settings.

func DefaultPathDeliveries added in v0.2.0 

func DefaultPathDeliveries() []PathDelivery

DefaultPathDeliveries is the implicit delivery set used when the extension is not declared: today's automatic behavior (swap in place + translation file).

type PathDeliveryKind added in v0.2.0 

type PathDeliveryKind string

PathDeliveryKind identifies one path-delivery mechanism of the SQI_PATH_TRANSLATION extension. 

const (
	// DeliverySwapInPlace substitutes concrete paths into the command/args.
	DeliverySwapInPlace PathDeliveryKind = "swap_in_place"
	// DeliveryTranslationFile writes the OpenJD pathmapping-1.0 file.
	DeliveryTranslationFile PathDeliveryKind = "translation_file"
	// DeliveryCommandFlags appends per-pair flags rendered from Pattern.
	DeliveryCommandFlags PathDeliveryKind = "command_flags"
	// DeliveryEnvironment sets Variable to the joined src=dest pairs.
	DeliveryEnvironment PathDeliveryKind = "environment"
	// DeliveryStageLocally copies inputs to worker-local scratch.
	DeliveryStageLocally PathDeliveryKind = "stage_locally"
)

type PathObjectType 

type PathObjectType string

PathObjectType is the objectType discriminator for a PATH JobParameter. An empty value means the field was not set; the OpenJD spec default when absent is DIRECTORY. Only valid on PATH parameters. 

const (
	// PathObjectTypeFile means the path refers to a regular file.
	PathObjectTypeFile PathObjectType = "FILE"
	// PathObjectTypeDirectory means the path refers to a directory.
	// This is the spec default when objectType is absent.
	PathObjectTypeDirectory PathObjectType = "DIRECTORY"
)

type PathTranslation added in v0.2.0 

type PathTranslation struct {
	Deliveries []PathDelivery
}

PathTranslation is the parsed SQI_PATH_TRANSLATION block.

type StepActions 

type StepActions struct {
	// OnRun is executed once per task.
	OnRun Action
}

StepActions holds the lifecycle actions for tasks of a step.

type StepDependency 

type StepDependency struct {
	// DependsOn is the name of the prerequisite step.
	DependsOn string
}

StepDependency names a step that must reach store.StepStatusCompleted before this step's tasks may be scheduled.

type StepParameterSpace 

type StepParameterSpace struct {
	// TaskParameterDefinitions declares all task parameters.  1–16 entries.
	TaskParameterDefinitions []TaskParamDefinition
	// Combination is an optional expression that controls how parameter ranges
	// are combined.  Identifiers connected with * form a Cartesian product;
	// identifiers grouped in parentheses (A,B,C) are zipped (association).
	// When nil, all parameters are joined with *.
	Combination *string
}

StepParameterSpace defines the multidimensional task parameter space for a step. Expanding it via ExpandParameterSpace yields one TaskParams per task.

type StepScript 

type StepScript struct {
	// EmbeddedFiles are plain-text files written to disk before each action.
	EmbeddedFiles []EmbeddedFile
	// Actions holds the task lifecycle hooks.
	Actions StepActions
}

StepScript is the executable definition attached to a StepTemplate.

type StepTemplate 

type StepTemplate struct {
	// Name identifies this step; must be unique within the job.
	Name string
	// Description is optional documentation.
	Description string
	// Script is the executable definition of the step's tasks.
	Script *StepScript
	// StepEnvironments are per-step environments entered after job environments.
	StepEnvironments []Environment
	// ParameterSpace defines the task parameter combinations for this step.
	// Nil means the step produces exactly one task with no parameters.
	ParameterSpace *StepParameterSpace
	// HostRequirements declares the host capabilities required by this step.
	HostRequirements *HostRequirements
	// Dependencies lists the steps that must complete before this one starts.
	Dependencies []StepDependency
}

StepTemplate defines one step within a JobTemplate.

type SubmitOptions 

type SubmitOptions struct {
	// FarmID is the farm this job belongs to. Required.
	FarmID string
	// QueueID is the queue this job is placed in. Required.
	QueueID string
	// Owner is the human responsible for the job (e.g. artist login name).
	Owner string
	// Submitter is the authenticated identity that placed the job (e.g. a
	// service account or API token owner). May differ from Owner.
	Submitter string
	// Priority overrides the default priority (50). Values ≤ 0 are treated as
	// the default.
	Priority int
	// Project is an optional label for grouping jobs in the UI and API.
	Project string
	// Name overrides the job name. When empty, the template's own name
	// (tmpl.Name) is used. Lets callers (e.g. product submissions) give each
	// job a distinct, human-meaningful name without editing the template.
	Name string
	// Parameters holds the caller-supplied values for job-level parameters
	// declared in the template's parameterDefinitions.  Keys are parameter
	// names; values are raw strings.  Missing entries are filled from each
	// parameter's Default; parameters with neither a supplied value nor a
	// default produce a [SubmitValidationError].
	Parameters map[string]string
	// MaxAttempts, RetryDelaySeconds, and FailureLimit are optional per-job
	// retry overrides. Nil means inherit (queue -> farm -> server default).
	MaxAttempts       *int
	RetryDelaySeconds *int
	FailureLimit      *int
	// DependsOn lists the IDs of upstream jobs this job must wait for (whole-job
	// cross-job dependencies). Each must already exist and be in the same farm;
	// none may already be failed or canceled. When any is not yet completed, the
	// job is created in store.JobStatusBlocked with all tasks held pending.
	DependsOn []string
}

SubmitOptions carries the caller-supplied metadata attached to a job at submission time. All fields are optional except FarmID and QueueID.

type SubmitResult 

type SubmitResult struct {
	// Job is the persisted job row, including the verbatim raw template.
	Job store.Job
	// Steps holds the persisted step rows in template order.
	Steps []store.Step
	// Tasks holds all persisted task rows across all steps.
	Tasks []store.Task
	// BoundParameters is the fully-resolved name→value map produced by
	// [BindJobParameters].  Defaults from the template are merged in; every
	// declared parameter is guaranteed to have an entry here.
	// Later tasks ({{Param.*}} resolution, worker carry) consume this map.
	BoundParameters map[string]string
}

SubmitResult is returned by Submitter.Submit and holds every persisted row created during submission.

type SubmitValidationError 

type SubmitValidationError struct {
	// Cause is the underlying parse, validation, or storage-location error.
	Cause error
}

SubmitValidationError is returned by Submitter.Submit when the job template fails to parse, fails OpenJD validation, or references storage locations that are not registered or lack the required root coverage.

The REST layer maps this error type to HTTP 422 (Unprocessable Entity) via errors.As so that callers get a precise status code regardless of how the underlying error message is phrased.

func (*SubmitValidationError) Error 

func (e *SubmitValidationError) Error() string

func (*SubmitValidationError) Unwrap 

func (e *SubmitValidationError) Unwrap() error

type Submitter 

type Submitter struct {
	// contains filtered or unexported fields
}

Submitter handles the full OpenJD submission pipeline: parse, validate, expand the parameter space, persist the normalized job/step/task rows alongside the verbatim raw template, and enforce storage-location coverage.

Create one with NewSubmitter or NewSubmitterWithOptions and reuse it across requests.

func NewSubmitter 

func NewSubmitter(st store.Store) *Submitter

NewSubmitter returns a Submitter backed by st with default options (EnforceLimits: true). The store must implement store.StorageLocationStore (which store.Store always does) so that named-location references in submitted templates can be validated at submission time.

func NewSubmitterWithOptions 

func NewSubmitterWithOptions(st store.Store, opts SubmitterOptions) *Submitter

NewSubmitterWithOptions returns a Submitter backed by st with the supplied options. Use this when the caller needs to control SubmitterOptions.EnforceLimits based on operator configuration.

func (*Submitter) Submit 

func (s *Submitter) Submit(
	ctx context.Context,
	rawTemplate string,
	format store.TemplateFormat,
	opts SubmitOptions,
) (*SubmitResult, error)

Submit parses rawTemplate, validates it, expands each step's parameter space into concrete tasks, and persists:

  • one store.Job row containing rawTemplate verbatim
  • one store.Step row per step in the template
  • one store.Task row per element of the parameter-space expansion

Steps whose DependsOn list is empty start in store.StepStatusReady; steps with dependencies start in store.StepStatusPending. Tasks inherit their step's initial status.

Submit does not run in a database transaction. If it fails partway through, orphaned rows may remain; the REST layer or a cleanup sweep should handle such cases by checking job.Status == pending with no tasks.

type SubmitterOptions 

type SubmitterOptions struct {
	// EnforceLimits controls whether quantitative OpenJD limit checks are run
	// during validation.  Mirrors [ValidateOptions.EnforceLimits].
	// Defaults to true when using [NewSubmitter].
	EnforceLimits bool
}

SubmitterOptions carries optional configuration for a Submitter.

type TaskChunks 

type TaskChunks struct {
	// DefaultTaskCount is the number of INT values to group per task.
	DefaultTaskCount int
	// TargetRuntimeSeconds, when > 0, enables adaptive chunking.
	TargetRuntimeSeconds *int
	// RangeConstraint is "CONTIGUOUS" (default) or "NONCONTIGUOUS".
	RangeConstraint string
}

TaskChunks configures chunked task grouping for a CHUNK[INT] parameter.

type TaskParamDefinition 

type TaskParamDefinition struct {
	// Name is the parameter identifier.
	Name string
	// Type discriminates INT / FLOAT / STRING / PATH / CHUNK[INT].
	Type TaskParamType
	// RangeExpr holds the raw range string for INT and CHUNK[INT] parameters
	// when the template uses range expression syntax (e.g. "1-100:2").
	// Exactly one of RangeExpr or RangeList is set for each definition.
	RangeExpr *string
	// RangeList holds the explicit value list for all other types, and also for
	// INT/CHUNK[INT] when the template provides an array of values.
	RangeList []string
	// Chunks is non-nil only for CHUNK[INT] parameters.
	Chunks *TaskChunks
}

TaskParamDefinition is one parameter declaration in a StepParameterSpace.

type TaskParamType 

type TaskParamType string

TaskParamType is the discriminator for a TaskParamDefinition. 

const (
	// TaskParamTypeInt is an integer task parameter.
	TaskParamTypeInt TaskParamType = "INT"
	// TaskParamTypeFloat is a float task parameter.
	TaskParamTypeFloat TaskParamType = "FLOAT"
	// TaskParamTypeString is a string task parameter.
	TaskParamTypeString TaskParamType = "STRING"
	// TaskParamTypePath is a path task parameter.
	TaskParamTypePath TaskParamType = "PATH"
	// TaskParamTypeChunkInt is a chunked integer task parameter that groups
	// multiple frame values into a single task.
	TaskParamTypeChunkInt TaskParamType = "CHUNK[INT]"
)

type TaskParams 

type TaskParams map[string]string

TaskParams is one materialized row from a step's parameter-space expansion. Keys are parameter names; values are their string representations. An empty map means the step has no parameters (produces a single task).

func ExpandParameterSpace 

func ExpandParameterSpace(ps *StepParameterSpace) ([]TaskParams, error)

ExpandParameterSpace materializes a step's parameter space into the concrete set of task parameter maps that the scheduler will use to create individual tasks.

When ps is nil the step has a zero-dimensional parameter space: a single task is produced with an empty TaskParams map.

The combination expression (if provided) controls how parameter ranges are joined:

  • Identifiers separated by * form a Cartesian product (outer join).
  • Identifiers grouped in parentheses (A,B,C) are zipped (inner join / association). All parameters in a group must have the same range length.

If no combination expression is provided, all parameters are joined with *.

type ValidateOptions 

type ValidateOptions struct {
	// EnforceLimits gates quantitative limit checks: maximum name lengths,
	// element counts, reserved-name rules, etc.
	//
	// When false those checks are skipped — useful in operator environments
	// that predate strict limit enforcement and cannot yet update all templates.
	//
	// NOTE: the quantitative limit checks live in [validateLimits] (and the
	// helpers it calls). Every limit check MUST be guarded by opts.EnforceLimits
	// and belong to validateLimits — do not scatter gated checks elsewhere.
	// Resource-exhaustion guards (e.g. the [maxRangeValues] cap in
	// parseIntRangeExpr) are NOT limit checks: they always apply, regardless of
	// this flag.
	EnforceLimits bool
}

ValidateOptions controls optional validation behavior passed to ValidateWithOptions.

type ValidationError 

type ValidationError struct {
	// Pointer is a JSON Pointer (RFC 6901) to the offending field, e.g.
	// "/steps/0/dependencies/0/dependsOn".
	Pointer string
	// Message is a human-readable description of the problem.
	Message string
}

ValidationError is a single validation failure with a JSON-pointer path to the offending field.

func (ValidationError) Error 

func (e ValidationError) Error() string

type ValidationErrors 

type ValidationErrors []ValidationError

ValidationErrors is the result of Validate: a slice of zero or more errors.

func BindJobParameters 

func BindJobParameters(params []JobParameter, provided map[string]string) (map[string]string, ValidationErrors)

BindJobParameters resolves each declared job parameter to a concrete string value, validates every value against its type and constraints, and returns the fully-bound name→value map.

Resolution order per declared parameter:

  1. provided[name] — the caller-supplied value, if present.
  2. p.Default — the template default, if the parameter has one.
  3. Required-but-missing error: the parameter has neither a supplied value nor a default.

Any key in provided that does not match a declared parameter name produces an "unknown parameter" error. All errors are accumulated before returning so callers receive a complete problem list in one round-trip.

Pointer conventions:

  • Unknown provided key → /parameterValues/<name>
  • Required-but-missing → /parameterDefinitions/<name>
  • Type / constraint err → /parameterValues/<name>

AllowedValues comparison:

  • INT — both value and each allowed entry are parsed as int64; "01" and "1" are considered equal.
  • FLOAT — both parsed as float64; "1.00" and "1.0" are considered equal.
  • STRING / PATH — exact string comparison.

On success, returns the resolved map and nil errors. On failure, returns (nil, errors).

func Validate 

func Validate(t *JobTemplate) ValidationErrors

Validate performs semantic validation of a parsed JobTemplate and returns all detected problems. An empty slice means the template is valid.

Validations performed:

  • specificationVersion must equal SpecVersion
  • name must not be empty
  • at least one step is required
  • job parameter names must be unique and match the identifier pattern
  • job parameter type must be INT, FLOAT, STRING, or PATH
  • step names must be unique and non-empty
  • each step dependency must reference a declared step name
  • step dependency graph must be acyclic
  • task parameter names within a step must be unique
  • task parameter types must be INT, FLOAT, STRING, PATH, or CHUNK[INT]
  • INT/CHUNK[INT] range expressions must be parseable
  • combination expression must reference only declared task parameter names

Validate is a thin wrapper around ValidateWithOptions with ValidateOptions.EnforceLimits set to true.

func ValidateWithOptions 

func ValidateWithOptions(t *JobTemplate, opts ValidateOptions) ValidationErrors

ValidateWithOptions is like Validate but accepts a ValidateOptions value so callers can control optional validation behavior — currently whether quantitative limit checks are enforced.

Existing callers should continue to use Validate; this entry point is for the submission pipeline where the operator may disable limit enforcement via config.

func (ValidationErrors) Error 

func (e ValidationErrors) Error() string

Source Files

View all Source files

Directories

Path Synopsis
Package fmtstring resolves Open Job Description (OpenJD) format strings.
 Package fmtstring resolves Open Job Description (OpenJD) format strings.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.