compute

README

compute

_{part of the togo-framework — the full-stack Go + React framework}

togo's compute capability — submit batch/stream jobs to a pluggable engine. Distinct from worker (supervised background goroutine workers) and queue (app job dispatch): compute is for data/compute engines.

Main plugin: defines Compute + ships the built-in local backend. Engine backends register into the same slot:

Backend	Repo
local (default)	built in
Apache Beam	togo-framework/compute-beam
Apache Spark	togo-framework/compute-spark
Apache Flink	togo-framework/compute-flink
Databricks	togo-framework/compute-databricks

togo install togo-framework/compute-spark
togo provider:use compute spark

c := compute.FromKernel(k)
run, _ := c.Submit(ctx, compute.Job{Name: "nightly-etl", Cmd: []string{"python", "etl.py"}})

MIT © fadymondy

Documentation

Overview

Package compute is togo's compute capability — submitting batch/stream jobs to a pluggable engine. It is distinct from togo's `worker` (supervised background goroutine workers) and `queue` (app job dispatch): compute is for data/compute engines (Apache Beam, Spark, Flink, Databricks), selectable at runtime.

This repo is the MAIN plugin: it defines the Compute contract and ships the built-in `local` backend (runs the job as a local process). Engine backends live in their own repos (compute-beam, compute-spark, compute-flink, compute-databricks) and register into the same slot; pick one with `togo provider:use compute <name>` (or TOGO_COMPUTE_PROVIDER).

Index

• type Compute
  • func FromKernel(k *togo.Kernel) Compute
• type Job
• type Run

Types

type Compute

type Compute interface {
	Submit(ctx context.Context, job Job) (Run, error)
}

Compute submits jobs to an engine. The built-in `local` backend runs them as a local process; engine backends submit to Spark/Beam/Flink/Databricks.

func FromKernel

func FromKernel(k *togo.Kernel) Compute

FromKernel returns the active Compute backend (or nil if none registered).

type Job

type Job struct {
	Name string            `json:"name"`
	Cmd  []string          `json:"cmd"`
	Env  map[string]string `json:"env,omitempty"`
	Args map[string]string `json:"args,omitempty"` // engine-specific options
}

Job is a unit of compute work. Cmd is the command (entrypoint + args) the engine runs — a script/binary locally, or a jar/module submitted to Spark/Beam.

type Run

type Run struct {
	ID     string `json:"id"`
	Status string `json:"status"` // succeeded | failed | running
	Output string `json:"output,omitempty"`
}

Run is the result/handle of a submitted job.