README
¶
goa
Pure-Go polyglot embedding. Run Go, Python, TypeScript/JavaScript, and
Rust/WASM services in one process — as goroutines, no cgo, no subprocesses, no
GIL — so a fleet of services collapses into a single static binary
(CGO_ENABLED=0) that scales horizontally by replication.
import "github.com/hanzoai/goa"
Why
hanzoai/cloud is one Go binary: subsystems register and mount onto a zip (gofiber) app. Most of our backends are not Go — Langflow is Python, the migrated explorer services are Rust, assorted tools are TypeScript. The choice was a swarm of sidecar containers, or embed the runtimes. goa embeds them, in pure Go, preserving the static build.
Model
One contract, every language: JSON in, JSON out.
type Module interface {
Invoke(ctx context.Context, fn string, payload []byte) ([]byte, error)
Funcs() []string
Close() error
}
type Engine interface {
Lang() string
Aliases() []string
Load(ctx context.Context, src []byte, opt LoadOptions) (Module, error)
}
Four engines register themselves, all pure-Go:
| Lang | Engine | Notes |
|---|---|---|
| Go | native | map[string]HandlerFunc, zero overhead |
| JavaScript / TS | goja + esbuild | TS transpiled to ES2020 CommonJS |
| Python | gpython | no cgo, no GIL → true goroutine parallelism |
| Rust / WASM | wazero | wasm32 + WASI, trivial string ABI |
Engines that run single-threaded code (goja, gpython, a wasm instance) are made concurrent by a Pool: N independent interpreters, one per in-flight request, so every HTTP route is still just a goroutine.
Use
Embed source directly
m, _ := goa.Load(ctx, "python", []byte(`
import json
def greet(p):
return json.dumps({"hello": json.loads(p)["name"]})
`), goa.LoadOptions{Name: "greeter"})
out, _ := m.Invoke(ctx, "greet", []byte(`{"name":"ada"}`))
// {"hello":"ada"}
Mount Go natively (zero overhead)
m := goa.NewNativeModule(map[string]goa.HandlerFunc{
"greet": func(ctx context.Context, p []byte) ([]byte, error) { return p, nil },
})
Drop-in services: a manifest + a source file
hello-py.manifest.json
{
"name": "hello-py",
"lang": "python",
"source": "hello.py",
"pool": 8,
"prefix": "/v1/hello-py",
"routes": [{ "method": "POST", "path": "/greet", "func": "greet" }]
}
services, _ := goa.LoadDir(ctx, "services") // every *.manifest.json
mux := http.NewServeMux()
for _, s := range services {
s.Mount(mux)
}
In hanzoai/cloud the same Service mounts onto the gofiber app via
zip.AdaptNetHTTP(svc.Handler()) — goa stays host-agnostic.
Python stdlib
gpython is "batteries not included." goa ships Go-backed implementations of
the modules our backends actually use, registered under their real names — so
the Python keeps import json and runs at Go speed (the modules that are
C-accelerated in CPython are Go-native here). json ships today; re,
hashlib, base64, datetime, uuid extend the same pyToGo/goToPy
pattern in stdlib_gpython.go.
Two tiers
goa is Tier 1: in-process logic, pure-Go, for handlers and transforms. Anything that needs a real OS process — a long-running server, a GPU workload, heavy native Python (NumPy/Torch), an unported C extension — is Tier 2: run it out-of-process and front it with a native proxy module. Tier 1 is the default; reach for Tier 2 only when the workload genuinely needs an OS boundary.
Rust / WASM ABI
A guest module exports:
alloc(len: i32) -> i32 // buffer pointer for the payload
dealloc(ptr: i32, len: i32) // optional
<fn>(ptr: i32, len: i32) -> i64 // returns (resultPtr << 32 | resultLen)
The host writes JSON at alloc()'s pointer, calls <fn>, reads JSON back. A
#[goa::handler] proc-macro generates this boilerplate on the Rust side.
License
Apache-2.0
Documentation
¶
Overview ¶
Package goa is a pure-Go polyglot embedding runtime. It runs Go, JavaScript/ TypeScript (goja), Python (gpython) and Rust/WASM (wazero) behind ONE interface, so every Hanzo cloud service — whatever language it is written in — can be compiled into a single static (CGO_ENABLED=0) Go binary and served as goroutines. No cgo, no subprocess, no per-language runtime to deploy.
The one calling convention, shared by every engine and language, is JSON-in / JSON-out: a handler receives a JSON payload (bytes) and returns a JSON result (bytes). That keeps the boundary trivial to port to — a service is just a function `(jsonIn) -> jsonOut` in its native language — and lets the host (e.g. hanzoai/zip / gofiber) mount any handler as an HTTP/ZAP route via the adapters in http.go.
Concurrency: an engine's Module is single-threaded (a goja Runtime, a gpython Context, a wasm instance are not safe to share). Pool (pool.go) keeps N Modules per service and hands one to each in-flight call, so N goroutines run N requests in true parallel — including Python, because gpython is plain Go with no shared GIL.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Engine ¶
type Engine interface {
// Lang is the canonical language id: "go", "javascript", "python", "wasm".
Lang() string
// Aliases are additional ids that resolve to this engine (e.g. "ts","py").
Aliases() []string
// Load compiles src into a fresh Module.
Load(ctx context.Context, src []byte, opt LoadOptions) (Module, error)
}
Engine compiles source in one language into a Module. Engines MUST be pure Go (no cgo) so the host binary stays statically linkable.
type HandlerFunc ¶
HandlerFunc is a native Go handler in the canonical JSON-in/JSON-out shape.
type LoadOptions ¶
type LoadOptions struct {
Name string // logical module name (diagnostics, require keys)
Env map[string]string // exposed as process.env / os.environ to the guest
}
LoadOptions configures a single Load.
type Manifest ¶
type Manifest struct {
Name string `json:"name"`
Lang string `json:"lang"`
Source string `json:"source"` // path to source, resolved within the manifest's fs.FS
Pool int `json:"pool"` // pooled interpreter count (default 8)
Prefix string `json:"prefix"`
Routes []Route `json:"routes"`
Env map[string]string `json:"env,omitempty"`
}
Manifest describes a polyglot service to mount: which language, where the source is, how many pooled interpreters, and the route table. Drop a service.manifest.json next to a .py/.ts/.wasm file and the loader does the rest — that is the "easy to port any Go/Python/Rust/JS service in" UX.
{
"name": "greeter",
"lang": "python",
"source": "greet.py",
"pool": 8,
"prefix": "/v1/greeter",
"routes": [{ "method": "POST", "path": "/greet", "func": "greet" }]
}
func LoadManifest ¶
LoadManifest reads and validates a manifest file from disk.
func ParseManifest ¶
ParseManifest validates raw manifest bytes. desc names the source for errors.
type Module ¶
type Module interface {
// Invoke calls fn(payload) where payload is JSON and the return is JSON.
Invoke(ctx context.Context, fn string, payload []byte) ([]byte, error)
// Funcs lists the callable exports discovered in the module.
Funcs() []string
// Close releases the module's runtime resources.
Close() error
}
Module is one loaded unit of code in some language. It is NOT required to be goroutine-safe; Pool provides concurrency. Invoke runs the named exported function with a JSON payload and returns a JSON result.
type NativeModule ¶
type NativeModule struct {
// contains filtered or unexported fields
}
NativeModule mounts plain Go functions through the same Module interface as the interpreted engines — zero overhead, full type safety. This is how Go services join the unified binary: register their handlers and they sit beside the Python/JS/Rust ones behind the identical routing surface.
NativeModule is goroutine-safe (the underlying Go funcs are expected to be), so it needs no Pool — though you may still pool it uniformly if you like.
func NewNativeModule ¶
func NewNativeModule(funcs map[string]HandlerFunc) *NativeModule
NewNativeModule wraps a set of named Go handlers as a Module.
func (*NativeModule) Close ¶
func (m *NativeModule) Close() error
func (*NativeModule) Funcs ¶
func (m *NativeModule) Funcs() []string
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool keeps a fixed set of Modules for one service and lends one to each Invoke, returning it when the call completes. Because each Module is used by exactly one goroutine at a time, single-threaded engines (goja, gpython, wasm instances) safely serve concurrent requests: N modules => N parallel in-flight calls. This is the unit that makes "every route a goroutine" real.
func LoadPool ¶
func LoadPool(ctx context.Context, lang string, src []byte, size int, opt LoadOptions) (*Pool, error)
LoadPool compiles src once per pool entry with the given engine, so each entry is an independent interpreter/instance.
type Route ¶
type Route struct {
Method string `json:"method"` // GET, POST, ...
Path string `json:"path"` // e.g. /greet (combined with the service Prefix)
Func string `json:"func"` // exported function name to invoke
}
Route binds an HTTP method+path to an exported function in the module.
type Service ¶
type Service struct {
Name string
Prefix string // e.g. /v1/greeter ; routes are mounted under it
Pool *Pool
Routes []Route
}
Service is a mounted polyglot module: a Pool of interpreters plus the route table that maps HTTP requests onto its functions. Handler() returns a plain net/http.Handler so any host can serve it — in hanzoai/cloud it is mounted onto the zip/gofiber app via zip.AdaptNetHTTP, keeping goa host-agnostic.
func LoadDir ¶
LoadDir builds every service whose manifest lives in dir on disk. Convenience over LoadFS(ctx, os.DirFS(dir)).
func LoadFS ¶
LoadFS discovers every *.manifest.json at the root of fsys and builds the services. fsys is the single source abstraction: pass an embed.FS to bake services into a static binary, or os.DirFS(dir) to load them from disk. Successful services are returned even if some fail; failures are joined.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
examples
|
|
|
serve
command
Command serve loads every *.manifest.json in the examples directory and serves the polyglot services on :8080 — Python (gpython) and TypeScript (goja) handlers running as goroutines inside one static Go binary.
|
Command serve loads every *.manifest.json in the examples directory and serves the polyglot services on :8080 — Python (gpython) and TypeScript (goja) handlers running as goroutines inside one static Go binary. |