README
¶
WALDO
WALDO is a command-line tool for building auditable AI training datasets and models. It connects a Git-governed corpus index, content-addressed Parquet objects, and model artifacts with verifiable bills of materials (BOMs).
WALDO is under active development. It is being opened early so users and contributors can help shape the project through frequent, incremental releases. Expect interfaces and formats outside the documented compatibility contract to evolve.
What works today
- Inspect, verify, audit, ingest, update, and export indexed corpora.
- Read and publish canonical Parquet objects through local or S3 lookaside storage.
- Create corpus, training-run, model, and release provenance records.
- Forecast and train models through MLX, PyTorch, or single- and multi-host TorchTitan when the required runtime is installed.
- Export native WALDO, Hugging Face, MLX, GGUF, and Ollama packages.
WALDO does not prove that a license assertion is legally correct, that model output is safe, or that a generated disclosure alone establishes regulatory compliance. It records attributable facts and verifies artifact identity.
Build
WALDO requires Go 1.25 or newer.
git clone https://github.com/openwaldo/waldo.git
cd waldo
go build -o ./waldo ./cmd/waldo
sudo install -m 0755 ./waldo /usr/local/bin/waldo
waldo --help
First steps
With no configured index, read-only index commands use a managed checkout at
~/.waldo/index:
waldo status
waldo index list
waldo index summary
waldo index verify --offline
Contributing data requires a separate writable index checkout:
git clone https://github.com/openwaldo/waldo-index.git
waldo config set index /path/to/waldo-index
waldo config set lookaside file:///tmp/waldo-lookaside
waldo index ingest --help
Use waldo <command> --help as the authoritative command reference.
Documentation
Start with the documentation index. In particular:
Early feedback, testing, documentation improvements, and focused code contributions are welcome. See Contributing.
Development
./testing/unit.sh
./testing/vet.sh
The full end-to-end suite is described in docs/TESTING.md.
License
WALDO is licensed under the Apache License 2.0. See NOTICE for attribution notices.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
waldo
command
|
|
|
composes
|
|
|
internal
|
|
|
calibration
Package calibration builds bounded, reproducible quantization-calibration inputs from verified WALDO corpus selections.
|
Package calibration builds bounded, reproducible quantization-calibration inputs from verified WALDO corpus selections. |
|
canon
Package canon contains the canonical JSON primitives used by WALDO formats.
|
Package canon contains the canonical JSON primitives used by WALDO formats. |
|
config
Package config owns machine-local preferences.
|
Package config owns machine-local preferences. |
|
corpus
Package corpus resolves indexed manifests into immutable, verified selections consumed by export and model workflows.
|
Package corpus resolves indexed manifests into immutable, verified selections consumed by export and model workflows. |
|
disclosure
Package disclosure maps verified model provenance into versioned public disclosure inputs.
|
Package disclosure maps verified model provenance into versioned public disclosure inputs. |
|
evaluation
Package evaluation owns immutable evaluation definitions, corpus BOM pins, contamination evidence, metric results, and deterministic promotion gates.
|
Package evaluation owns immutable evaluation definitions, corpus BOM pins, contamination evidence, metric results, and deterministic promotion gates. |
|
git
Package git owns Git transport and checkout inspection for WALDO.
|
Package git owns Git transport and checkout inspection for WALDO. |
|
host
Package host reports local machine resources that are independent of any training framework.
|
Package host reports local machine resources that are independent of any training framework. |
|
index
Package index reads and verifies WALDO's Git metadata tree.
|
Package index reads and verifies WALDO's Git metadata tree. |
|
inference
Package inference owns ephemeral model generation sessions.
|
Package inference owns ephemeral model generation sessions. |
|
ingest
Package ingest probes acquired artifacts and builds immutable execution plans for the corpus ingestion workflow.
|
Package ingest probes acquired artifacts and builds immutable execution plans for the corpus ingestion workflow. |
|
lookaside
Package lookaside provides verified access to content-addressed objects.
|
Package lookaside provides verified access to content-addressed objects. |
|
mlxruntime
Package mlxruntime owns the Python model definition shared by WALDO's MLX training and inference workers.
|
Package mlxruntime owns the Python model definition shared by WALDO's MLX training and inference workers. |
|
model
Package model owns model compose files, immutable architecture identity, build plans, model/run BOMs, and durable lifecycle state.
|
Package model owns model compose files, immutable architecture identity, build plans, model/run BOMs, and durable lifecycle state. |
|
modelexport
Package modelexport converts one verified WALDO model run into separately executable release formats without changing the managed model.
|
Package modelexport converts one verified WALDO model run into separately executable release formats without changing the managed model. |
|
modelquant
Package modelquant adapts the upstream llama.cpp quantization tools without giving them ownership of WALDO model, index, or provenance state.
|
Package modelquant adapts the upstream llama.cpp quantization tools without giving them ownership of WALDO model, index, or provenance state. |
|
modelweights
Package modelweights owns the portable tensor-name contract shared by model acquisition, training artifacts, and release conversion.
|
Package modelweights owns the portable tensor-name contract shared by model acquisition, training artifacts, and release conversion. |
|
provenance
Package provenance owns durable records that connect OpenWALDO BOMs, training observations, and model artifacts.
|
Package provenance owns durable records that connect OpenWALDO BOMs, training observations, and model artifacts. |
|
pytorchruntime
Package pytorchruntime owns the Python model definition shared by WALDO's PyTorch/TorchTitan training and PyTorch inference workers.
|
Package pytorchruntime owns the Python model definition shared by WALDO's PyTorch/TorchTitan training and PyTorch inference workers. |
|
record
Package record defines the OpenWALDO schema-1 interchange record.
|
Package record defines the OpenWALDO schema-1 interchange record. |
|
shard
Package shard reads WALDO's native shard containers.
|
Package shard reads WALDO's native shard containers. |
|
signing
Package signing owns detached signatures for exported release documents.
|
Package signing owns detached signatures for exported release documents. |
|
status
Package status reports whether the current machine is ready for WALDO data and model workflows without changing configuration or durable state.
|
Package status reports whether the current machine is ready for WALDO data and model workflows without changing configuration or durable state. |
|
tokenizer
Package tokenizer provides the reference token estimates recorded in index manifests.
|
Package tokenizer provides the reference token estimates recorded in index manifests. |
|
training
Package training is the narrow adapter boundary between WALDO's durable model lifecycle and an execution framework.
|
Package training is the narrow adapter boundary between WALDO's durable model lifecycle and an execution framework. |
|
testing
|
|
|
e2e
command
|
|
|
e2e/validate_conversation
command
|
Click to show internal directories.
Click to hide internal directories.