Help us shape the future of Go! Take the 2026 Go Developer Survey

xgoboost

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 21, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

README

xgoboost

Gradient-boosted decision trees with monotone constraints, in pure Go.

  • Second-order boosting: Newton leaf values -G/(H+λ) and the usual second-order split gain, under a pluggable Loss.
  • Built-in losses: squared error, Huber, logistic (binary), quantile.
  • Per-feature monotone constraints (+1, -1, 0), guaranteed across the whole ensemble, not just per split.
  • Histogram split finding (quantile binning, per-node gradient/hessian histograms, histogram subtraction) plus an exact greedy mode for small data.
  • Missing values (NaN) with a learned default direction at every split.
  • L2 λ, min child weight, min split gain, depth, learning rate, row and column subsampling, early stopping on a validation set.
  • Deterministic for a given seed, whatever the number of workers.
  • JSON model serialisation that round-trips bit-for-bit.
  • Standard library only. Go 1.22+.
go get github.com/natemsz/xgoboost

Usage

package main

import (
	"fmt"
	"log"
	"math"
	"os"

	xgoboost "github.com/natemsz/xgoboost"
)

func main() {
	// X: one row per sample, NaN where a value is unknown.
	X := [][]float64{
		{0.9, 0.1, 0.3},
		{0.2, math.NaN(), 0.8},
		// ...
	}
	y := []float64{9.1, 1.7 /* ... */}
	validX, validY := X, y // use genuinely held-out rows

	p := xgoboost.DefaultParams()
	p.Monotone = []int{+1, -1, 0} // up in feature 0, down in 1, free in 2
	p.Rounds = 500
	p.Subsample, p.ColSample, p.Seed = 0.8, 0.8, 42

	// Optional: early stopping on held-out data.
	p.Validation = &xgoboost.ValidationSet{X: validX, Y: validY}
	p.EarlyStoppingRounds = 20

	m, err := xgoboost.Train(X, y, nil /* unit weights */, p)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(m.Predict([]float64{0.5, 0.2, 0.4}))
	fmt.Println(m.PredictBatch(X))
	fmt.Println(m.BestIteration, m.FeatureImportance(xgoboost.TotalGain))

	// Save and load (error handling elided).
	f, _ := os.Create("model.json")
	_ = m.Save(f)
	f.Close()
	f, _ = os.Open("model.json")
	defer f.Close()
	m2, _ := xgoboost.Load(f)
	fmt.Println(m2.Predict([]float64{0.5, 0.2, 0.4})) // identical
}

Classification uses p.Loss = xgoboost.Logistic{} with labels in [0, 1]; Predict then returns a probability and PredictRaw the log-odds. Quantile regression uses xgoboost.Quantile{Alpha: 0.9}, and robust regression uses xgoboost.Huber{Delta: 1}.

Parameters
Field Default Meaning
Loss SquaredError{} training objective
Rounds 100 number of trees (upper bound with early stopping)
LearningRate 0.1 shrinkage applied to every tree
MaxDepth 6 maximum tree depth
Lambda 1 * L2 penalty on leaf values
MinChildWeight 1 * least hessian sum per child
MinSplitGain 0 least loss reduction per split (XGBoost's γ)
Monotone nil per-feature +1 / -1 / 0
Subsample 1 share of rows per tree, without replacement
ColSample 1 share of features per tree
Seed 0 sampling seed
Method Auto Exact up to 2000 rows, else Hist
MaxBins 256 histogram bins per feature (2–65535)
Validation, EarlyStoppingRounds none early stopping
Workers GOMAXPROCS goroutines for histogram building and split search

* In DefaultParams(). In a zero Params, the fields marked "0 means default" in the godoc take the defaults above. Lambda, MinChildWeight and MinSplitGain are used as given, so a zero Params is valid but unregularised.

Custom losses

Implement Loss (Name, InitScore, GradHess, Transform, Metric). Optionally also implement:

  • LossValidator to check labels and parameters before training;
  • ParamLoss to persist parameters;
  • LeafRenewer to replace Newton leaf values after a tree is grown (the quantile loss does this).

Call RegisterLoss(name, decoder) before loading a saved model that uses the loss. Transform must be non-decreasing so that constraints carry over from raw scores to predictions.

How the monotone constraint is enforced

The construction is the one XGBoost uses. Each node carries an interval [lo, hi], and every leaf value in its subtree is clamped into it. The root's interval is (-∞, +∞).

When the split finder considers a split on feature j with constraint c[j], it computes the children's Newton values vL, vR and clamps them to the node's interval. Then:

  1. Reject the split if it runs the wrong way: vL > vR when c[j] = +1, or vL < vR when c[j] = -1. Gain is computed with the clamped values, ½(score(GL,HL,vL) + score(GR,HR,vR) − score(G,H,v)), where score(G,H,w) = −(2Gw + (H+λ)w²). This is the loss reduction the split actually achieves, not the one it would achieve unclamped.
  2. If the split is kept, bound the children at the midpoint m = (vL + vR)/2. For +1, the left subtree gets [lo, m] and the right subtree gets [m, hi]. For −1, the bounds are mirrored. For an unconstrained feature both children inherit [lo, hi].

Why one tree is monotone. Take two inputs x ≤ x' that differ only in a +1 feature j, both non-missing. Walk them down the tree together:

  • At a split on any other feature, they go the same way.
  • At a split on j, either they go the same way, or x goes left and x' goes right.

The first time they part, x lands in the left subtree, whose leaves all lie in [lo, m]. x' lands in the right subtree, whose leaves all lie in [m, hi]. So f(x) ≤ m ≤ f(x'). If they never part, they reach the same leaf. Either way f(x) ≤ f(x').

The bounds are inherited and only ever tighten. A later split on some other feature, deep inside the left subtree, therefore cannot produce a leaf above m. This is the case a split-local check alone gets wrong (see TestMonotoneInteraction).

Why the ensemble is monotone. Each tree is monotone, and so is a sum of monotone functions. Scaling by a positive learning rate preserves order. Floating-point addition is monotone too: a ≤ b implies fl(a + c) ≤ fl(b + c). The tests therefore check the property exactly, with zero tolerance: they sweep each constrained feature from −∞ to +∞ while holding the other features fixed (some of them NaN). Loss.Transform is non-decreasing, so Predict inherits the guarantee from PredictRaw.

Leaf renewal. Losses such as quantile replace a leaf's Newton value after the tree is grown. The new value is clamped to that leaf's envelope, so the argument above still holds.

Missing values. NaN is not ordered, so it has no place on the constrained axis. A row whose constrained feature is missing follows the split's learned default direction. The guarantee covers every pair of non-missing values.

Missing values

When it evaluates a split, the finder tries sending the node's NaN rows left and then right, and keeps whichever gives the larger gain. That choice is stored as the node's DefaultLeft. If a feature had no missing values at a node during training, NaN at prediction time follows the child with more hessian cover. A row shorter than NumFeatures is read as NaN for the missing positions.

Histogram vs exact

Hist bins each feature once into at most MaxBins weighted-quantile bins. A feature with at most MaxBins distinct values gets one bin per value, with cuts at the midpoints between them. Each node accumulates per-bin gradient and hessian sums. Only the smaller child's histogram is built from its rows; the larger child's is the parent's minus the smaller one.

Exact sorts each feature once up front. Every node keeps its rows in each feature's sorted order by stable partitioning, so no node sorts again.

When every feature has at most MaxBins distinct values, both methods evaluate the same candidate splits and grow the same trees (TestHistMatchesExact).

Determinism

Row and column sampling use a PCG generator seeded from Params.Seed. Parallel work writes results by feature index and reduces them in a fixed order, with ties going to the lower feature index. The same data, parameters and seed therefore produce a byte-identical model for any Workers value (TestDeterministic).

Results can differ across CPU architectures where the compiler fuses multiply-adds.

Benchmarks

Measured with go test -bench . -benchtime 3x on a 12-thread Windows desktop with Go 1.27, 50 rounds at depth 6:

Benchmark Rows × features Time per fit
TrainHist100k 100 000 × 20 ~1.0 s
TrainHist20k 20 000 × 20 ~0.36 s
TrainHist20kMono (2 constrained) 20 000 × 20 ~0.36 s
TrainExact2k 2 000 × 10 ~0.13 s
PredictBatch (100 trees) 20 000 × 20 ~5 µs / row

The library is not yet tuned to the level of XGBoost or LightGBM. Both are far faster on large data.

Not (yet) supported

  • Multiclass, ranking and Poisson/Tweedie losses (all possible through the Loss interface plus a wrapper, but none is built in).
  • Categorical splits, interaction constraints, max_delta_step, and per-level or per-node column sampling.
  • Sparse input, external memory, and GPU.

License

Apache-2.0. See LICENSE.

Documentation

Overview

Package xgoboost trains gradient-boosted decision trees with per-feature monotone constraints, in pure Go with no dependencies outside the standard library.

Training is second order: every round computes each row's gradient and hessian under a pluggable Loss, grows one regression tree whose leaves take the Newton value -G/(H+λ), and adds it, scaled by the learning rate, to the ensemble. Splits are scored by the usual second-order gain and found either by exact greedy enumeration or on quantile-binned feature histograms with histogram subtraction.

Monotone constraints use the XGBoost envelope construction: a split on a constrained feature is kept only when its children's values are ordered the right way, and each child's subtree is then bounded at the midpoint between them. Every tree, and therefore the ensemble, is monotone in each constrained feature; see the README for the proof.

Missing values are NaN. Each split learns a default direction for them from the training data.

A minimal regression: 

p := xgoboost.DefaultParams()
p.Monotone = []int{+1, 0, -1}
m, err := xgoboost.Train(X, y, nil, p)
if err != nil { ... }
yhat := m.Predict(x)
Example

Price rises with quality and falls with wear, whatever the noise says locally; the constraints make the model promise the same. 

package main

import (
	"fmt"
	"math"
	"math/rand/v2"

	xgoboost "github.com/natemsz/xgoboost"
)

func main() {
	r := rand.New(rand.NewPCG(1, 2))
	var X [][]float64
	var y []float64
	for i := 0; i < 2000; i++ {
		quality, wear, colour := r.Float64(), r.Float64(), r.Float64()
		if i%10 == 0 {
			wear = math.NaN() // unknown
		}
		price := 10*quality - 4*wear + math.Sin(6*colour) + r.NormFloat64()
		if math.IsNaN(wear) {
			price = 10*quality - 2 + math.Sin(6*colour) + r.NormFloat64()
		}
		X = append(X, []float64{quality, wear, colour})
		y = append(y, price)
	}

	p := xgoboost.DefaultParams()
	p.Monotone = []int{+1, -1, 0} // quality up, wear down, colour free
	p.Rounds = 200
	m, err := xgoboost.Train(X, y, nil, p)
	if err != nil {
		panic(err)
	}

	better := m.Predict([]float64{0.9, 0.2, 0.5})
	worse := m.Predict([]float64{0.8, 0.2, 0.5})
	fmt.Println("better quality never prices lower:", better >= worse)
	fmt.Printf("unknown wear: %.0f\n", m.Predict([]float64{0.5, math.NaN(), 0.5}))
}

Output:
better quality never prices lower: true
unknown wear: 3

Index

Examples

Constants

View Source 
const AutoExactMaxRows = 2000

AutoExactMaxRows is the largest training set Auto fits with Exact.

Variables

This section is empty.

Functions

func RegisterLoss 

func RegisterLoss(name string, dec LossDecoder)

RegisterLoss makes a custom loss loadable from serialised models. It panics if name is empty, dec is nil, or name is already registered.

Types

type Huber 

type Huber struct {
	Delta float64
}

Huber is squared error within Delta of the label and absolute error beyond it, so outliers pull the fit with bounded force. Its gradient is the residual clipped to ±Delta and its hessian is taken as 1 (the true curvature, 0 outside the band, would make Newton steps unbounded). The metric is the mean Huber loss.

func (Huber) GradHess 

func (l Huber) GradHess(y, raw float64) (g, h float64)

func (Huber) InitScore 

func (l Huber) InitScore(y, w []float64) float64

func (Huber) LossParams 

func (l Huber) LossParams() map[string]float64

func (Huber) Metric 

func (l Huber) Metric(y, raw, w []float64) float64

func (Huber) Name 

func (l Huber) Name() string

func (Huber) Transform 

func (l Huber) Transform(raw float64) float64

func (Huber) Validate 

func (l Huber) Validate([]float64) error

type ImportanceType 

type ImportanceType int

ImportanceType selects what FeatureImportance sums. 

const (
	// TotalGain sums the loss reduction of every split on a feature.
	TotalGain ImportanceType = iota
	// SplitCount counts the splits on a feature.
	SplitCount
)

type LeafRenewer 

type LeafRenewer interface {
	RenewLeaf(residual, w []float64) float64
}

LeafRenewer is implemented by losses whose Newton step is a poor leaf value (the quantile loss has zero curvature). After a tree is grown, each leaf's value is replaced by RenewLeaf over the residuals y - raw of the training rows in it, then clamped to the leaf's monotone envelope, so constraints still hold.

type Logistic 

type Logistic struct{}

Logistic is binary cross-entropy on labels in [0, 1]; raw predictions are log-odds and Transform is the logistic function. The metric is the mean log loss.

func (Logistic) GradHess 

func (Logistic) GradHess(y, raw float64) (g, h float64)

func (Logistic) InitScore 

func (Logistic) InitScore(y, w []float64) float64

func (Logistic) Metric 

func (Logistic) Metric(y, raw, w []float64) float64

func (Logistic) Name 

func (Logistic) Name() string

func (Logistic) Transform 

func (Logistic) Transform(raw float64) float64

func (Logistic) Validate 

func (Logistic) Validate(y []float64) error

type Loss 

type Loss interface {
	// Name identifies the loss in serialised models. A custom loss must
	// be registered under this name with RegisterLoss before a model
	// using it can be loaded.
	Name() string
	// InitScore is the constant raw prediction boosting starts from.
	InitScore(y, w []float64) float64
	// GradHess returns the first and second derivative of one row's
	// loss with respect to its raw prediction. hess must be >= 0.
	GradHess(y, raw float64) (grad, hess float64)
	// Transform maps a raw prediction to the output scale. It must be
	// non-decreasing, so that monotone constraints carry over to
	// Model.Predict.
	Transform(raw float64) float64
	// Metric scores raw predictions against labels; lower is better.
	// It drives early stopping.
	Metric(y, raw, w []float64) float64
}

Loss is a training objective. Boosting works on raw predictions (the margin); Transform maps a raw prediction to the output scale.

Train always passes non-nil weight slices. Implementations used elsewhere should treat a nil w as unit weights.

type LossDecoder 

type LossDecoder func(params map[string]float64) (Loss, error)

LossDecoder rebuilds a loss from its serialised parameters.

type LossValidator 

type LossValidator interface {
	Validate(y []float64) error
}

LossValidator is implemented by losses that restrict their labels or their own parameters. Train calls Validate before fitting.

type Model 

type Model struct {
	NumFeatures  int
	BaseScore    float64 // the raw prediction before any tree
	LearningRate float64 // informational: leaf values are pre-scaled
	Loss         Loss
	Monotone     []int // nil when unconstrained
	Trees        []Tree
	// BestIteration is the 0-based round with the lowest validation
	// metric, and BestScore that metric. BestIteration is -1 when
	// training had no validation set.
	BestIteration int
	BestScore     float64
}

Model is a trained ensemble. It is safe for concurrent use by multiple goroutines once trained or loaded.

func Load 

func Load(r io.Reader) (*Model, error)

Load reads a model written by Save.

func Train 

func Train(X [][]float64, y, w []float64, p Params) (*Model, error)

Train fits a boosted ensemble to rows X (n rows of equal length; NaN marks a missing value, ±Inf is allowed), labels y and row weights w (nil means unit weights; weights must be finite and >= 0 with a positive sum).

func (*Model) FeatureImportance 

func (m *Model) FeatureImportance(kind ImportanceType) []float64

FeatureImportance returns one unnormalised score per feature.

func (*Model) MarshalJSON 

func (m *Model) MarshalJSON() ([]byte, error)

MarshalJSON encodes the model. Floats round-trip exactly, so a reloaded model predicts bit-for-bit the same.

func (*Model) Predict 

func (m *Model) Predict(x []float64) float64

Predict returns the prediction for x on the output scale: the raw prediction passed through Loss.Transform (a probability for Logistic).

func (*Model) PredictBatch 

func (m *Model) PredictBatch(X [][]float64) []float64

PredictBatch returns Predict for every row of X.

func (*Model) PredictRaw 

func (m *Model) PredictRaw(x []float64) float64

PredictRaw returns the raw prediction (the margin) for x: the base score plus every tree's leaf value. NaN, or a feature index beyond len(x), counts as missing.

func (*Model) Save 

func (m *Model) Save(w io.Writer) error

Save writes the model as JSON.

func (*Model) UnmarshalJSON 

func (m *Model) UnmarshalJSON(b []byte) error

UnmarshalJSON decodes and validates a model. Losses other than the built-ins must be registered with RegisterLoss first.

type Node 

type Node struct {
	Leaf bool
	// Value is a leaf's contribution to the raw prediction, already
	// scaled by the learning rate.
	Value       float64
	Feature     int
	Threshold   float64
	DefaultLeft bool
	Left, Right int
	// Gain is the split's loss reduction (training only; used for
	// feature importance).
	Gain float64
	// Cover is the hessian sum of the training rows that reached the
	// node.
	Cover float64
}

Node is one node of a Tree. A split node sends x left when x[Feature] <= Threshold, right when greater, and by DefaultLeft when x[Feature] is NaN (or x is shorter than Feature+1). Children always have larger indices than their parent; node 0 is the root.

type ParamLoss 

type ParamLoss interface {
	LossParams() map[string]float64
}

ParamLoss is implemented by losses with parameters that must survive serialisation. The map is passed back to the loss's LossDecoder.

type Params 

type Params struct {
	// Loss is the objective. nil means SquaredError.
	Loss Loss
	// Rounds is the number of trees (at most, with early stopping).
	// 0 means 100.
	Rounds int
	// LearningRate scales every tree. 0 means 0.1.
	LearningRate float64
	// MaxDepth limits tree depth; a depth-d tree has at most 2^d
	// leaves. 0 means 6.
	MaxDepth int
	// Lambda is the L2 penalty on leaf values: leaves are -G/(H+λ).
	Lambda float64
	// MinChildWeight is the least hessian sum a child may hold. For
	// squared error with unit weights it is a row count.
	MinChildWeight float64
	// MinSplitGain is the least loss reduction a split must achieve
	// (XGBoost's gamma).
	MinSplitGain float64
	// Monotone holds one constraint per feature: +1 non-decreasing,
	// -1 non-increasing, 0 free. nil leaves every feature free.
	Monotone []int
	// Subsample is the share of rows drawn, without replacement, for
	// each tree. 0 means 1.
	Subsample float64
	// ColSample is the share of features drawn for each tree. 0 means 1.
	ColSample float64
	// Seed drives row and column sampling. Training is deterministic
	// for a given Seed, whatever Workers is.
	Seed uint64
	// Method picks the split finder. The zero value is Auto.
	Method TreeMethod
	// MaxBins bounds the histogram bins per feature, between 2 and
	// 65535. 0 means 256.
	MaxBins int
	// Validation, if set, is scored with Loss.Metric after each round;
	// the model records the best round.
	Validation *ValidationSet
	// EarlyStoppingRounds > 0 stops training once the validation
	// metric has not improved for that many rounds, and keeps only the
	// trees up to the best round. It requires Validation.
	EarlyStoppingRounds int
	// Workers bounds the goroutines used to build histograms and
	// search splits. 0 means GOMAXPROCS.
	Workers int
}

Params configures Train. Fields marked "0 means default" take the default when left zero; the others are used as given, so the zero Params is valid but unregularised. DefaultParams returns the recommended starting point.

func DefaultParams 

func DefaultParams() Params

DefaultParams returns sensible defaults: squared error, 100 rounds of depth-6 trees at learning rate 0.1, λ = 1, min child weight 1.

type Quantile 

type Quantile struct {
	Alpha float64
}

Quantile is the pinball loss for the Alpha quantile of y given x. Its curvature is zero, so trees are grown on gradients with unit hessians and each leaf is then renewed to the Alpha quantile of its rows' residuals. The metric is the mean pinball loss.

func (Quantile) GradHess 

func (l Quantile) GradHess(y, raw float64) (g, h float64)

func (Quantile) InitScore 

func (l Quantile) InitScore(y, w []float64) float64

func (Quantile) LossParams 

func (l Quantile) LossParams() map[string]float64

func (Quantile) Metric 

func (l Quantile) Metric(y, raw, w []float64) float64

func (Quantile) Name 

func (l Quantile) Name() string

func (Quantile) RenewLeaf 

func (l Quantile) RenewLeaf(residual, w []float64) float64

func (Quantile) Transform 

func (l Quantile) Transform(raw float64) float64

func (Quantile) Validate 

func (l Quantile) Validate([]float64) error

type SquaredError 

type SquaredError struct{}

SquaredError is ½(raw - y)², the default. Its metric is RMSE.

func (SquaredError) GradHess 

func (SquaredError) GradHess(y, raw float64) (g, h float64)

func (SquaredError) InitScore 

func (SquaredError) InitScore(y, w []float64) float64

func (SquaredError) Metric 

func (SquaredError) Metric(y, raw, w []float64) float64

func (SquaredError) Name 

func (SquaredError) Name() string

func (SquaredError) Transform 

func (SquaredError) Transform(raw float64) float64

type Tree 

type Tree struct {
	Nodes []Node
}

Tree is one regression tree, stored as a flat node slice.

type TreeMethod 

type TreeMethod int

TreeMethod selects how split candidates are found. 

const (
	// Auto uses Exact for training sets of at most AutoExactMaxRows
	// rows and Hist above that.
	Auto TreeMethod = iota
	// Hist bins each feature into at most MaxBins quantile bins once,
	// then finds splits on per-node gradient/hessian histograms. Only
	// the smaller child's histogram is built; its sibling's is the
	// parent's minus it.
	Hist
	// Exact enumerates every threshold between distinct values present
	// at a node. It is O(n log n) per node and feature.
	Exact
)

type ValidationSet 

type ValidationSet struct {
	X [][]float64
	Y []float64
	W []float64 // nil means unit weights
}

ValidationSet is held-out data scored after every round.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.