builders

README

github.com/Highload-fun/builders

Go library that provides sandboxed compilation for multiple programming languages. Each builder downloads compiler toolchains on demand (or uses system-installed compilers), then compiles user source code inside a Linux sandbox using libsandbox with cgroups, mount namespaces, and network isolation.

go get github.com/Highload-fun/builders

Supported Languages

Package	Builder ID	Version source	Default version
cpp/	cpp	System-installed g++/clang++	g++13.3.0
csharp/	csharp	dotnet releases-index.json (>=8.x)	8.0.0
golang/	go	go.dev release page	go1.13.8
rust/	rust	GitHub RELEASES.md	rust-1.47.0
zig/	zig	ziglang.org JSON index	0.13.0

How It Works

Each builder implements the Builder interface:

type Builder interface {
	GetVersions(ctx context.Context) ([]Version, error)
	Build(ctx context.Context, sb *sandbox.Sandbox, version string, flags []string) error
}

When builders.Build is called, it:

1. Creates a temporary directory for the sandbox root
2. Configures the sandbox with a 5 GB memory limit and network disabled
3. Mounts the source directory at /src and output directory at /out
4. Delegates to the language-specific builder, which downloads/locates the compiler and runs it inside the sandbox
5. The compiled binary is written to /out/main

Compiler toolchains are downloaded once and cached in /tmp.

Adding a New Language

1. Create a subpackage (e.g., mylang/)
2. Implement the Builder interface
3. Register via init():

   func init() {
       builders.Register("mylang", &MyLang{})
   }
   

4. Add test/src/ with a minimal source file that compiles to a binary
5. Write a test file (mylang_test.go) using the shared test harness:

   package mylang_test
   
   import (
       "embed"
       "io/fs"
       "testing"
   
       "github.com/stretchr/testify/assert"
   
       "github.com/Highload-fun/builders"
       "github.com/Highload-fun/builders/mylang"
   )
   
   //go:embed test/*
   var testSrcFs embed.FS
   
   func TestMyLang(t *testing.T) {
       subFs, err := fs.Sub(testSrcFs, "test/src")
       if !assert.NoError(t, err) {
           t.FailNow()
       }
   
       // Validates GetVersions and builds with the default version
       builders.Check(t, mylang.BuilderId, subFs)
   
       // Optionally test a specific version
       builders.CheckBuilding(t, "with v1.2.3", mylang.BuilderId, "v1.2.3", []string{}, subFs)
   }
   

   builders.Check verifies that GetVersions returns a non-empty list and that the default version compiles the embedded source and produces a runnable binary. builders.CheckBuilding does the same for a specific version and flags.

Requirements

• Linux with cgroup v2 support
• gcc-13 (used as linker by Go, Rust, and C++ builders)
• Root or appropriate sandbox permissions
• Network access (for downloading compiler toolchains on first use)

Testing

# Run all tests
go test ./...

# Run tests for a specific builder
go test ./golang/ -run TestGolang
go test ./rust/ -run TestRust
go test ./zig/ -run TestZig
go test ./csharp/ -run TestCSharp
go test ./cpp/ -run TestCpp

Tests download real compiler toolchains and execute sandboxed builds, so they require network access and can be slow on first run. Toolchains are cached in /tmp for subsequent runs.

Documentation

Overview

Package builders provides sandboxed compilation for multiple programming languages.

Each language is implemented as a separate subpackage that registers itself via Register during init(). To use a builder, import its package for side effects:

import _ "github.com/Highload-fun/builders/golang"

Then call GetVersions and Build with the builder's ID.

Index

• Constants
• func Build(ctx context.Context, builderId, version string, flags []string, ...) error
• func Check(t *testing.T, builderId string, sources fs.FS)
• func CheckBuilding(t *testing.T, description, builderId, version string, flags []string, ...)
• func DownloadAndExtractArchive(ctx context.Context, archiveUrl, dstDir string) error
• func GetBuildersIds() []string
• func HttpGet(ctx context.Context, url string) (*http.Response, error)
• func Register(id string, builder Builder)
• type Builder
• type FakeCloser
  • func (FakeCloser) Close() error
• type Version
  • func GetVersions(ctx context.Context, builderId string) ([]Version, error)

Constants

const (
	// CompilersHostDir is the host directory where downloaded compiler toolchains are cached.
	CompilersHostDir = "/tmp"
	// SourceDir is the path where user source code is mounted inside the sandbox.
	SourceDir = "/src"
	// OutDir is the path where the compiled binary is written inside the sandbox.
	OutDir = "/out"
)

Functions

func Build

func Build(ctx context.Context, builderId, version string, flags []string, srcDir, outDir string) error

Build compiles the source code in srcDir using the specified builder and version, writing the resulting binary to outDir. It creates an isolated sandbox with a 5 GB memory limit and network disabled. If version is empty, the builder's default is used.

func Check

func Check(t *testing.T, builderId string, sources fs.FS)

Check is a test helper that validates a builder's version listing and default-version build. It first asserts that GetVersions returns a non-empty list of valid versions, then delegates to CheckBuilding with an empty version to exercise the default.

func CheckBuilding

func CheckBuilding(t *testing.T, description, builderId, version string, flags []string, sources fs.FS)

CheckBuilding is a test helper that compiles sources with the given builder and version, then verifies the resulting binary can execute inside a sandbox.

func DownloadAndExtractArchive

func DownloadAndExtractArchive(ctx context.Context, archiveUrl, dstDir string) error

DownloadAndExtractArchive fetches a tar archive from archiveUrl and extracts it into dstDir. Supported formats are .tar.gz, .tar.bz2, and .tar.xz.

func GetBuildersIds

func GetBuildersIds() []string

GetBuildersIds returns the sorted IDs of all registered builders.

func HttpGet

func HttpGet(ctx context.Context, url string) (*http.Response, error)

HttpGet performs a context-aware HTTP GET request.

func Register

func Register(id string, builder Builder)

Register adds a builder under the given id. It panics if the id is already taken. Language subpackages call this from their init() functions.

Types

type Builder

type Builder interface {
	// GetVersions returns the available compiler versions for this language.
	GetVersions(ctx context.Context) ([]Version, error)
	// Build compiles source code inside the given sandbox, producing a "main" binary in [OutDir].
	Build(ctx context.Context, sb *sandbox.Sandbox, version string, flags []string) error
}

Builder is the interface that each language compiler must implement.

type FakeCloser

type FakeCloser struct {
	io.Reader
}

FakeCloser wraps an io.Reader with a no-op Close method to satisfy io.ReadCloser.

func (FakeCloser) Close

func (FakeCloser) Close() error

Close is a no-op that always returns nil.

type Version

type Version struct {
	Id          string
	ReleaseDate time.Time
}

Version represents a single compiler release.

func GetVersions

func GetVersions(ctx context.Context, builderId string) ([]Version, error)

GetVersions returns the available compiler versions for the given builder, sorted by release date (newest first).