protoc

README

go-protoc-wasi

A Go module that embeds the Protocol Buffers compiler (protoc) as a WASI WebAssembly reactor.

Related Projects

• aperturerobotics/protobuf - Fork with WASI reactor build support (branch: wasi)
• protocolbuffers/protobuf - Official Protocol Buffers repository
• tetratelabs/wazero - Zero-dependency WebAssembly runtime for Go

About

This module provides the Protocol Buffers compiler (protoc) compiled to WebAssembly with WASI support using the reactor model. The WASM binary is embedded directly in the Go module, enabling protoc to run in Go applications without external dependencies or native binaries.

Reactor Model

Unlike the standard WASI "command" model that blocks in _start(), the reactor model exports functions that can be called multiple times, enabling:

• Multiple compilations per instance
• Reusable runtime without reloading the module
• Full control over the compilation lifecycle

Exported Functions

Protoc API:

• protoc_init - Initialize the compiler and register generators
• protoc_run - Run protoc with command-line arguments
• protoc_destroy - Clean up and free resources

Memory Management:

• malloc, free, realloc, calloc - For host to allocate memory

Built-in Generators

The WASM binary includes only the C++ generator (--cpp_out) to minimize size. All other languages (Go, Java, Python, etc.) are supported via plugins through the host's plugin handler.

Plugin Support

External protoc plugins (like protoc-gen-go) are supported via host function imports. When protoc needs to communicate with a plugin, it calls the plugin_communicate host function, which spawns the native plugin process on the host system.

Features

• Embeds protoc as a 3.1MB WASI WebAssembly binary
• Reactor model for multiple compilations per instance
• Plugin support via host function imports
• Virtual filesystem support for .proto files
• Thread-safe with mutex protection

Usage

package main

import (
    "bytes"
    "context"
    "fmt"
    "testing/fstest"

    "github.com/tetratelabs/wazero"
    protoc "github.com/aperturerobotics/go-protoc-wasi"
)

func main() {
    ctx := context.Background()
    r := wazero.NewRuntime(ctx)
    defer r.Close(ctx)

    // Create in-memory filesystem with .proto files
    memFS := fstest.MapFS{
        "example.proto": &fstest.MapFile{Data: []byte(`
syntax = "proto3";
package example;

message Person {
  string name = 1;
  int32 age = 2;
}
`)},
    }

    var stdout, stderr bytes.Buffer
    p, err := protoc.NewProtoc(ctx, r, &protoc.Config{
        Stdout: &stdout,
        Stderr: &stderr,
        FS:     memFS,
    })
    if err != nil {
        panic(err)
    }
    defer p.Close(ctx)

    // Initialize the compiler
    if err := p.Init(ctx); err != nil {
        panic(err)
    }

    // Show version
    exitCode, _ := p.Run(ctx, []string{"protoc", "--version"})
    fmt.Println(stdout.String())

    // Compile with a plugin (e.g., protoc-gen-go)
    // The plugin runs natively on the host via the PluginHandler
    exitCode, err = p.Run(ctx, []string{
        "protoc",
        "--go_out=/output",
        "--go_opt=paths=source_relative",
        "-I/",
        "/example.proto",
    })
    if exitCode != 0 {
        fmt.Println("Error:", stderr.String())
    }
}

Configuration

type Config struct {
    // Stdin is the standard input for protoc. Default: empty.
    Stdin io.Reader
    // Stdout is the standard output for protoc. Default: discard.
    Stdout io.Writer
    // Stderr is the standard error for protoc. Default: discard.
    Stderr io.Writer
    // FS is the filesystem for reading .proto files and writing output.
    FS fs.FS
    // FSConfig allows configuring the wazero filesystem.
    FSConfig wazero.FSConfig
    // PluginHandler handles spawning plugin processes.
    // Default: DefaultPluginHandler (uses os/exec).
    PluginHandler PluginHandler
}

Custom Plugin Handler

The default plugin handler spawns native processes using os/exec. You can provide a custom handler:

type PluginHandler interface {
    // Communicate spawns a plugin process and handles IPC.
    // program: plugin program name (e.g., "protoc-gen-go")
    // searchPath: if true, search PATH for the program
    // input: serialized CodeGeneratorRequest
    // Returns: serialized CodeGeneratorResponse, or error
    Communicate(ctx context.Context, program string, searchPath bool, input []byte) ([]byte, error)
}

Building the WASM Binary

The WASM binary is built from aperturerobotics/protobuf (branch: wasi):

# Clone the repository
git clone -b wasi https://github.com/aperturerobotics/protobuf.git
cd protobuf

# Build Abseil for WASI (required dependency)
ABSEIL_SOURCE=/path/to/abseil-cpp ./build-abseil-wasi.sh

# Build protoc for WASI
./build-wasi.sh

# Output: build-wasi/protoc.wasm (approximately 3.1MB)

Build Requirements

• WASI SDK 29.0+
• Abseil C++
• CMake 3.16+

Testing

go test -v ./...

License

MIT

protoc.wasm is covered by the Apache 2.0 license from the protobuf project.

Documentation

Overview

Package protoc provides a Go wrapper for running protoc via WASI/wazero.

Index

• Constants
• Variables
• func CompileProtoc(ctx context.Context, r wazero.Runtime) (wazero.CompiledModule, error)
• type Config
• type DefaultPluginHandler
  • func (h *DefaultPluginHandler) Communicate(ctx context.Context, program string, searchPath bool, input []byte) ([]byte, error)
• type PluginHandler
• type Protoc
  • func NewProtoc(ctx context.Context, r wazero.Runtime, cfg *Config) (*Protoc, error)
  • func NewProtocWithModule(ctx context.Context, r wazero.Runtime, compiled wazero.CompiledModule, ...) (*Protoc, error)
  • func (p *Protoc) Close(ctx context.Context) error
  • func (p *Protoc) Init(ctx context.Context) error
  • func (p *Protoc) Run(ctx context.Context, args []string) (int, error)

Constants

const (
	// ExportProtocInit initializes the protoc reactor.
	// Creates CLI instance and registers all built-in generators.
	// Signature: protoc_init() -> i32
	// Returns: 0 on success, non-zero on error
	ExportProtocInit = "protoc_init"

	// ExportProtocRun runs protoc with the given arguments.
	// protoc_init() must be called first.
	// Signature: protoc_run(argc: i32, argv: i32) -> i32
	// Returns: protoc exit code (0 on success)
	ExportProtocRun = "protoc_run"

	// ExportProtocDestroy destroys the protoc reactor and frees resources.
	// Signature: protoc_destroy() -> void
	ExportProtocDestroy = "protoc_destroy"
)

Protoc reactor exports

const (
	// ExportMalloc allocates memory in WASM linear memory.
	// Signature: malloc(size: i32) -> i32 (pointer)
	ExportMalloc = "malloc"

	// ExportFree frees memory in WASM linear memory.
	// Signature: free(ptr: i32) -> void
	ExportFree = "free"

	// ExportRealloc reallocates memory in WASM linear memory.
	// Signature: realloc(ptr: i32, size: i32) -> i32 (pointer)
	ExportRealloc = "realloc"

	// ExportCalloc allocates zeroed memory in WASM linear memory.
	// Signature: calloc(nmemb: i32, size: i32) -> i32 (pointer)
	ExportCalloc = "calloc"
)

Memory management exports

const (
	// ImportModuleProtoc is the import module name for protoc host functions.
	ImportModuleProtoc = "protoc"

	// ImportPluginCommunicate is the host function for plugin subprocess IPC.
	// This allows protoc to spawn plugin processes on the host.
	ImportPluginCommunicate = "plugin_communicate"
)

Host import module name for plugin subprocess communication

const ProtocWASMFilename = "protoc.wasm"

ProtocWASMFilename is the filename for ProtocWASM.

Variables

var ProtocWASM []byte

ProtocWASM contains the binary contents of the protoc WASI reactor build.

This is a reactor-model WASM that exports the protoc compiler API for reentrant execution in host environments. The reactor model allows multiple compilations per instance without reloading the module.

Functions

func CompileProtoc

func CompileProtoc(ctx context.Context, r wazero.Runtime) (wazero.CompiledModule, error)

CompileProtoc compiles the embedded protoc WASM module. The compiled module can be reused across multiple Protoc instances.

Types

type Config

type Config struct {
	// Stdin is the standard input for protoc. Default: empty.
	Stdin io.Reader
	// Stdout is the standard output for protoc. Default: discard.
	Stdout io.Writer
	// Stderr is the standard error for protoc. Default: discard.
	Stderr io.Writer
	// FS is the filesystem for reading .proto files and writing output.
	// Default: no filesystem access.
	FS fs.FS
	// FSConfig allows configuring the wazero filesystem.
	// If set, FS is ignored.
	FSConfig wazero.FSConfig
	// PluginHandler handles spawning plugin processes.
	// Default: DefaultPluginHandler (uses os/exec).
	PluginHandler PluginHandler
}

Config holds configuration for creating a new Protoc instance.

type DefaultPluginHandler

type DefaultPluginHandler struct{}

DefaultPluginHandler spawns plugin processes using os/exec.

func (*DefaultPluginHandler) Communicate

func (h *DefaultPluginHandler) Communicate(ctx context.Context, program string, searchPath bool, input []byte) ([]byte, error)

Communicate spawns a plugin and communicates via stdin/stdout.

type PluginHandler

type PluginHandler interface {
	// Communicate spawns a plugin process and handles IPC.
	// program: plugin program name (e.g., "protoc-gen-go")
	// searchPath: if true, search PATH for the program
	// input: serialized CodeGeneratorRequest
	// Returns: serialized CodeGeneratorResponse, or error
	Communicate(ctx context.Context, program string, searchPath bool, input []byte) ([]byte, error)
}

PluginHandler handles spawning and communicating with protoc plugins. The default implementation uses os/exec to spawn native processes.

type Protoc

type Protoc struct {
	// contains filtered or unexported fields
}

Protoc wraps a protoc WASI reactor module providing a high-level API for Protocol Buffer compilation.

func NewProtoc

func NewProtoc(ctx context.Context, r wazero.Runtime, cfg *Config) (*Protoc, error)

NewProtoc creates a new Protoc instance using the embedded WASM reactor. Call Close() when done to release resources.

func NewProtocWithModule

func NewProtocWithModule(ctx context.Context, r wazero.Runtime, compiled wazero.CompiledModule, cfg *Config) (*Protoc, error)

NewProtocWithModule creates a new Protoc instance using a pre-compiled module.

func (*Protoc) Close

func (p *Protoc) Close(ctx context.Context) error

Close destroys the protoc reactor and releases resources.

func (*Protoc) Init

func (p *Protoc) Init(ctx context.Context) error

Init initializes the protoc reactor. This must be called before Run.

func (*Protoc) Run

func (p *Protoc) Run(ctx context.Context, args []string) (int, error)

Run runs protoc with the given arguments. Init() must be called first. Returns the protoc exit code (0 on success).