serpent

README

Serpent 🐍

A Go library for embedding and running Python code directly in your Go applications with high performance and concurrency support.

Features

• Embed Python code directly in Go using go:embed
• Type-safe API with Go generics for input/output
• Concurrent execution with a pool of sub-interpreters
• Automatic Python library discovery on macOS, Linux, and other Unix systems
• Multiple execution modes: return values, write to streams, or use custom I/O

Installation

go get github.com/adamkeys/serpent

Requirements

• Go 1.18 or later
• Python 3 shared library installed on your system

Quick Start

package main

import (
    "fmt"
    "log"
    "github.com/adamkeys/serpent"
)

func main() {
    lib, err := serpent.Lib()
    if err != nil {
        log.Fatal(err)
    }
    if err := serpent.Init(lib); err != nil {
        log.Fatal(err)
    }

    program := serpent.Program[int, int]("def run(input): return input * 2")
    result, err := serpent.Run(program, 21)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(result) // Output: 42
}

See the examples/ directory for more complete demonstrations including concurrent execution, embedding Python files, and using external libraries like HuggingFace Transformers.

API Reference

Initialization

• Lib() (string, error) - Automatically discovers the Python shared library path
• Init(libPath string) error - Initializes the Python interpreter with a worker pool
• InitSingleWorker(libPath string) error - Initializes with a single worker (for libraries that don't support sub-interpreters)
• Close() error - Cleans up and shuts down the interpreter

Execution

• Run[I, O](program Program[I, O], input I) (O, error) - Executes Python code and returns the result
• RunWrite[I](w io.Writer, program Program[I, Writer], input I) error - Executes Python code that writes to a Go io.Writer

Reusable Executables

For programs you want to call multiple times, use Load to create a reusable executable:

• Load[I, O](program Program[I, O]) (*Executable[I, O], error) - Loads a program for repeated execution
• LoadWriter[I](program Program[I, Writer]) (*WriterExecutable[I], error) - Loads a writer program for repeated execution

exec, err := serpent.Load(program)
if err != nil {
    log.Fatal(err)
}
defer exec.Close()

// Call multiple times - state persists between calls
result1, _ := exec.Run(input1)
result2, _ := exec.Run(input2)

An Executable is pinned to a single worker on its first Run() call, so module-level state (imports, global variables) persists across calls. This is useful for expensive initialization like loading ML models:

from transformers import pipeline

# Loaded once, reused across all Run() calls
ner = pipeline("ner")

def run(input):
    return ner(input)

Program Definition

A Program[I, O] is simply a string containing Python code:

type Program[I, O any] string

Your Python code should define a run function:

• def run(input): - For Run, return the result value
• def run(input, writer): - For RunWrite, write to the provided writer

Python Code Guidelines

Returning Values

Define a run function that takes input and returns the result:

def run(input):
    return input.upper()

Writing Output

When using RunWrite, your run function receives a writer object:

def run(input, writer):
    writer.write(f"Hello from Python: {input}\n")

The writer object provides:

• write(data) - Write string or bytes (strings are auto-encoded as UTF-8)
• flush() - Flush the output (no-op, writes are unbuffered)

The writer is automatically closed when your function returns.

Using External Libraries

Python code can import any library available in the Python environment:

from transformers import pipeline

ner = pipeline("ner", grouped_entities=True)

def run(input):
    entities = ner(input)
    return [e["word"] for e in entities]

Note: Libraries that don't support sub-interpreters require initialization with InitSingleWorker() instead of Init().

Examples

The examples/ directory contains several demonstrations:

• hello/ - Basic concurrent "Hello World" example
• identity/ - Simple identity transformation
• transformers/ - Named entity recognition using HuggingFace Transformers

Environment Variables

• LIBPYTHON_PATH - Override automatic library discovery by specifying the Python shared library path directly

How It Works

Serpent uses purego to dynamically load and call Python's C API without CGO. It manages a pool of Python sub-interpreters (each running on its own OS thread) to enable safe concurrent execution of Python code from multiple goroutines.

Input and output values are serialized as JSON, providing a simple and type-safe interface between Go and Python.

Platform Support

• ✅ macOS (Darwin)
• ✅ Linux
• ✅ Unix-like systems

Documentation

Overview

Package serpent provides functions for interacting with a Python interpreter.

Index

• Variables
• func Close() error
• func Init(libraryPath string) error
• func InitSingleWorker(libraryPath string) error
• func Lib() (string, error)
• func Run[TInput, TResult any](program Program[TInput, TResult], arg TInput) (TResult, error)
• func RunWrite[TInput any](w io.Writer, program Program[TInput, Writer], arg TInput) error
• type Executable
  • func Load[TInput, TResult any](program Program[TInput, TResult]) (*Executable[TInput, TResult], error)
  • func (b *Executable) Close() error
  • func (e *Executable[TInput, TResult]) Run(arg TInput) (TResult, error)
• type Program
• type PythonNotInitialized
• type Writer
• type WriterExecutable
  • func LoadWriter[TInput any](program Program[TInput, Writer]) (*WriterExecutable[TInput], error)
  • func (b *WriterExecutable) Close() error
  • func (e *WriterExecutable[TInput]) Run(w io.Writer, arg TInput) error

Variables

var (
	// ErrAlreadyInitialized is returned when the Python interpreter is initialized more than once.
	ErrAlreadyInitialized = errors.New("already initialized")
	// ErrRunFailed is returned when the Python program fails to run.
	ErrRunFailed = errors.New("run failed")
	// ErrSubInterpreterFailed is returned when a sub-interpreter fails to initialize.
	ErrSubInterpreterFailed = errors.New("sub-interpreter creation failed")
	// ErrNoHealthyWorkers is returned when all workers have failed.
	ErrNoHealthyWorkers = errors.New("no healthy workers available")
	// ErrNotInitialized is returned when Close is called before Init.
	ErrNotInitialized = errors.New("not initialized")
)

var ErrLibraryNotFound = errors.New("library not found")

ErrLibraryNotFound is returned when the Python shared library cannot be found.

Functions

func Close

func Close() error

Close shuts down the Python interpreter and all workers.

func Init

func Init(libraryPath string) error

Init initializes the Python interpreter with runtime.NumCPU() workers. This must be called before any other functions in this package. When using packages that are incompatible with sub-interpreters, use InitSingleWorker instead.

func InitSingleWorker

func InitSingleWorker(libraryPath string) error

InitSingleWorker initializes the Python interpreter with a single worker, disabling sub-interpreters. Use this when running Python code that uses C extension modules incompatible with sub-interpreters. This must be called before any other functions in this package. Use Init for normal usage.

func Lib

func Lib() (string, error)

Lib attempts to find a Python shared library on the system and returns the path if found. If the library cannot be found, ErrLibraryNotFound is returned. If the LIBPYTHON_PATH envrionment variable is set, the value of that environment variable is returned.

func Run

func Run[TInput, TResult any](program Program[TInput, TResult], arg TInput) (TResult, error)

Run runs a Program with the supplied argument and returns the result. The Python code must define a run() function that accepts the input and returns a JSON-serializable value.

Example Python program:

def run(input):
    return input + 1

func RunWrite

func RunWrite[TInput any](w io.Writer, program Program[TInput, Writer], arg TInput) error

RunWrite runs a Program with the supplied argument with the Python program writing to the supplied writer. The Python code must define a run() function that accepts the input and a writer object.

Example Python program:

def run(input, writer):
    writer.write(b'OK')

Types

type Executable

type Executable[TInput, TResult any] struct {
	// contains filtered or unexported fields
}

Executable represents a loaded Python program that can be called multiple times. It is pinned to a single worker on first Run(), preserving module-level state across calls. An Executable is not safe for concurrent use; create a separate instance for each goroutine.

func Load

func Load[TInput, TResult any](program Program[TInput, TResult]) (*Executable[TInput, TResult], error)

Load loads a Python program and returns an Executable that can be called multiple times. The executable is pinned to a worker on first Run(), and all subsequent calls use the same worker.

func (*Executable) Close

func (b *Executable) Close() error

Close releases resources associated with the executable.

func (*Executable[TInput, TResult]) Run

func (e *Executable[TInput, TResult]) Run(arg TInput) (TResult, error)

Run executes the loaded program with the given input. On first call, the program is loaded on a worker and pinned to it. Subsequent calls reuse the same worker and loaded state.

type Program

type Program[TInput, TResult any] string

Program identifies a Python program.

type PythonNotInitialized

type PythonNotInitialized string

PythonNotInitialized is a panic type indicating that the Python interpreter has not been initialized.

type Writer

type Writer struct{}

Writer is a result type which indicates that the program writes to the output. e.g. Program[string, Writer] is a program that writes to the output.

type WriterExecutable

type WriterExecutable[TInput any] struct {
	// contains filtered or unexported fields
}

WriterExecutable represents a loaded Python program that writes to an output stream. A WriterExecutable is not safe for concurrent use; create a separate instance for each goroutine.

func LoadWriter

func LoadWriter[TInput any](program Program[TInput, Writer]) (*WriterExecutable[TInput], error)

LoadWriter loads a Python program that writes to an output stream.

func (*WriterExecutable) Close

func (b *WriterExecutable) Close() error

Close releases resources associated with the executable.

func (*WriterExecutable[TInput]) Run

func (e *WriterExecutable[TInput]) Run(w io.Writer, arg TInput) error

Run executes the loaded program, writing output to the provided writer.