docd

Documentation

Overview

Package docd hosts granite-docling behind the ocr/ipc wire.

It is the lightweight stand-in for a real inference daemon, and being a stand-in is the design. inferd already speaks this protocol and already embeds llama.cpp, so it is where this belongs — but it loads one warm model per process and that model is not granite-docling today. Rather than block the OCR path on someone else's roadmap, docd serves the identical wire: when inferd carries docling, the CLI points at its socket instead and nothing else changes, because neither side can tell the difference. That is what the byte-compatibility in ocr/ipc buys, and this package is the thing it buys it from.

The shape is three moving parts and no cleverness:

ocr verb  --ipc-->  docd  --http-->  llama-server  --->  granite-docling

docd owns the model's lifetime and the protocol translation. It does not own the inference: llama.cpp does that, because a VLM runtime is not a thing to reimplement for this.

Why a subprocess and not a library

llama.cpp is C++, so linking it means CGO, and CGO means the single static cross-compiling binary this repo values stops existing. A subprocess keeps that: the CLI stays pure Go and stays small, the GPU-linked code lives in a process that can be missing entirely, and a model that segfaults on a malformed page kills a child rather than the run. The cost is a process boundary and an HTTP hop, which against tens of seconds of generation per page does not register.

The executable is found, never downloaded

docd locates llama-server on PATH and reports how to install it when absent. It does not fetch it. Downloading and executing a binary is a supply-chain step of a different kind than downloading data, and it is not one a PDF tool should take on its user's behalf silently.

Model weights are different, and llama.cpp's own -hf flag fetches them into its cache with its own integrity checks. That is deliberately not reimplemented here: a second downloader would mean a second cache, a second checksum policy, and two places for a partially written GGUF to hide.

Index

• Constants
• type Host
  • func Start(ctx context.Context, o Options) (*Host, error)
  • func (h *Host) Close() error
  • func (h *Host) Generate(ctx context.Context, img *image.RGBA, prompt string, maxTokens int, ...) error
  • func (h *Host) Name() string
• type Options

Constants

const (
	DefaultPort  = 18080
	DefaultReady = 10 * time.Minute
)

Defaults. The readiness bound is generous because it covers a first run, where llama.cpp is downloading ~500 MB of GGUF and mmproj before it can load anything; a subsequent start is seconds. Failing at 30 s would make the first run of the tool look broken exactly once, which is the worst time for it.

const Model = "ibm-granite/granite-docling-258M-GGUF"

Model is the HuggingFace repo llama.cpp downloads when none is configured.

granite-docling-258M rather than a larger docling model, for reasons that are all about this being the default: it is around an eighth the size, IBM publishes an official GGUF with a matching mmproj so the vision tower is not a separate hunt, the base weights are Apache-2.0, and it emits DocTags — structured output the parser reads directly, rather than prose that would have to be re-analysed. A repo whose license is MIT cannot make a copyleft model its default, so the license is a gate here, not a preference.

Types

type Host

type Host struct {
	// contains filtered or unexported fields
}

Host is a running llama-server plus the translation from this repo's wire to its HTTP API. It satisfies ipc.Handler.

func Start

func Start(ctx context.Context, o Options) (*Host, error)

Start launches llama-server and waits until its model is loaded.

Returns only when the server reports ready, so a caller can bind its own socket immediately afterwards and have a successful connect mean the model is warm. That ordering is the protocol's readiness signal — there is no separate health frame — so getting it backwards would make every client's first request fail instead of wait.

func (*Host) Close

func (h *Host) Close() error

Close stops llama-server.

func (*Host) Generate

func (h *Host) Generate(ctx context.Context, img *image.RGBA, prompt string, maxTokens int, emit func(string)) error

Generate converts one page, streaming DocTags to emit as they arrive.

Uses /v1/chat/completions with a data-URI image rather than /completion with multimodal_data, and that choice is not stylistic. /completion requires the caller to place llama.cpp's media marker in the prompt itself, and that marker is randomized per server process — get_media_marker in server-common.cpp returns "<__media_" + random + "__>" unless LLAMA_MEDIA_MARKER is set, while mtmd's own mtmd_default_marker still returns the documented "<__media__>". The two disagree, so a client hardcoding the documented value fails against a real server. The chat endpoint substitutes the live marker itself, which removes the whole problem instead of working around it.

func (*Host) Name

func (h *Host) Name() string

Name identifies the backend on the wire's done frame.

type Options

type Options struct {
	// Model is the -hf repo. Empty means Model.
	Model string

	// Exe is the llama-server executable. Empty means look on PATH.
	Exe string

	// Port is llama-server's HTTP port. Zero means DefaultPort.
	//
	// Bound on loopback only, and there is no option to change that: this is a
	// process-private channel between two halves of one tool, and an inference
	// endpoint reachable off-box is an unauthenticated GPU for anyone who finds it.
	Port int

	// GPULayers is -ngl. Zero means CPU only, which is the honest default — a
	// machine without a usable GPU would otherwise fail at model load with an error
	// from a layer of the stack that cannot explain itself.
	GPULayers int

	// Ctx is the context window, -c. Zero means llama-server's default.
	Ctx int

	// Ready bounds the wait for the model to load. Zero means DefaultReady.
	Ready time.Duration

	Log *slog.Logger
}

Options configure a host.