README
¶
go-anydoc
English | 简体中文
Go bindings for anydoc — convert Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV and text-based PDF to Markdown.
No cgo. No subprocesses. No system dependencies. The Rust library is compiled to WebAssembly and embedded, so a binary using this package is self-contained and cross-compiles wherever Go does:
CGO_ENABLED=0 GOOS=windows GOARCH=arm64 go build ./...
Install
go get github.com/xusenlin/go-anydoc
Usage
c, err := anydoc.New()
if err != nil {
return err
}
defer c.Close(ctx)
md, err := c.Convert(ctx, docBytes, "docx")
Pass "" as the hint to detect the format from content. CSV has no signature
and must be named explicitly.
Errors are matched with errors.Is:
switch {
case errors.Is(err, anydoc.ErrEncrypted): // password-protected
case errors.Is(err, anydoc.ErrUnsupported): // not a format anydoc parses
case errors.Is(err, anydoc.ErrMalformed): // recognised but corrupt
}
Design notes
Interpreter, not the optimising compiler. wazero can either translate the
module to native machine code up front (fast to run, expensive to load, amd64
and arm64 only) or interpret it instruction by instruction (cheap to load,
portable everywhere, slower to run). This package interprets by default,
because a library that ships inside other people's binaries cannot assume it
may pre-warm a compilation cache, cannot assume the host allows the executable
pages the compiler needs — macOS hardened runtime and some seccomp profiles
refuse them — and cannot assume the target is amd64 or arm64. WithCompiler
opts into the other side of that trade.
The trade is real and it scales with document size, so measure against your own corpus before assuming it is free:
| interpreter (default) | compiler (WithCompiler) |
|
|---|---|---|
New() — once per process |
~85 ms | ~2.1 s |
| 1 KB docx | 5.0 ms | 1.0 ms |
| docx with a 5 MB uncompressed body | 16.9 s | 1.1 s |
RSS after New() |
135 MB | 576 MB |
Ordinary office documents are in the second row's territory and cost nothing
worth optimising. Multi-megabyte ones are ~16× slower than they would be
compiled, so if you convert those, either bound the tail with
WithMaxInputBytes and a context deadline — cancellation interrupts the guest
mid-conversion — or opt into WithCompiler.
Measured on Apple M5 Pro, macOS 26.5, Go 1.26.1, wazero v1.9.0, against
anydoc.wasm 5,069,778 bytes (anydoc 0.1.7). The 5 MB figure is a 25,000-row
table: 42 KB zipped, since the size that matters is the uncompressed body.
A fresh guest per document. Each Convert instantiates its own linear
memory, so a large document cannot leave memory permanently claimed and a
malformed one cannot leak state into the next call. Compilation, the expensive
part, happens once in New.
A command module, not exported functions. The guest reads stdin and writes
stdout, so the Go side never touches linear memory, pointers, or UTF-8
boundaries. The exit code carries the error kind. See rust/src/main.rs — the
code table there and in errors.go is a two-language contract.
Sandboxed. WASI is instantiated for stdio and random_get (which HashMap
seeding reaches for). No filesystem, no clock, no sockets.
Controlling size
The module is embedded by default so go get and New() just work. Builds that
would rather ship it out of band:
go build -tags anydoc_nowasm # 4.87 MB smaller
embeddedWASM is then nil and New requires WithWASM(r) or
WithWASMBytes(b). Useful for container layering, serverless size limits,
pinning a different anydoc build, or environments that forbid opaque embedded
blobs.
The saving is the 4.83 MB module plus ~35 KB of embed machinery. For scale,
examples/convert is 12.4 MB built normally and 7.3 MB with the tag.
The build tag affects the compiled binary, not go get: the module is in the
Go module either way.
Tuning
anydoc.New(
anydoc.WithConcurrency(4), // simultaneous guests; the main memory lever
anydoc.WithMemoryLimitPages(1024), // 64 MiB per guest
anydoc.WithMaxInputBytes(64<<20),
anydoc.WithCompiler(), // throughput over startup cost; see below
)
WithCompiler
Compiles the module to native code instead of interpreting it, turning the
table above from the left column into the right one. The cost is paid once,
in New — Convert only instantiates the already-compiled module — so it
pays off in a long-lived process that reuses one Converter, and is a pure
loss in a short-lived one that converts a single small document and exits
(~2.1 s bought to save ~4 ms).
It is a request, not a guarantee. The backend needs amd64 or arm64 on a mainstream OS, plus a host that permits mmap'd executable pages. Where that does not hold — riscv64, ppc64le, 386, macOS hardened runtime, some seccomp profiles — wazero falls back to the interpreter silently and the conversion still happens, just at interpreter speed. Cross-compilation is unaffected either way: wazero is pure Go, and this option changes no build constraints.
WithMemoryLimitPages is validated against the module's declared minimum at
New time, not at conversion time — setting it too low fails fast with a clear
message rather than surfacing later as a mysterious conversion error.
How much a guest actually needs tracks the uncompressed content, not the size of the file on disk — a zip bomb is small on disk and large in memory, which is why the limit exists:
| document | pages needed |
|---|---|
| module minimum, converts nothing | 64 (4 MiB) |
| docx, 0.4 MB body | 192 (12 MiB) |
| docx, 2 MB body | 576 (36 MiB) |
| PDF, 7.5 MB file | 512–1024 (32–64 MiB) |
| docx, 5 MB body | 1280 (80 MiB) |
The default of 1024 pages converts an ordinary 7.5 MB PDF. Raise it if
conversions fail on inputs that convert fine elsewhere — the error says so
explicitly when the guest ran out of memory, and names this option. Each
concurrent guest can claim up to this much, so WithConcurrency multiplied by
this value is your real ceiling.
Development
Tasks are run with Task; task --list shows them all.
task test # builds the wasip1 test stub, runs the harness suite
task verify # everything CI runs, and the checklist before tagging
task wasm # rebuilds the embedded module; needs Rust 1.88 + wasm-opt
There are two suites, split by build tag:
anydoc_test.goruns under-tags anydoc_nowasmagainst a stub built with Go's ownwasip1target. The stub speaks the same ABI without converting anything, so the harness — stdio wiring, exit-code mapping, concurrency, cancellation, memory containment — is testable with no Rust toolchain in the loop. Seetestdata/README.md.anydoc_real_test.gois tagged!anydoc_nowasm, so it compiles exactly whenanydoc.wasmis in the tree, and covers what the harness is for: real conversion output, format detection, and the error codes the crate actually emits.
anydoc.wasm is a build artifact committed to the repo, because Go modules
have no build step: whatever is committed is what go get delivers. It is
built from the pinned crate by task wasm, run by hand rather than by CI, and
anydoc.wasm.sha256 records the checksum of the committed copy. See
anydoc.wasm.README for the toolchain versions needed to reproduce it.
Versioning
This module's version is independent of the crate's. The crate is pinned with
= in rust/Cargo.toml, and moving to a new upstream release is a deliberate
act: edit the pin, run task wasm, review what changed in the conversion
output, then tag. An upstream release on its own changes nothing here.
anydoc.EmbeddedAnydocVersion reports which crate version the embedded module
was built from, and anydoc.Info() prints it with the payload size. It is
copied from rust/Cargo.lock by task sync-version, and task check-version
fails the build if the two ever disagree — so it cannot quietly misreport what
is actually embedded.
rust/Cargo.lock is committed too. The = pin only fixes anydoc itself; the
lockfile is what stops a transitive dependency from silently changing the
conversion output between one rebuild and the next.
License
MIT for this package — see LICENSE. The embedded module is built from
anydoc, also MIT — see LICENSE-anydoc.
Documentation
¶
Overview ¶
Package anydoc converts office documents to Markdown by running the anydoc Rust library as WebAssembly, with no cgo and no external processes.
The wasm module is embedded, so a binary built with this package is self-contained and cross-compiles anywhere Go does:
CGO_ENABLED=0 GOOS=windows GOARCH=arm64 go build ./...
Basic use:
c, err := anydoc.New() defer c.Close(ctx) md, err := c.Convert(ctx, docBytes, "docx")
Index ¶
Constants ¶
const EmbeddedAnydocVersion = "0.1.7"
EmbeddedAnydocVersion is the upstream crate version this wasm was built from. Copied from rust/Cargo.lock by `task sync-version`; `task check-version` fails the build if the two drift apart.
Variables ¶
var ( // ErrUnsupported means the input was not a format anydoc can parse, either // because the content signature matched nothing or the hint named a format // that does not exist. ErrUnsupported = errors.New("anydoc: unsupported format") // ErrMalformed means the format was recognised but the bytes were corrupt. ErrMalformed = errors.New("anydoc: malformed document") // ErrEncrypted means the document is password-protected. anydoc does not // decrypt; obtain a decrypted copy first. ErrEncrypted = errors.New("anydoc: document is encrypted") // ErrResourceLimit means a guard inside anydoc tripped, e.g. a zip bomb or // an implausible declared size. Distinct from the host-side memory limit, // which surfaces as ErrWASM. ErrResourceLimit = errors.New("anydoc: resource limit exceeded") // ErrMissingPart means a required part of a container format was absent, // e.g. a .docx with no word/document.xml. ErrMissingPart = errors.New("anydoc: missing required part") // ErrWASM means the guest trapped, ran out of memory, or otherwise failed // in a way that is not a document problem. Treat as an internal error. ErrWASM = errors.New("anydoc: wasm execution failed") // ErrNoWASM means no module was available: the build used the // anydoc_nowasm tag and no WithWASM option was supplied. ErrNoWASM = errors.New("anydoc: no wasm module (build without anydoc_nowasm, or pass WithWASM)") )
Sentinel errors returned by Convert. Match them with errors.Is.
The numeric codes behind these live in rust/src/main.rs and are part of the module ABI: a wasm built from a different revision of that file may not agree with this table. That is what EmbeddedAnydocVersion is for.
Functions ¶
Types ¶
type ConvertError ¶
type ConvertError struct {
Kind error // one of the sentinels above
Detail string // guest stderr, may be empty
Code int // raw exit code, for debugging
}
ConvertError carries the guest's stderr alongside a sentinel, so callers can branch on the class of failure and still surface anydoc's own message.
func (*ConvertError) Error ¶
func (e *ConvertError) Error() string
func (*ConvertError) Unwrap ¶
func (e *ConvertError) Unwrap() error
type Converter ¶
type Converter struct {
// contains filtered or unexported fields
}
Converter runs conversions. It is safe for concurrent use and should be created once and reused: New compiles the module, which is the expensive part, and Convert then instantiates a cheap short-lived guest per document.
func New ¶
New compiles the wasm module and prepares the runtime.
Compilation happens once, here, and costs a few hundred milliseconds. Do it at startup, not per request.
func (*Converter) Convert ¶
Convert turns a document into Markdown.
hint is a bare extension such as "docx" that selects the parser. Pass "" to detect from content, which works for every format except CSV, which has no signature and must be named.
Each call gets a fresh guest instance with its own linear memory. A large document therefore cannot leave memory permanently claimed, and a malformed one cannot leak state into the next call.
func (*Converter) ConvertReader ¶
ConvertReader is a convenience wrapper that reads r fully before converting. The whole document has to be in memory regardless: it is copied into the guest's linear memory.
type Option ¶
type Option func(*config)
Option configures a Converter.
func WithCompiler ¶
func WithCompiler() Option
WithCompiler trades startup cost and memory for throughput, by having wazero compile the module to native code instead of interpreting it.
Measured on the real module, per document: a 1 KB docx goes from 5.0ms to 1.0ms, and one with a 5 MB uncompressed body from 16.9s to 1.1s. The price is paid once in New, which goes from ~85ms and 135 MB of RSS to ~2.1s and 576 MB. That 576 MB is an OOM kill in a 512 MB container, which is why this is opt-in rather than the default.
The compilation is paid once, in New, not per document: Convert only instantiates the already-compiled module. So this pays off in a long-lived process that reuses one Converter, and is a pure loss in a short-lived one that converts a single small document and exits -- there it buys ~4ms for ~2s. Leave it off for tight containers too.
This is a request, not a guarantee. The compiler backend needs amd64 or arm64 (with SSE4.1 on amd64) on a mainstream OS, and needs the host to allow mmap'd executable pages -- macOS hardened runtime and some seccomp profiles refuse them. Where it is unavailable, wazero falls back to the interpreter silently: the conversion still happens, at interpreter speed. Nothing here affects cross-compilation, since wazero is pure Go.
func WithConcurrency ¶
WithConcurrency caps simultaneous guest instances. Each holds its own linear memory, so this is the main lever on peak memory. Default 4.
func WithMaxInputBytes ¶
WithMaxInputBytes rejects documents larger than n before any wasm runs. Zero disables the check. Default 64 MiB.
func WithMemoryLimitPages ¶
WithMemoryLimitPages caps one guest's linear memory in 64 KiB pages. Default 1024 (64 MiB).
What a guest needs tracks the uncompressed content rather than the input's size on disk, so a compact file can still be expensive. Measured against the real module: 64 pages is the module's own floor and converts nothing, a docx with a 0.4 MB body needs 192, a 2 MB body 576, a 5 MB body 1280, and a 7.5 MB PDF needs between 512 and 1024.
Raise this when conversions fail with ErrWASM on inputs that convert fine elsewhere; the error says so explicitly when the guest ran out of memory. Each concurrent guest can claim up to this much, so this multiplied by WithConcurrency is the real ceiling.
Validated at New time against the module's declared minimum, so too low a value fails immediately rather than at conversion.
func WithWASM ¶
WithWASM supplies the module instead of using the embedded one.
This is the counterpart to building with -tags anydoc_nowasm, where nothing is embedded and this option becomes mandatory. It is also useful without the tag: pinning a different anydoc build, loading from a shared volume, or keeping the payload in its own container layer.
The module is read immediately; r is not retained.
func WithWASMBytes ¶
WithWASMBytes is WithWASM for a module already in memory.
Source Files
Directories