downmark

README

downmark

Convert documents to Markdown. Pure Go — no Python, no cgo, no external tools. One static binary; also usable as a library where you only link the formats you import.

downmark is a native Go re-imagining of Microsoft's markitdown: the same job (get documents into clean, LLM-friendly Markdown) without the Python runtime and native dependencies.

Supported formats

Format	Package	Notes
PDF	convert/pdf	Text extraction via github.com/giraffesyo/pdf: Form XObjects (Google Docs exports), Identity-H composite fonts, ToUnicode CMaps, segmented content streams, inline images; hard budgets against decompression bombs. No OCR: scanned PDFs return a clear error.
DOCX	convert/docx	Headings, bold/italic/strikethrough, sub/superscript, nested lists, tables (incl. gridSpan/vMerge), hyperlinks, image placeholders, tracked changes. Equations degrade to plain text.
XLSX	convert/xlsx	Every sheet as ## SheetName + a Markdown table.
PPTX	convert/pptx	Slides in order with <!-- Slide number: N --> markers, position-based reading order, tables, chart data tables, image alt text, speaker notes.
HTML	convert/html	DOM sanitization (scripts/styles dropped, javascript: links unwrapped, data URIs truncated) + GFM tables.
CSV	convert/csv	Charset-aware (incl. Shift-JIS/cp932), delimiter sniffing (, ; \t |).
Plain text	built into core	Charset-detected passthrough (UTF-8/16, legacy encodings).

Install the CLI

go install github.com/giraffesyo/downmark/cmd/downmark@latest

usage: downmark [flags] [file]      # stdin if file omitted or "-"; Markdown → stdout
  -o file           write output to file
  -x .ext           extension hint (leading dot optional)
  -m type           MIME type hint
  -c charset        charset hint, e.g. shift_jis
  -keep-data-uris   keep full data: URIs
  -version          print version

Exit codes: 0 success, 1 conversion failed, 2 usage error.

Library

Pay for what you import: the core engine pulls only lightweight detection dependencies; each format package brings its own.

import (
    "github.com/giraffesyo/downmark"
    "github.com/giraffesyo/downmark/convert/pdf"
)

e := downmark.New() // core engine (plain-text passthrough builtin)
pdf.Register(e)     // link only the PDF stack

res, err := e.ConvertFile(ctx, "report.pdf")
// res.Markdown, res.Title

Everything wired up (what the CLI uses):

import "github.com/giraffesyo/downmark/all"

res, err := all.ConvertFile(ctx, "report.docx")
// or: e := all.New(all.Options{KeepDataURIs: true})

Indicative binary sizes (-trimpath -ldflags "-s -w", darwin/arm64): a PDF-only consumer ≈ 4.9 MB, CSV-only ≈ 4.5 MB, everything ≈ 7.5 MB.

Custom converters implement downmark.Converter (use StreamInfo.Matches in Accepts) and register with e.Register(conv, downmark.PrioritySpecific); converters registered later at the same priority are tried first, so yours shadow the builtins.

Errors: errors.Is(err, downmark.ErrUnsupportedFormat) when nothing matched; *downmark.ConversionError (with per-converter attempts) when converters matched but failed.

PDF extraction

PDF text extraction lives in its own module, github.com/giraffesyo/pdf, usable without the converter framework — positioned glyphs, line/word reconstruction, and hardening against malformed and hostile files.

Limitations

• No OCR. Scanned/image-only PDFs and text-converted-to-outlines fail with "no extractable text" rather than silently emitting nothing.
• PDF: complex multi-column layouts may interleave; fonts lacking both a ToUnicode map and a standard encoding are dropped rather than emitted as garbage; borderless table-layout reconstruction is not yet implemented.
• DOCX: headers/footers, footnotes, comments, and text boxes are skipped; equations render as plain text; nested tables flatten to text.
• Encrypted Office files and PDFs are not decrypted.
• Non-seekable inputs (stdin, network streams) are buffered fully in memory.

Development

go test ./...                       # unit + vector + golden tests
go test -run TestGolden -update .   # refresh golden files
golangci-lint run ./...             # lint (config in .golangci.yml)

Test fixtures under testdata/ partly come from microsoft/markitdown (MIT); see testdata/NOTICE.md.

License

MIT

Documentation

Overview

Package downmark converts documents to Markdown.

This package is the core: the Engine, converter registry, content-type detection, and a plain-text converter. Format converters live in separate packages so binaries link only the formats they use:

• github.com/giraffesyo/downmark/convert/pdf
• github.com/giraffesyo/downmark/convert/docx
• github.com/giraffesyo/downmark/convert/xlsx
• github.com/giraffesyo/downmark/convert/pptx
• github.com/giraffesyo/downmark/convert/html
• github.com/giraffesyo/downmark/convert/csv

The github.com/giraffesyo/downmark/all package wires every converter into one engine for consumers who want the full matrix. Standalone PDF text extraction lives in the separate github.com/giraffesyo/pdf module.

Picking formats

e := downmark.New()          // engine with plain-text passthrough only
pdf.Register(e)              // + PDF
docx.Register(e, docx.Options{}) // + DOCX

res, err := e.ConvertFile(ctx, "report.pdf")
if err != nil { ... }
fmt.Println(res.Markdown)

Streams work too; pass whatever type hints you have and detection fills in the rest by sniffing content:

res, err := e.Convert(ctx, resp.Body, downmark.StreamInfo{
	MIMEType: resp.Header.Get("Content-Type"),
})

Custom converters

Implement Converter and register it; among converters with equal priority the most recently registered wins, so user converters shadow builtins. StreamInfo.Matches is the standard Accepts building block. A converter may also implement interface{ Name() string } to control how it is reported in errors.

Errors

When no converter accepts an input, errors.Is(err, ErrUnsupportedFormat) reports true. When converters accepted but all failed, the error is a *ConversionError carrying each attempt.

Index

• Variables
• type AttemptError
  • func (e *AttemptError) Error() string
  • func (e *AttemptError) Unwrap() error
• type ConversionError
  • func (e *ConversionError) Error() string
  • func (e *ConversionError) Unwrap() []error
• type Converter
• type Engine
  • func New(opts ...Option) *Engine
  • func (e *Engine) Convert(ctx context.Context, r io.Reader, hints StreamInfo) (*Result, error)
  • func (e *Engine) ConvertFile(ctx context.Context, path string) (*Result, error)
  • func (e *Engine) Register(c Converter, p Priority)
• type Option
  • func WithHTTPClient(c *http.Client) Option
  • func WithoutBuiltins() Option
• type Priority
• type Result
• type StreamInfo
  • func (s StreamInfo) Matches(exts []string, mimePrefixes []string) bool

Variables

var ErrUnsupportedFormat = errors.New("downmark: unsupported format")

ErrUnsupportedFormat is returned when no registered converter accepted the input under any detected interpretation of its type.

Types

type AttemptError

type AttemptError struct {
	// Converter is the name of the converter that failed, e.g. "pdf".
	Converter string
	// Err is the error the converter returned.
	Err error
}

AttemptError records a single converter's failed conversion attempt.

func (*AttemptError) Error

func (e *AttemptError) Error() string

func (*AttemptError) Unwrap

func (e *AttemptError) Unwrap() error

type ConversionError

type ConversionError struct {
	Attempts []*AttemptError
}

ConversionError is returned when at least one converter accepted the input but every attempt failed. Attempts lists each failure in the order tried.

func (*ConversionError) Error

func (e *ConversionError) Error() string

func (*ConversionError) Unwrap

func (e *ConversionError) Unwrap() []error

type Converter

type Converter interface {
	// Accepts reports whether the converter wants to try this stream.
	Accepts(info StreamInfo) bool
	// Convert reads input (positioned at offset 0) and produces Markdown.
	Convert(ctx context.Context, input io.ReadSeeker, info StreamInfo) (*Result, error)
}

Converter converts one class of documents to Markdown.

Implementations must judge Accepts from the StreamInfo alone, never by reading the stream; content sniffing happens in the Engine's detection layer before Accepts is called.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine holds a converter registry and conversion options. Create one with New; the zero value is not usable.

func New

func New(opts ...Option) *Engine

New returns an Engine with only the builtin plain-text converter registered (unless WithoutBuiltins is given). Register format converters explicitly from the convert/... packages, or use the all package for an engine with every format wired up.

func (*Engine) Convert

func (e *Engine) Convert(ctx context.Context, r io.Reader, hints StreamInfo) (*Result, error)

Convert converts an arbitrary reader to Markdown. hints may be the zero value. If r is not an io.ReadSeeker its content is buffered into memory; seekable inputs are rewound to offset 0.

func (*Engine) ConvertFile

func (e *Engine) ConvertFile(ctx context.Context, path string) (*Result, error)

ConvertFile opens path and converts it, deriving Extension, Filename, and LocalPath from the path.

func (*Engine) Register

func (e *Engine) Register(c Converter, p Priority)

Register adds a converter at the given priority. Among converters with equal priority, the most recently registered is tried first, so user converters shadow builtins.

type Option

type Option func(*Engine)

Option configures an Engine created by New.

func WithHTTPClient

func WithHTTPClient(c *http.Client) Option

WithHTTPClient sets the HTTP client used for URL-based conversions.

func WithoutBuiltins

func WithoutBuiltins() Option

WithoutBuiltins starts the Engine with an empty converter registry (omitting even the plain-text converter). Use Register to add converters.

type Priority

type Priority int

Priority orders converters: lower values are tried first.

const (
	// PrioritySpecific is for converters keyed to a specific file format
	// (docx, xlsx, pptx, pdf, csv).
	PrioritySpecific Priority = 0
	// PriorityGeneric is for near-catch-all converters (html, plain text).
	PriorityGeneric Priority = 100
)

type Result

type Result struct {
	// Markdown is the normalized Markdown output.
	Markdown string
	// Title is the document title if the format provides one, else "".
	Title string
}

Result is the outcome of a successful conversion.

type StreamInfo

type StreamInfo struct {
	// MIMEType is the media type without parameters, e.g. "application/pdf".
	MIMEType string
	// Extension is lowercase with a leading dot, e.g. ".docx".
	Extension string
	// Charset is an IANA charset name, e.g. "utf-8" or "shift_jis".
	Charset string
	// Filename is the base name of the source, if known.
	Filename string
	// LocalPath is the path of the source file, if the input came from disk.
	LocalPath string
	// URL is the source URL, if the input came from the network.
	URL string
}

StreamInfo carries what is known or guessed about an input stream. The zero value means "nothing known"; every field is optional.

func (StreamInfo) Matches

func (s StreamInfo) Matches(exts []string, mimePrefixes []string) bool

Matches reports whether the stream's extension is one of exts (lowercase, with leading dot) or its MIME type starts with any of mimePrefixes. It is the standard building block for Converter.Accepts implementations.