atomdown

README

Atomdown

Atomdown adds persistent block IDs, groups, and extensible metadata to Markdown. Atomdown documents remain valid CommonMark.

Use Atomdown when people and AI agents need to address and review individual Markdown blocks. Visible Markdown remains the source of truth.

Atomdown uses XML-shaped directives inside HTML comments:

<!-- <atomdown version="1"/> -->

<!-- <atom id="4P8W2H6K" slug="launch-claim"/> -->

The product launched in March.

Normal Markdown tools treat each directive as an invisible comment. Atomdown tools read the same directive as block metadata.

Why Atomdown

Many block editors store documents as application-owned JSON. Atomdown keeps Markdown canonical and adds a small annotation layer.

Atomdown supports:

• Stable Markdown block IDs for links, comments, and review workflows.
• Ordered groups of addressable blocks.
• Application metadata that core tools preserve but do not interpret.
• Lossless token output for editors and embedded applications.

Core defines only version, id, and slug. Applications extend Atomdown with their own XML attributes:

<!-- <atom id="4P8W2H6K" acme-owner="research"/> -->

The acme-owner attribute is an application-defined example. It is not part of Atomdown Core. Core tools preserve it and do not interpret it.

Atomdown Core 1 is an early specification. Use the current syntax and conformance corpus for experiments and review.

Design

• Documents remain valid CommonMark.
• Directives use XML 1.0 syntax inside HTML comments.
• The normalized metadata model uses XML Schema Definition (XSD) 1.0.
• Unknown XML attributes extend Atomdown without changing Core.
• Core defines identity, grouping, order, and preservation rules.

The implementation is a pure-Go library and command-line interface (CLI). It uses encoding/xml and the pure-Go goldmark parser. It does not use CGO.

Install

go install github.com/srhopkins/atomdown/cmd/atomdown@latest

Prebuilt binaries for macOS and Linux are on the releases page.

CLI

Run the CLI from the repository:

go run ./cmd/atomdown lint testdata/example.md
go run ./cmd/atomdown parse testdata/example.md
go run ./cmd/atomdown tokens testdata/example.md
go run ./cmd/atomdown xml testdata/example.md
go run ./cmd/atomdown strip testdata/example.md
go run ./cmd/atomdown materialize testdata/example.md
go run ./cmd/atomdown id

Each file command accepts one file. Use - or omit the file to read standard input.

• lint checks syntax, IDs, block associations, and groups.
• lint --strict also reports unmarked top-level blocks. Default lint permits mixed documents so teams can adopt Atomdown in stages.
• parse writes the semantic document model as JSON.
• tokens writes a lossless stream of Markdown, whitespace, and Atomdown directives.
• xml writes the normalized XML metadata model.
• strip removes Atomdown directives and writes plain Markdown.
• materialize adds a new atom marker before each unmarked top-level block. Use materialize -w FILE to update the file in place.
• id creates an eight-character Crockford Base32 ID.

Go library

Parse and validate a document:

document := atomdown.Parse(source)
if document.HasErrors() {
    // Reject or repair the source.
}

Register an extension in an embedded application:

processor := atomdown.NewProcessor(myExtension)
document, err := processor.Process(ctx, source)

Extensions implement atomdown.Extension. Extensions run in registration order and can add metadata or diagnostics.

Atomdown does not use Go's runtime plugin package. This choice keeps the library portable and compatible with CGO_ENABLED=0.

Standard and tests

• SPEC.md defines Atomdown Core 1.
• schema/atomdown-1.xsd defines the normalized XML model.
• testdata/ provides valid, mixed, malformed, and exact golden files.
• conformance/ provides a language-neutral test suite. Second implementations run it without Go.
• llms.txt gives AI agents a short guide to the repository.

Contributing

Read CONTRIBUTING.md, then open a Bug report or Proposal. Coding agents must also read AGENTS.md.

License

Atomdown uses the MIT License.

Documentation

Overview

Package atomdown parses and validates persistent block IDs, ordered groups, and extensible XML-shaped metadata embedded in ordinary Markdown comments. Atomdown documents remain valid CommonMark and visible Markdown remains the source of truth.

Index

• func Materialize(source []byte) ([]byte, error)
• func NewID() (string, error)
• func NormalizedXML(document Document) ([]byte, error)
• func Strip(source []byte) []byte
• type Atom
• type AtomGroup
• type Attribute
• type Diagnostic
• type DirectiveToken
• type Document
  • func Parse(source []byte) Document
  • func (d Document) HasErrors() bool
• type Extension
• type ExtensionFunc
  • func (extension ExtensionFunc) Name() string
  • func (extension ExtensionFunc) Transform(ctx context.Context, source []byte, document *Document) error
• type Position
• type Processor
  • func NewProcessor(extensions ...Extension) *Processor
  • func (processor *Processor) Process(ctx context.Context, source []byte) (Document, error)
• type Range
• type Severity
• type Token
• type TokenKind
• type TokenStream
  • func Tokenize(source []byte) TokenStream

Functions

func Materialize

func Materialize(source []byte) ([]byte, error)

Materialize inserts an explicit atom marker before every implicit atom. Existing source bytes and explicit directives remain unchanged.

func NewID

func NewID() (string, error)

NewID returns an eight-character Crockford Base32 identifier with 40 random bits.

func NormalizedXML

func NormalizedXML(document Document) ([]byte, error)

NormalizedXML returns the Atomdown metadata model as a conventional XML document. Markdown content remains in the source file and is represented by atom IDs.

func Strip

func Strip(source []byte) []byte

Strip returns the pure Markdown projection. It removes only recognized, valid Atomdown directive lines and preserves all non-directive content.

Types

type Atom

type Atom struct {
	ID         string      `json:"id,omitempty"`
	Slug       string      `json:"slug,omitempty"`
	Attributes []Attribute `json:"attributes,omitempty"`
	Marker     *Range      `json:"marker,omitempty"`
	Content    Range       `json:"content"`
	NodeType   string      `json:"nodeType"`
	Text       string      `json:"text"`
	Implicit   bool        `json:"implicit"`
	GroupID    string      `json:"groupId,omitempty"`
}

Atom is one explicit or implicit top-level Markdown unit.

type AtomGroup

type AtomGroup struct {
	ID         string      `json:"id"`
	Slug       string      `json:"slug,omitempty"`
	Attributes []Attribute `json:"attributes,omitempty"`
	Marker     Range       `json:"marker"`
	EndMarker  *Range      `json:"endMarker,omitempty"`
	AtomIDs    []string    `json:"atomIds,omitempty"`
}

AtomGroup is a contiguous, ordered collection of explicit atoms.

type Attribute

type Attribute struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

Attribute is an XML attribute not defined by Atomdown Core.

type Diagnostic

type Diagnostic struct {
	Code     string   `json:"code"`
	Severity Severity `json:"severity"`
	Message  string   `json:"message"`
	Position Position `json:"position"`
	Fix      string   `json:"fix,omitempty"`
}

Diagnostic describes a syntax or semantic defect.

type DirectiveToken

type DirectiveToken struct {
	Element    string      `json:"element"`
	Operation  string      `json:"operation"`
	ID         string      `json:"id,omitempty"`
	Slug       string      `json:"slug,omitempty"`
	Version    string      `json:"version,omitempty"`
	Attributes []Attribute `json:"attributes,omitempty"`
}

DirectiveToken is the public XML-shaped view of an Atomdown directive.

type Document

type Document struct {
	Declared    bool         `json:"declared"`
	Version     string       `json:"version,omitempty"`
	Attributes  []Attribute  `json:"attributes,omitempty"`
	Atoms       []Atom       `json:"atoms"`
	Groups      []AtomGroup  `json:"groups,omitempty"`
	Diagnostics []Diagnostic `json:"diagnostics,omitempty"`
}

Document is the parsed Atomdown view of a Markdown source file.

func Parse

func Parse(source []byte) Document

Parse builds an Atomdown document model from Markdown source.

func (Document) HasErrors

func (d Document) HasErrors() bool

HasErrors reports whether the document contains an error diagnostic.

type Extension

type Extension interface {
	Name() string
	Transform(context.Context, []byte, *Document) error
}

Extension adds application-specific behavior to the parsed document model. Extensions run in registration order and may decorate the document, preserve private state in attributes, or append extension-specific diagnostics.

type ExtensionFunc

type ExtensionFunc struct {
	ExtensionName string
	TransformFunc func(context.Context, []byte, *Document) error
}

ExtensionFunc adapts a function into an Extension.

func (ExtensionFunc) Name

func (extension ExtensionFunc) Name() string

Name returns the extension's stable name.

func (ExtensionFunc) Transform

func (extension ExtensionFunc) Transform(ctx context.Context, source []byte, document *Document) error

Transform invokes the wrapped transform function.

type Position

type Position struct {
	Offset int `json:"offset"`
	Line   int `json:"line"`
	Column int `json:"column"`
}

Position identifies a byte offset and its one-based line and column.

type Processor

type Processor struct {
	// contains filtered or unexported fields
}

Processor parses Atomdown and applies embedded extensions.

func NewProcessor

func NewProcessor(extensions ...Extension) *Processor

NewProcessor creates an embedded Atomdown processor.

func (*Processor) Process

func (processor *Processor) Process(ctx context.Context, source []byte) (Document, error)

Process parses source and runs each extension in registration order.

type Range

type Range struct {
	Start Position `json:"start"`
	End   Position `json:"end"`
}

Range identifies a half-open source range.

type Severity

type Severity string

Severity describes the effect of a diagnostic.

const (
	SeverityError   Severity = "error"
	SeverityWarning Severity = "warning"
)

type Token

type Token struct {
	Kind      TokenKind       `json:"kind"`
	Range     Range           `json:"range"`
	Raw       string          `json:"raw"`
	NodeType  string          `json:"nodeType,omitempty"`
	Directive *DirectiveToken `json:"directive,omitempty"`
}

Token is one lossless Markdown, whitespace, or Atomdown source segment. Concatenating Raw for every token reconstructs the original document.

type TokenKind

type TokenKind string

TokenKind identifies one lossless source segment.

const (
	TokenMarkdown   TokenKind = "markdown"
	TokenDirective  TokenKind = "atomdown-directive"
	TokenWhitespace TokenKind = "whitespace"
)

type TokenStream

type TokenStream struct {
	Tokens      []Token      `json:"tokens"`
	Diagnostics []Diagnostic `json:"diagnostics,omitempty"`
}

TokenStream contains a lossless ordered source stream and lexical diagnostics.

func Tokenize

func Tokenize(source []byte) TokenStream

Tokenize returns a lossless ordered stream containing Markdown blocks, Atomdown directives, and interstitial whitespace.