md4agents

README

caddy-md4agents

A Caddy v2 HTTP middleware that serves a Markdown rendition of HTML pages when a client (typically an AI agent) negotiates for it. It implements Cloudflare's Markdown for Agents conventions on top of Caddy's static and dynamic handlers.

Design

Three things kept it small and fast:

1. Static-first. When root is set, requests resolve to disk: an author-written *.md wins, otherwise the matching *.html is converted on first hit and written to a sidecar cache.
2. Lazy in-memory + disk write-through. First request to a page pays the conversion cost (~ms); subsequent requests serve from a sized LRU. Disk sidecars survive restarts and worker recycling.
3. Stat-based invalidation. Every request stats the source HTML and keys the cache on path | mtime | size. Edits invalidate automatically, no watcher required.

A capture-and-convert fallback handles dynamic upstreams (reverse proxy, templates, anything that doesn't resolve to a file on disk).

Content negotiation

A request is served Markdown when any of these is true (in order):

Trigger	Example
URL suffix	GET /docs/page.md
Query param	GET /docs/page?format=md
Accept header	Accept: text/markdown (with q-value handling vs text/html)

The first two are stripped before the inner handler sees the request, so the upstream still resolves the underlying HTML resource.

Build

This is a Caddy plugin, so it needs to be compiled into a Caddy binary with xcaddy:

xcaddy build --with github.com/mhupfauer/caddy-md4agents

Docker

A pre-built image follows the upstream caddy release cadence and is rebuilt daily so it picks up Caddy base-image updates automatically:

docker pull ghcr.io/mhupfauer/caddy-md4agents:latest

Tags:

Tag	Pointer
latest	Last successful build of main
caddy-<version>	Built against that upstream Caddy release
sha-<short-sha>	Built from that exact commit

Run with a Caddyfile mounted at the standard path:

docker run --rm -p 80:80 -p 443:443 -p 443:443/udp \
  -v $PWD/Caddyfile:/etc/caddy/Caddyfile:ro \
  -v $PWD/site:/srv/site:ro \
  ghcr.io/mhupfauer/caddy-md4agents:latest

Or build locally against the current checkout:

docker build -t caddy-md4agents:dev .

Caddyfile

Minimal — static site, defaults everywhere:

example.com {
    root * /var/www/site
    markdown_for_agents {
        root /var/www/site
    }
    file_server
}

Caddyfile gotcha: do NOT write markdown_for_agents /var/www/site. Any first argument starting with / is consumed by Caddy as a path matcher, not as a positional argument — so the directive would only fire for requests to literally /var/www/site, with an empty root. Always use the block form to set root.

Full options:

example.com {
    root * /var/www/site

    markdown_for_agents {
        root                /var/www/site
        cache_dir           /var/cache/md4agents
        url_suffix          .md
        query_param         format
        cache_size          8192
        cache_ttl           24h
        max_body_bytes      8388608
        convert_timeout     5s
        pregenerate
        allow_authenticated     # opt-in: cache responses for authenticated requests
        main_selector       article
        strip_tags          script style noscript nav footer aside
        strip_selectors     .ads "#cookie-banner"   # quote id selectors — # is a Caddyfile comment
    }

    file_server
}

Reverse-proxy mode (no root → uses capture path with the LRU only):

example.com {
    markdown_for_agents {
        strip_selectors nav footer .site-chrome
        main_selector   main
    }
    reverse_proxy backend:8080
}

Configuration reference

Field	Default	Notes
root	—	Static file root. Enables the static-first path.
cache_dir	<caddy-data-dir>/md4agents/<hash>	Disk write-through cache; lives outside root.
url_suffix	.md	URL suffix that requests Markdown. Empty disables.
query_param	format	Query param checked for md/markdown. Empty disables.
cache_size	4096	In-memory LRU entry count.
cache_bytes	268435456 (256 MiB)	Total in-memory cache byte budget.
cache_entry_bytes	1048576 (1 MiB)	Per-entry cap; oversized entries are rejected.
cache_ttl	15m	TTL for in-memory entries; disk uses mtime. Use cache_ttl 0 or cache_ttl never to disable TTL eviction entirely (operator must purge manually).
max_body_bytes	4194304 (4 MiB)	Source HTML size cap (both static disk reads and dynamic captures).
convert_timeout	5s	Per-conversion timeout; on exceed, returns 503.
max_concurrent	max(4, NumCPU)	Conversion semaphore — bounds CPU/goroutine usage.
pregenerate	false	Walk root on startup and warm the cache.
janitor_interval	0 (off)	Periodic orphan-sidecar cleanup interval.
allow_authenticated	false	If true, cache responses to requests carrying Authorization/Cookie.
main_selector	—	If set, only this element's subtree is converted.
strip_tags	script style noscript iframe svg	Tags removed entirely from output.
strip_selectors	—	Simple tag, .class, #id selectors removed pre-conversion.

Cache safety

The shared cache is, well, shared, so a few hard rules apply:

• Only GET and HEAD are cacheable.
• Requests with Authorization or Cookie headers bypass the cache by default. Set allow_authenticated only when upstream content is not user-specific.
• Upstream responses carrying Set-Cookie, Cache-Control: private, Cache-Control: no-store, or a non-trivial Vary (anything beyond Accept-Encoding) are converted and served once but never cached.
• The dynamic-path cache key is path + ?query, so /api/p?id=1 and /api/p?id=2 do not collide.
• The in-memory cache is bounded by both entry count and total bytes; per-entry oversized responses are rejected outright.

Authorization placement

The static-first path serves matched files directly without calling the next handler. That means any basicauth, forward_auth, jwtauth, or similar middleware that comes after markdown_for_agents in the Caddyfile chain will not run for markdown responses.

In Caddy, the directive runs immediately before file_server, so the default placement of auth middleware (which is before file_server) is safe. If you place auth in a handle block that wraps both this module and file_server, ordering is preserved and auth still runs first.

If your config puts auth after file_server (unusual), or applies path matchers that only target *.html, ensure the matcher also covers the URL suffix you've configured for markdown negotiation — e.g. path *.html *.md.

Personalization beyond cookies

The Authorization and Cookie headers bypass the shared cache by default. If your application personalizes responses on other signals — mTLS, X-Forwarded-User, IP-based ACL — those are not considered cache-safe automatically. Either set Cache-Control: private on the upstream response (this module will honor it) or run separate module instances per personalization dimension.

File system semantics

• Paths are canonicalized at provision time and on every request via filepath.EvalSymlinks. If a request's target resolves outside root (e.g. via a symlink in the tree), the module returns 404 and refuses to fall through, preventing a downstream file_server + dynamic-path conversion from leaking the file.
• Source HTML is opened once and stat'd from the file descriptor, closing the TOCTOU window between stat and read.
• The default cache_dir lives in caddy.AppDataDir()/md4agents/<hash> outside root. The segment name is a SHA-256 prefix only — the original path's basename is never written into the data dir.
• HEAD responses include all headers (including Content-Length) but no body, per RFC 9110 §15.3.
• Upstream headers from the dynamic path are whitelist-forwarded: Cache-Control, Expires, Last-Modified, Content-Language, Content-Security-Policy, Strict-Transport-Security, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy. Everything else (notably Set-Cookie, Server, X-Powered-By) is dropped.

Cache layout

<root>/                          # original site
  docs/about.html
  docs/about.md                  # OPTIONAL: author-written, served verbatim

<cache_dir>/                     # generated; safe to delete
  docs/about.html.md             # sidecar for docs/about.html

The .html.md double-extension keeps generated artifacts impossible-by- convention to confuse with author Markdown.

Response headers

• Content-Type: text/markdown; charset=utf-8
• ETag: "<sha256[:16]>" (strong; supports If-None-Match → 304)
• Vary: Accept

Performance notes

• The HTML→Markdown converter is built once at provision time and is goroutine-safe (per html-to-markdown/v2 contract).
• A single-flight pattern collapses concurrent identical conversions into one execution, preventing thundering-herd cost on a cold cache.
• The hot path on a warmed cache is a stat() + map lookup + write — no parsing, no allocations beyond the response itself.

Security

Two scanners run on every push and weekly on a cron:

• CodeQL (GitHub native) — Go SAST + dependency scanning. Findings appear under the repo's Security → Code scanning tab.
• Snyk — SAST (Snyk Code → SARIF → GitHub Code Scanning), SCA (snyk monitor --all-projects), Infrastructure-as-Code, and Container scans against the built Docker image.

Snyk needs a SNYK_TOKEN repo secret (free Snyk account → API token → GitHub Settings → Secrets → SNYK_TOKEN). When the token is missing the workflow short-circuits in its preflight job so PRs and the initial setup window don't fail noisily.

Keeping the build patched

The published binary's Go stdlib is whatever the toolchain directive in go.mod pins (currently go1.26.5) — that's the version Snyk reads from the embedded buildinfo, so stdlib CVEs are cleared by bumping it to the latest 1.26.x patch, not by changing the go 1.26.0 minimum. The Dockerfile's build stage tracks the matching 1.26-alpine floating tag so the image stays in lockstep, and its runtime stage runs apk upgrade for base-image packages (c-ares, curl, openssl) to pull Alpine's patched builds ahead of an upstream caddy rebuild; base-image-refresh.yml then picks up the upstream fix automatically.

Caddy itself is a direct dependency, so CVEs against it are cleared by bumping github.com/caddyserver/caddy/v2 in go.mod. When the upstream fix has only landed on master (no tagged release yet), pin to the fix commit with a pseudo-version — go get github.com/caddyserver/caddy/v2@<commit> — and switch back to the release once it ships. xcaddy is passed the version resolved from go.mod (go list -m …) in both the CI build and the Dockerfile, so the runtime binary's embedded buildinfo matches the pin; without that, xcaddy would silently build the latest release and ship the vulnerable Caddy.

Vulnerabilities can also be reported privately via GitHub Security Advisories.

License

MIT

Documentation

Overview

Package md4agents implements a Caddy v2 HTTP middleware that serves a Markdown rendition of HTML pages when an agent (or any client) negotiates for it. See the README for design notes and the Cloudflare "Markdown for Agents" RFC this is modelled after.

Index

• type MarkdownForAgents
  • func (MarkdownForAgents) CaddyModule() caddy.ModuleInfo
  • func (m *MarkdownForAgents) Cleanup() error
  • func (m *MarkdownForAgents) Provision(ctx caddy.Context) error
  • func (m *MarkdownForAgents) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error
  • func (m *MarkdownForAgents) UnmarshalCaddyfile(d *caddyfile.Dispenser) error
  • func (m *MarkdownForAgents) Validate() error

Types

type MarkdownForAgents

type MarkdownForAgents struct {
	// Root is the static file root to resolve requests against. When set,
	// the static-first path is enabled: author `.md` files are served
	// verbatim, generated artifacts are written to CacheDir and reused.
	// When unset, the module only acts as a streaming converter in front
	// of dynamic handlers (reverse_proxy, templates, etc.).
	Root string `json:"root,omitempty"`

	// CacheDir holds the on-disk write-through cache for generated
	// markdown. Defaults to caddy.AppDataDir()/md4agents/<hash> so it
	// can never be served by file_server.
	CacheDir string `json:"cache_dir,omitempty"`

	// URLSuffix appended to a path to explicitly request markdown
	// (e.g. ".md"). Empty disables URL-suffix negotiation.
	URLSuffix string `json:"url_suffix,omitempty"`

	// QueryParam to check for markdown opt-in (e.g. "format"). The value
	// must be "md" or "markdown". Empty disables query-param negotiation.
	QueryParam string `json:"query_param,omitempty"`

	// StripTags lists HTML tag names removed entirely (with their
	// subtree) before conversion. Default:
	// `script style noscript iframe svg`. Useful for stripping
	// inline analytics, embedded videos, decorative SVGs, etc.
	StripTags []string `json:"strip_tags,omitempty"`

	// StripSelectors lists simple selectors removed before
	// conversion. Supported forms: bare tag (`nav`), class
	// (`.ads`), or id (`#cookie-banner`). Quote id selectors in
	// the Caddyfile — `#` is a comment marker there.
	StripSelectors []string `json:"strip_selectors,omitempty"`

	// MainSelector, if set, restricts conversion to the subtree
	// rooted at the first element matching this simple selector
	// (e.g. `article`, `main`, `.post-body`). Everything outside
	// is discarded, which is the cleanest way to strip site
	// chrome on theme-heavy pages.
	MainSelector string `json:"main_selector,omitempty"`

	// CacheSize bounds the number of cached markdown responses held in
	// memory. 0 → 4096.
	CacheSize int `json:"cache_size,omitempty"`

	// CacheBytes is the total in-memory cache byte budget. 0 → 256 MiB.
	CacheBytes int64 `json:"cache_bytes,omitempty"`

	// CacheEntryBytes caps the size of a single cached entry, both to
	// reject pathological pages and to make the byte budget meaningful.
	// 0 → 1 MiB.
	CacheEntryBytes int64 `json:"cache_entry_bytes,omitempty"`

	// CacheTTL bounds how long an in-memory cache entry is reused.
	//   0  → use default (15m)
	//   <0 → never expire (use with care; the on-disk cache already
	//         mtime-invalidates, so this only matters for the dynamic
	//         path)
	CacheTTL caddy.Duration `json:"cache_ttl,omitempty"`

	// MaxBodyBytes limits the size of an HTML body we'll attempt to
	// convert (both static disk reads and dynamic captures). 0 → 4 MiB.
	MaxBodyBytes int64 `json:"max_body_bytes,omitempty"`

	// ConvertTimeout bounds an individual HTML→MD conversion. 0 → 5s.
	ConvertTimeout caddy.Duration `json:"convert_timeout,omitempty"`

	// MaxConcurrent caps the number of conversions that can run at once.
	// 0 → max(4, NumCPU). New requests wait on a buffered semaphore;
	// hitting the ConvertTimeout while waiting returns 503.
	MaxConcurrent int `json:"max_concurrent,omitempty"`

	// Pregenerate, when true and Root is set, walks the root at startup
	// and converts every .html file ahead of the first request.
	Pregenerate bool `json:"pregenerate,omitempty"`

	// AllowAuthenticated, when true, allows caching responses for
	// requests carrying Authorization or Cookie headers. Default false —
	// any such request bypasses the shared cache to avoid serving one
	// user's markdown to another.
	AllowAuthenticated bool `json:"allow_authenticated,omitempty"`

	// JanitorInterval, when >0 and Root is set, runs a periodic cleanup
	// of orphaned sidecar files whose source HTML no longer exists.
	// 0 → off (the lazy mtime check is enough for correctness).
	JanitorInterval caddy.Duration `json:"janitor_interval,omitempty"`
	// contains filtered or unexported fields
}

MarkdownForAgents serves a Markdown rendition of HTML pages when a client — typically an AI agent — negotiates for it. It implements Cloudflare's "Markdown for Agents" convention on top of Caddy's static and dynamic handlers, with caching, content negotiation, and HTML sanitization built in.

## Why this matters

Modern AI agents (Claude, ChatGPT, Perplexity, crawler bots) waste tokens parsing HTML chrome — navigation, scripts, cookie banners, analytics — before reaching the content. Serving the same URL as Markdown gives them roughly 5–10× more useful content per token and measurably improves answer quality on long documents. Same URL, same auth, just `Accept: text/markdown` (or a `.md` suffix).

## Content negotiation

A request is served Markdown when any of these is true:

Trigger | Example -------------|-------- URL suffix | `GET /docs/page.md` Query param | `GET /docs/page?format=md` Accept hdr | `Accept: text/markdown` (q-value aware vs `text/html`)

The first two are stripped before the inner handler sees the request, so the upstream still resolves the underlying HTML.

## Quick start (static site)

```caddy

example.com {
    root * /var/www/site
    markdown_for_agents {
        root /var/www/site
    }
    file_server
}

```

Caddyfile note: always use the block form to set `root`. A bare `markdown_for_agents /var/www/site` would be parsed by Caddy as a path matcher (`/var/www/site`), not as a positional argument to the directive.

Author-written `*.md` files win over generated ones; generated artifacts are written to a sidecar cache (`/var/cache/md4agents` by default) and reused on every subsequent request. Edits to source HTML invalidate cache entries automatically (mtime + size stat) — no watcher required.

## Reverse-proxy mode

Omit `root` and the module becomes a streaming converter in front of any dynamic upstream:

```caddy

example.com {
    markdown_for_agents {
        main_selector   article
        strip_selectors nav footer .ads
    }
    reverse_proxy backend:8080
}

```

## Cache safety

Only `GET` and `HEAD` are cacheable. Requests carrying `Authorization` or `Cookie` headers bypass the shared cache by default; upstream responses with `Set-Cookie`, `Cache-Control: private/no-store`, or a non-trivial `Vary` are converted and served once but never cached.

## More

Full documentation, performance notes, and security guidance live at https://github.com/mhupfauer/caddy-md4agents.

All durations and sizes are zero-value safe: any unset field falls back to a documented default during provisioning.

func (MarkdownForAgents) CaddyModule

func (MarkdownForAgents) CaddyModule() caddy.ModuleInfo

func (*MarkdownForAgents) Cleanup

func (m *MarkdownForAgents) Cleanup() error

func (*MarkdownForAgents) Provision

func (m *MarkdownForAgents) Provision(ctx caddy.Context) error

func (*MarkdownForAgents) ServeHTTP

func (m *MarkdownForAgents) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error

ServeHTTP dispatches to the static-first path when Root is configured and the request resolves to an HTML file on disk, falling back to the dynamic capture path otherwise.

func (*MarkdownForAgents) UnmarshalCaddyfile

func (m *MarkdownForAgents) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

func (*MarkdownForAgents) Validate

func (m *MarkdownForAgents) Validate() error