markdown

Documentation

Overview

Package markdown turns an HTML response body into Markdown, so a crawl can store a clean text rendering of each page alongside the raw bytes. The heavy lifting is reused from yomi: extract.FromHTML isolates the article with go-readability and sanitises it, and mdconv.Convert renders that subtree to GitHub-Flavored Markdown and tidies the result (tables, strikethrough, entity decoding, dropped share widgets and duplicate captions, cleaned headings).

What this package adds on top is what a web-scale crawl needs and a reader of a single modern article does not: a Content-Type gate so only HTML is touched, and a charset transcode so a page served as GBK, Shift-JIS, or Latin-1 reaches the parser as UTF-8 instead of mojibake. yomi parses bodies as UTF-8 directly; across the open web that assumption does not hold.

Conversion is CPU work, a few milliseconds a page, so callers run it on the fetch worker pool rather than the single pack consumer that would otherwise serialise it.

Index

• func Convert(body []byte, contentType, pageURL string) (string, bool)
• func IsHTML(contentType string) bool

Functions

func Convert

func Convert(body []byte, contentType, pageURL string) (string, bool)

Convert renders the main content of an HTML body to Markdown. contentType is the response Content-Type (used both for the HTML check and to seed charset detection) and pageURL is the absolute URL the body came from (used to resolve relative links and images to absolute ones). It returns the Markdown and true on success, or "" and false when the body is not HTML, is empty, fails to parse, or yields no extractable article.

func IsHTML

func IsHTML(contentType string) bool

IsHTML reports whether a Content-Type names an HTML document. Conversion is only attempted for these; everything else (images, JSON, PDFs, plain text) is left alone.