goldmark

package module
v1.1.27 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 31, 2020 License: MIT Imports: 6 Imported by: 1,031

README

goldmark

A Markdown parser written in Go. Easy to extend, standards-compliant, well-structured.

goldmark is compliant with CommonMark 0.29.

Motivation

I needed a Markdown parser for Go that satisfies the following requirements:

  • Easy to extend.
    • Markdown is poor in document expressions compared to other light markup languages such as reStructuredText.
    • We have extensions to the Markdown syntax, e.g. PHP Markdown Extra, GitHub Flavored Markdown.
  • Standards-compliant.
    • Markdown has many dialects.
    • GitHub-Flavored Markdown is widely used and is based upon CommonMark, effectively mooting the question of whether or not CommonMark is an ideal specification.
      • CommonMark is complicated and hard to implement.
  • Well-structured.
    • AST-based; preserves source position of nodes.
  • Written in pure Go.

golang-commonmark may be a good choice, but it seems to be a copy of markdown-it.

blackfriday.v2 is a fast and widely-used implementation, but is not CommonMark-compliant and cannot be extended from outside of the package, since its AST uses structs instead of interfaces.

Furthermore, its behavior differs from other implementations in some cases, especially regarding lists: Deep nested lists don't output correctly #329, List block cannot have a second line #244, etc.

This behavior sometimes causes problems. If you migrate your Markdown text from GitHub to blackfriday-based wikis, many lists will immediately be broken.

As mentioned above, CommonMark is complicated and hard to implement, so Markdown parsers based on CommonMark are few and far between.

Features

  • Standards-compliant. goldmark is fully compliant with the latest CommonMark specification.
  • Extensible. Do you want to add a @username mention syntax to Markdown? You can easily do so in goldmark. You can add your AST nodes, parsers for block-level elements, parsers for inline-level elements, transformers for paragraphs, transformers for the whole AST structure, and renderers.
  • Performance. goldmark's performance is on par with that of cmark, the CommonMark reference implementation written in C.
  • Robust. goldmark is tested with go-fuzz, a fuzz testing tool.
  • Built-in extensions. goldmark ships with common extensions like tables, strikethrough, task lists, and definition lists.
  • Depends only on standard libraries.

Installation

$ go get github.com/yuin/goldmark

Usage

Import packages:

import (
    "bytes"
    "github.com/yuin/goldmark"
)

Convert Markdown documents with the CommonMark-compliant mode:

var buf bytes.Buffer
if err := goldmark.Convert(source, &buf); err != nil {
  panic(err)
}

With options

var buf bytes.Buffer
if err := goldmark.Convert(source, &buf, parser.WithContext(ctx)); err != nil {
  panic(err)
}
Functional option Type Description
parser.WithContext A parser.Context Context for the parsing phase.

Context options

Functional option Type Description
parser.WithIDs A parser.IDs IDs allows you to change logics that are related to element id(ex: Auto heading id generation).

Custom parser and renderer

import (
    "bytes"
    "github.com/yuin/goldmark"
    "github.com/yuin/goldmark/extension"
    "github.com/yuin/goldmark/parser"
    "github.com/yuin/goldmark/renderer/html"
)

md := goldmark.New(
          goldmark.WithExtensions(extension.GFM),
          goldmark.WithParserOptions(
              parser.WithAutoHeadingID(),
          ),
          goldmark.WithRendererOptions(
              html.WithHardWraps(),
              html.WithXHTML(),
          ),
      )
var buf bytes.Buffer
if err := md.Convert(source, &buf); err != nil {
    panic(err)
}
Functional option Type Description
goldmark.WithParser parser.Parser This option must be passed before goldmark.WithParserOptions and goldmark.WithExtensions
goldmark.WithRenderer renderer.Renderer This option must be passed before goldmark.WithRendererOptions and goldmark.WithExtensions
goldmark.WithParserOptions ...parser.Option
goldmark.WithRendererOptions ...renderer.Option
goldmark.WithExtensions ...goldmark.Extender

Parser and Renderer options

Parser options
Functional option Type Description
parser.WithBlockParsers A util.PrioritizedSlice whose elements are parser.BlockParser Parsers for parsing block level elements.
parser.WithInlineParsers A util.PrioritizedSlice whose elements are parser.InlineParser Parsers for parsing inline level elements.
parser.WithParagraphTransformers A util.PrioritizedSlice whose elements are parser.ParagraphTransformer Transformers for transforming paragraph nodes.
parser.WithASTTransformers A util.PrioritizedSlice whose elements are parser.ASTTransformer Transformers for transforming an AST.
parser.WithAutoHeadingID - Enables auto heading ids.
parser.WithAttribute - Enables custom attributes. Currently only headings supports attributes.
HTML Renderer options
Functional option Type Description
html.WithWriter html.Writer html.Writer for writing contents to an io.Writer.
html.WithHardWraps - Render newlines as <br>.
html.WithXHTML - Render as XHTML.
html.WithUnsafe - By default, goldmark does not render raw HTML or potentially dangerous links. With this option, goldmark renders such content as written.
Built-in extensions
Attributes

The parser.WithAttribute option allows you to define attributes on some elements.

Currently only headings support attributes.

Attributes are being discussed in the CommonMark forum. This syntax may possibly change in the future.

Headings
## heading ## {#id .className attrName=attrValue class="class1 class2"}

## heading {#id .className attrName=attrValue class="class1 class2"}

heading {#id .className attrName=attrValue}
============
Typographer extension

The Typographer extension translates plain ASCII punctuation characters into typographic-punctuation HTML entities.

Default substitutions are:

Punctuation Default entity
' &lsquo;, &rsquo;
" &ldquo;, &rdquo;
-- &ndash;
--- &mdash;
... &hellip;
<< &laquo;
>> &raquo;

You can override the defualt substitutions via extensions.WithTypographicSubstitutions:

markdown := goldmark.New(
    goldmark.WithExtensions(
        extension.NewTypographer(
            extension.WithTypographicSubstitutions(extension.TypographicSubstitutions{
                extension.LeftSingleQuote:  []byte("&sbquo;"),
                extension.RightSingleQuote: nil, // nil disables a substitution
            }),
        ),
    ),
)
Linkify extension

The Linkify extension implements Autolinks(extension), as defined in GitHub Flavored Markdown Spec.

Since the spec does not define details about URLs, there are numerous ambiguous cases.

You can override autolinking patterns via options.

Functional option Type Description
extension.WithLinkifyAllowedProtocols [][]byte List of allowed protocols such as [][]byte{ []byte("http:") }
extension.WithLinkifyURLRegexp *regexp.Regexp Regexp that defines URLs, including protocols
extension.WithLinkifyWWWRegexp *regexp.Regexp Regexp that defines URL starting with www.. This pattern corresponds to the extended www autolink
extension.WithLinkifyEmailRegexp *regexp.Regexp Regexp that defines email addresses`

Example, using xurls:

import "mvdan.cc/xurls/v2"

markdown := goldmark.New(
    goldmark.WithRendererOptions(
        html.WithXHTML(),
        html.WithUnsafe(),
    ),
    goldmark.WithExtensions(
        extension.NewLinkify(
            extension.WithLinkifyAllowedProtocols([][]byte{
                []byte("http:"),
                []byte("https:"),
            }),
            extension.WithLinkifyURLRegexp(
                xurls.Strict(),
            ),
        ),
    ),
)

Security

By default, goldmark does not render raw HTML or potentially-dangerous URLs. If you need to gain more control over untrusted contents, it is recommended that you use an HTML sanitizer such as bluemonday.

Benchmark

You can run this benchmark in the _benchmark directory.

against other golang libraries

blackfriday v2 seems to be the fastest, but as it is not CommonMark compliant, its performance cannot be directly compared to that of the CommonMark-compliant libraries.

goldmark, meanwhile, builds a clean, extensible AST structure, achieves full compliance with CommonMark, and consumes less memory, all while being reasonably fast.

goos: darwin
goarch: amd64
BenchmarkMarkdown/Blackfriday-v2-12                  326           3465240 ns/op         3298861 B/op      20047 allocs/op
BenchmarkMarkdown/GoldMark-12                        303           3927494 ns/op         2574809 B/op      13853 allocs/op
BenchmarkMarkdown/CommonMark-12                      244           4900853 ns/op         2753851 B/op      20527 allocs/op
BenchmarkMarkdown/Lute-12                            130           9195245 ns/op         9175030 B/op     123534 allocs/op
BenchmarkMarkdown/GoMarkdown-12                        9         113541994 ns/op         2187472 B/op      22173 allocs/op
against cmark (CommonMark reference implementation written in C)
----------- cmark -----------
file: _data.md
iteration: 50
average: 0.0037760639 sec
go run ./goldmark_benchmark.go
------- goldmark -------
file: _data.md
iteration: 50
average: 0.0040964230 sec

As you can see, goldmark's performance is on par with cmark's.

Extensions

goldmark internal(for extension developers)

Overview

goldmark's Markdown processing is outlined in the diagram below.

            <Markdown in []byte, parser.Context>
                           |
                           V
            +-------- parser.Parser ---------------------------
            | 1. Parse block elements into AST
            |   1. If a parsed block is a paragraph, apply 
            |      ast.ParagraphTransformer
            | 2. Traverse AST and parse blocks.
            |   1. Process delimiters(emphasis) at the end of
            |      block parsing
            | 3. Apply parser.ASTTransformers to AST
                           |
                           V
                      <ast.Node>
                           |
                           V
            +------- renderer.Renderer ------------------------
            | 1. Traverse AST and apply renderer.NodeRenderer
            |    corespond to the node type

                           |
                           V
                        <Output>
Parsing

Markdown documents are read through text.Reader interface.

AST nodes do not have concrete text. AST nodes have segment information of the documents, represented by text.Segment .

text.Segment has 3 attributes: Start, End, Padding .

(TBC)

TODO

See extension directory for examples of extensions.

Summary:

  1. Define AST Node as a struct in which ast.BaseBlock or ast.BaseInline is embedded.
  2. Write a parser that implements parser.BlockParser or parser.InlineParser.
  3. Write a renderer that implements renderer.NodeRenderer.
  4. Define your goldmark extension that implements goldmark.Extender.

Donation

BTC: 1NEDSyUmo4SMTDP83JJQSWi1MvQUGGNMZB

License

MIT

Author

Yusuke Inuzuka

Documentation

Overview

Package goldmark implements functions to convert markdown text to a desired format.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Convert 

func Convert(source []byte, w io.Writer, opts ...parser.ParseOption) error

Convert interprets a UTF-8 bytes source in Markdown and write rendered contents to a writer w.

func DefaultParser 

func DefaultParser() parser.Parser

DefaultParser returns a new Parser that is configured by default values.

func DefaultRenderer 

func DefaultRenderer() renderer.Renderer

DefaultRenderer returns a new Renderer that is configured by default values.

Types

type Extender 

type Extender interface {
	// Extend extends the Markdown.
	Extend(Markdown)
}

An Extender interface is used for extending Markdown.

type Markdown 

type Markdown interface {
	// Convert interprets a UTF-8 bytes source in Markdown and write rendered
	// contents to a writer w.
	Convert(source []byte, writer io.Writer, opts ...parser.ParseOption) error

	// Parser returns a Parser that will be used for conversion.
	Parser() parser.Parser

	// SetParser sets a Parser to this object.
	SetParser(parser.Parser)

	// Parser returns a Renderer that will be used for conversion.
	Renderer() renderer.Renderer

	// SetRenderer sets a Renderer to this object.
	SetRenderer(renderer.Renderer)
}

A Markdown interface offers functions to convert Markdown text to a desired format.

func New 

func New(options ...Option) Markdown

New returns a new Markdown with given options.

type Option 

type Option func(*markdown)

Option is a functional option type for Markdown objects.

func WithExtensions 

func WithExtensions(ext ...Extender) Option

WithExtensions adds extensions.

func WithParser 

func WithParser(p parser.Parser) Option

WithParser allows you to override the default parser.

func WithParserOptions 

func WithParserOptions(opts ...parser.Option) Option

WithParserOptions applies options for the parser.

func WithRenderer 

func WithRenderer(r renderer.Renderer) Option

WithRenderer allows you to override the default renderer.

func WithRendererOptions 

func WithRendererOptions(opts ...renderer.Option) Option

WithRendererOptions applies options for the renderer.

Source Files

View all Source files

Directories

Path Synopsis
_benchmark
cmark
Package ast defines AST nodes that represent markdown elements.
 Package ast defines AST nodes that represent markdown elements.
ast
Package ast defines AST nodes that represents extension's elements
 Package ast defines AST nodes that represents extension's elements
Package parser contains stuff that are related to parsing a Markdown text.
 Package parser contains stuff that are related to parsing a Markdown text.
Package renderer renders the given AST to certain formats.
 Package renderer renders the given AST to certain formats.
html
Package util provides utility functions for the goldmark.
 Package util provides utility functions for the goldmark.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.