htmlstream

README

htmlstream

Go library for robust streaming html/xhtml/xml parsing and transformation, suitable for untrusted/malformed input.

Features

• Token-level context-aware streaming HTML/XHTML/XML parser.
• Support for emitting polyglot HTML/XHTML/XML output.
• Hybrid HTML/XHTML parse mode (tolerating XMLisms in HTML).
• DOM-like helpers for HTML parsing.
• Optional simplified SAX-style wrapper.
• Byte-identical decode/encode roundtrip by default.
• Good tolerance for malformed input.
• Streaming transformations with backtracking.
• Fast and allocation-efficient.
• Works with plain XML too.
• Good test coverage and fuzzing.
• Parser can be used with stdlib encoding/xml.

Packages

The htmlstream package reads and transforms HTML/XHTML as a context-aware stream of tokens. By default, it is byte-exact unless modified, even for malformed input. It is safe to use with untrusted content without using unbounded memory or crashing. It supports multiple modes, including a universal mode which is able to parse HTML while tolerating XMLisms (or parsing it as XML if it has an explicit <?xml declaration and xmlns) (it is not intended for arbitrary XML, there's a specific mode for that). It will automatically balance unclosed or misnested HTML5 tags almost exactly the same as browsers (some cases are slightly different since it works in a single pass without building the full tree). There are some helpers for matching elements.

The htmlstream package is also capable of transforming XML, but provides limited XML-specific functionality (e.g., no doctype processing, very limited namespace awareness, no checks for well-formedness, no check for XML name validity). Notably, it does not currently support custom entities declared in the doctype. This is intended to be used for doing minimally intrusive modifications to XML documents, or efficiently extracting content from them.

The htmlutil package implements utility functions and wrappers for working with HTML content, including some DOM-like helpers for getting node text, sniffing character encoding, parsing attributes, and handling URLs.

The polyglot package converts HTML to polyglot XHTML/HTML4/HTML5 markup, parseable by XML parsers in almost all cases (warning if not). It does its best if the input is malformed, and has an optional strict mode. It emulates most of the HTML5 "adoption agency" algorithm and error recovery, so the final output ends up more or less equivalent to how a browser would parse it. There are hooks to allow a subset of transformations to be made in a single pass. There is also a CLI version.

The xmlutil package provides XML-specific helpers, including XML-style charset detection and declaration parsing. It is compatible with the XML spec for most documents, but does not strictly adhere to it and is generally a bit more lenient (since this library is primarily designed for the HTML spec).

The sax package provides a simplified SAX-style wrapper for parsing HTML/XHTML/XML. It also provides an encoding/xml.TokenReader adapter so it can be used with the stdlib encoding/xml.Decoder or other libraries. It supports all htmlstream reader modes (universal, html, xml), and behaves accordingly. It also supports reading a partial fragment.

The ascii package has ASCII string/bytes helpers, exported since I'm sick of copying and pasting them around.

Robustness

This library is optimized for allocations, and, to a slightly lesser degree, throughput. There are zero-copy versions of most methods. It is designed to not crash even on malformed inputs. The htmlstream, polyglot, and sax packages have configurable bounded resource requirements, even for untrusted input.

This library is also designed to pass through malformed input as faithfully as possible (byte-identical in most cases). If content cannot be represented or parsed appropriately, it will emit warnings/errors, which can be handled by the caller. Most of the errors are sticky to reduce the amount of error handling boilerplate.

This library is designed to work on UTF-8 or ASCII-compatible input, but has helpers for sniffing/parsing the charset which can be used with x/text/encoding to convert it on-the-fly.

This library is well-tested, fuzzed, and used in quite a few of my projects. Although it is relatively new, a lot of the core code and design is based on previous things I've written. The high-level htmlutil/xmlutil/sax packages are a little bit less battle-tested, but should still be stable.

I haven't released a v1 yet since I want to be able to make minor changes to the API, but it can be considered stable enough for production use regardless.

LLM Usage

I used Claude to write a lot of the actual code and do the fuzzing, but I designed the API myself, and I reviewed (and tweaked) a lot of the code it wrote (and corrected a few misconceptions it had). If anything, this significantly increased the code quality and robustness since the LLM wrote a lot of fuzzers, and spent more time on the rarer edge cases than I would have (e.g., I wasn't originally planning on matching the HTML5 spec for adopting misnested or unclosed elements, but it spent the time to handle that correctly). I spent almost a week's worth of time iterating upon this library with Claude, even after it finished my original specification. Furthermore, I've already done a lot of other work in this area (kepubify, ebook tooling, multiple scrapers, some private projects), and I had Claude use it as a starting point. Note that unlike my other projects, I let Claude write some of the doc comments and examples given the large API surface (I understand the API and semantics since I designed it, so I'm not really motivated to spend too much time writing extensive docs, but it can be a bit daunting without them).

Documentation

Overview

Package htmlstream reads and transforms HTML/XHTML as a context-aware stream of tokens.

By default, it is byte-exact unless modified, even for malformed input. It is safe to use with untrusted content without using unbounded memory or crashing. It supports multiple modes, including a universal mode (Universal) which is able to parse HTML while tolerating XMLisms (or parsing it as XML if it has an explicit <?xml declaration and xmlns) (it is not intended for arbitrary XML, there's a specific mode for that, XML). It will automatically balance unclosed or misnested HTML5 tags almost exactly the same as browsers (some cases are slightly different since it works in a single pass without building the full tree). There are some helpers for matching elements.

It is also capable of transforming XML, but provides limited XML-specific functionality (e.g., no doctype processing, very limited namespace awareness, no checks for well-formedness, no check for XML name validity). Notably, it does not currently support custom entities declared in the doctype. This is intended to be used for doing minimally intrusive modifications to XML documents, or efficiently extracting content from them.

Unlike golang.org/x/net/html, it does not build a tree, and unlike encoding/xml, it accepts malformed markup and preserves the exact input bytes: every byte of the input belongs to exactly one token, so a transformation which makes no changes outputs the input unchanged.

Reading

A Reader (see NewReader) reads the input one Token at a time: Reader.Next advances to the next token and Reader.Token returns it, or Reader.All iterates over the tokens. Afterwards, Reader.Err returns the error which stopped it, if any. A token has a Kind, its bytes in the input (Token.Raw), and depending on its kind, a name and attributes (see Token.Name and Token.Attr) or text, decoded on request (see Token.AppendText and Attr.AppendValue). It is valid until the next token is read (see Token.Clone and TokenBuffer for copies).

Reader.Find advances to the next start tag matching a Matcher (e.g., Tag, AttrEquals and HasClass). At a start tag, Reader.Skip advances to the element's end tag, and Reader.Children iterates over the tokens inside the element.

github.com/pgaskin/htmlstream/sax delivers balanced element events (and encoding/xml tokens) instead of tokens, github.com/pgaskin/htmlstream/htmlutil reads the text of elements and classifies them, and github.com/pgaskin/htmlstream/polyglot converts HTML to polyglot markup. For a tree, use golang.org/x/net/html.

Transforming

A Transformer (see NewTransformer) reads tokens like a Reader and writes each one to an io.Writer when it advances past it, with the edits made while it was the current token (e.g., Transformer.SetAttr, Transformer.Remove, Transformer.Replace and Transformer.Wrap). Tokens without edits are written unchanged. Output can be held back until the tokens which follow are known (see Transformer.Hold). Transformer.Close writes the rest of the input and flushes the output. An edit which doesn't apply to the current token returns an error wrapping ErrInvalidOp, and the Transformer continues.

The Append functions write escaped text and attribute values, tags and comments which HTML and XML parsers read the same way, and CDATA sections and processing instructions for XML, for markup inserted by a Transformer. Text and values passed on their own are a string or a []byte, and tag names and attributes (name and value pairs) are strings. ValidXMLName and ValidXMLChars check names and text for XML, and AppendXMLChars replaces the characters XML doesn't allow. AdjustSVGTagName, AdjustSVGAttrName and AdjustMathMLAttrName give SVG and MathML names the case HTML parsers give them.

Modes

A Reader reads HTML by default (HTML mode). XML mode reads XML, including XHTML, and Universal mode reads HTML or XHTML of unknown type, choosing the rules of each document (see Reader.SetMode).

Browsers choose the parser by the media type: HTML for text/html, whatever the markup looks like, and XML for application/xhtml+xml and other XML types. Choose the mode the same way:

• HTML mode for documents which browsers parse as HTML: web pages and anything else served as text/html, including XHTML-like markup served that way.
• HTML mode with XML syntax (Reader.SetXMLSyntax) for markup written as XHTML (an XML declaration or the XHTML namespace declaration, and self-closing tags meant as empty elements) which may be malformed or ends up rendered by an HTML engine (e.g., EPUB content documents written by hand or converted from HTML), and for polyglot conversion (github.com/pgaskin/htmlstream/polyglot only accepts HTML mode). It reads the content like browsers read text/html, but closes self-closed elements (e.g., <script src="a.js"/>, which otherwise contains the rest of the document) and void elements at their end tag (<br></br> is one br element), and reads processing instructions, and CDATA sections outside raw text and RCDATA elements.
• XML mode for documents which XML parsers read: XML formats (e.g., RSS and Atom feeds, EPUB package documents, SVG files), and XHTML known to be well-formed (e.g., served as application/xhtml+xml), where the result should be what an XML reading system sees (namespaces resolved, XML constructs passed through, a p start tag doesn't close an open p, and script contents are markup). XML mode doesn't recover malformed documents like HTML mode does, and XML parsers reject them.
• Universal mode for HTML or XHTML of unknown type, e.g., a mix of HTML pages and XHTML files. It reads a document starting with an XML declaration whose first start tag declares the XHTML namespace (e.g., <html xmlns="http://www.w3.org/1999/xhtml">, as in most EPUB content documents) like XML mode, and other documents like HTML mode with XML syntax (see Universal). It is only for HTML: never use it for other XML, which it reads as HTML (e.g., the link element of RSS is void, and its URL is text after it).

Without XML syntax, HTML mode reads HTML as browsers read text/html: self-closing tags on non-void elements are start tags (e.g., everything after <div class="clear"/> is in the div), </br> after <br> is another br element, CDATA sections are bogus comments whose contents are hidden, processing instructions are comments, and a doctype ends at the first ">", also inside an internal subset.

In XML mode, elements named like the HTML elements with special parsing rules (e.g., script, title, br and p) are ordinary elements, and names are case-sensitive. A Transformer in XML mode transforms XML documents, including malformed ones, without changing what it doesn't edit: the XML declaration and other processing instructions, doctypes with an internal subset, CDATA sections, comments, character and entity references, namespace prefixes, and the case, order, quoting and whitespace of names and attributes are written as they are in the input. Edits to elements (e.g., Transformer.Remove on a start tag, Transformer.Unwrap, Transformer.Wrap and Transformer.Empty) and attributes keep a well-formed document well-formed, as long as the names and markup passed to them are (see the Append functions), except that removing or unwrapping the root element can leave a document without exactly one. Removing or replacing an end tag on its own, or a part of a comment or CDATA section split into multiple tokens (see Token.Partial), doesn't. Edits which join text ending with "]]" and text starting with ">" (e.g., removing an element between them, or replacing a part of a text split into multiple tokens) form "]]>", which XML doesn't allow in text. Elements closed without an end tag in malformed documents are closed by an implied end tag, which writes nothing (see Element model).

The XML support doesn't process doctypes: entities declared in them aren't expanded (only numeric character references and the named character references of HTML are decoded, see XML), and default attributes aren't added. Markup isn't checked for well-formedness or namespace well-formedness: malformed markup is read like other markup, without an error (e.g., an end tag which doesn't match an open element is stray), and edits don't add or check namespace declarations (see Transformer.Rename). XML documents in UTF-16 or other encodings in which ASCII characters don't have their ASCII bytes must be decoded first (see Input).

The Reader follows XML 1.0, except that in documents declared as XML 1.1 (by an XML declaration with the version 1.1 at the start of the input, after an optional UTF-8 byte order mark), decoded text follows the end-of-line handling of XML 1.1: NEL (U+0085), U+2028, and CR followed by NEL are line ends too, which Token.AppendText decodes as a newline, like CR and CR LF, and Attr.AppendValue as a space. This applies in XML mode, and in Universal mode to documents read like in XML mode (and to the processing instructions before the choice, see Universal). NEL and U+2028 are only recognized in UTF-8, and only as line ends in decoded text: in markup (e.g., between attributes, or between the target and the data of a processing instruction), they aren't whitespace. Character references to them (e.g., &#x85;) are decoded to the character, as in XML 1.1. Otherwise, XML 1.1 documents are read like XML 1.0 ones (the Reader doesn't check the characters and names which XML 1.1 allows differently).

The functions which write and check XML (AppendEscapeText, AppendEscapeAttr, AppendXMLChars, ValidXMLChars, ValidXMLName and github.com/pgaskin/htmlstream/xmlutil.AppendEscape) follow XML 1.0, and github.com/pgaskin/htmlstream/polyglot writes XML 1.0 documents. In an XML 1.1 document, NEL and U+2028 written by them are read as line ends, and the control characters U+007F to U+009F other than NEL, which XML 1.1 only allows as character references, aren't escaped. The attribute values set by a Transformer are escaped for XML 1.1 in documents declared as XML 1.1 (see Transformer).

Input

A Reader reads bytes without decoding them: markup is recognized by the bytes of its ASCII characters, and other bytes are kept as they are. It reads UTF-8, and other encodings in which ASCII characters have their ASCII bytes (e.g., windows-1252 and EUC-JP), without converting them. In Shift_JIS, Big5, GBK and gb18030, the second byte of some characters is that of "[" or "]", which can start or end the internal subset of a doctype or end a CDATA section early. In those and EUC-KR, it can be an ASCII letter, which is compared ASCII case-insensitively and lowercased in names under HTML rules (see Token.HTMLRules). Long text is split into tokens between UTF-8 sequences (see Token.Partial), which can split characters of other multibyte encodings. The encoding declared by the document (by a meta element or the XML declaration) isn't used.

Decoding text (see Token.AppendText and Attr.AppendValue) gives UTF-8 for character references and for the U+FFFD replacing NUL, so with input in another encoding, decoded text mixes encodings. Input in another encoding can be decoded to UTF-8 before reading it, e.g., with golang.org/x/text/encoding/htmlindex and golang.org/x/text/transform. github.com/pgaskin/htmlstream/htmlutil.PrescanCharset finds the encoding an HTML document declares at its start, and github.com/pgaskin/htmlstream/htmlutil.MetaCharset the one declared by a meta element. github.com/pgaskin/htmlstream/xmlutil.DetectEncoding detects the encoding of an XML document from its byte order mark, its first bytes and its XML declaration, and github.com/pgaskin/htmlstream/xmlutil.Encoding.ASCIICompatible reports whether a Reader can read it without decoding it. github.com/pgaskin/htmlstream/polyglot requires UTF-8 input, since its output declares UTF-8.

A Reader returns ErrUnsupportedEncoding for input starting with a UTF-16 or UTF-32 byte order mark, or with "<" followed by another ASCII character in UTF-16. A UTF-8 byte order mark is kept, as the start of the first token (a Text token).

Limits and strict mode

Text, comments and CDATA sections are not limited in length. Long ones are split into multiple tokens at about the chunk size (see Reader.SetChunkSize) and marked by Token.Partial and Token.Continued. Splits never fall inside a UTF-8 sequence, or between CR and LF or NEL (a line end in XML 1.1), or inside a character reference of up to the maximum token size (see Reader.SetMaxTokenSize), so the tokens can be decoded separately. A numeric character reference, which can have any number of leading zeros, is kept whole in one text token up to the maximum token size, whatever the chunk size. A longer one is split there, and the parts are decoded separately: the first as a reference ending at the split, and the rest as text. Reader.ReadText reads the decoded text of all of them, up to a limit.

Tags, doctypes and processing instructions are limited in size (see Reader.SetMaxTokenSize): a larger one is an error (ErrTokenTooLarge), except for a processing instruction read with XML syntax in HTML mode (see Reader.SetXMLSyntax) whose first ">" is past the maximum token size: it is read as a bogus comment, as HTML parsers read it. In Universal mode, such an XML declaration isn't recognized, so the document is read like in HTML mode. The open elements kept are limited in number and size (see Reader.SetMaxDepth and Reader.SetMaxStackBytes, and Token.Untracked for the elements past the limits). The output held back by a Transformer is limited too (see Transformer.SetMaxHold).

The memory used by a Reader is proportional to its limits. Its buffer holds the largest token read, and the attributes of a tag use memory for each attribute in addition to their bytes, so a tag of many short attributes uses many times its size. The default maximum token size allows large tags, as legitimate tags with data: URLs in attribute values can be several megabytes long. For untrusted input, set a lower maximum token size where tags this large aren't expected.

In HTML mode, and in documents read like in HTML mode in Universal mode, end tags and some start tags (e.g., li, dd, dt, a, nobr, the ruby elements, and start tags closing a p) check the open elements, which can take time proportional to the number of open elements. For untrusted input, set a lower maximum depth (see Reader.SetMaxDepth).

Reader.ReadText, github.com/pgaskin/htmlstream/htmlutil.InnerText, github.com/pgaskin/htmlstream/htmlutil.TextContent and github.com/pgaskin/htmlstream/sax.Reader.ReadText collect the text of several tokens, and limit it only when given a positive limit: pass one for untrusted input.

In strict mode (see Reader.SetStrict), the operations which depend on the structure of the document (e.g., Reader.Skip) return a SyntaxError instead of guessing where the markup is structurally unsound (see Token.Unsound), e.g., where an element whose end tag may not be omitted is closed without one.

Strict mode doesn't check that the input is structurally sound according to the HTML or XML specifications, only that the operations used on it are unambiguous.

Element model

A Reader keeps a stack of the open elements (see Reader.Stack and Reader.Depth) with their names, namespaces and namespace declarations, and with Reader.SetKeepAttrs, their start tags. Start and end tags are balanced: every element opened by a StartTag token (see Token.Opens) is closed by exactly one EndTag token. Elements closed without an end tag (optional end tags, misnesting, the end of the input) are closed by an implied EndTag with an empty Raw (see Token.Implied). Void elements and self-closed elements are not opened and have no EndTag, except for void elements in HTML mode with XML syntax (see Reader.SetXMLSyntax), and in the XHTML namespace in documents read like in XML mode in Universal mode (e.g., <br> in <html xmlns="http://www.w3.org/1999/xhtml">): they are opened, and closed by their end tag if it is the next token or follows whitespace (e.g., <br> </br>, where the whitespace is in the element), and otherwise by an implied end tag before the next token, which isn't unsound (see Token.Unsound). Other content before the end tag (e.g., a comment or a character reference) closes them first, so the end tag is stray. End tags which don't close an element are stray (see Token.Stray).

Elements nested past the stack limits (see Reader.SetMaxDepth and Reader.SetMaxStackBytes) are not tracked (see Token.Untracked): they are counted by Depth, but their names aren't kept, and when one is the innermost open element, the next end tag closes it, whatever its name (a tracked element can be open inside an element untracked past the stack size limit), except that a void element under XML rules or with XML syntax is closed like a tracked one (by its end tag only if it is the next token or follows whitespace). Their end tags are still balanced, but have a StartPos of -1, and the implied ones an empty Name. Their contents are read as if they were in the innermost tracked element, with its namespace and rules (e.g., the elements in an untracked svg element aren't SVG elements), and the namespace declarations of untracked elements don't apply to their contents, except that the contents of untracked raw text and RCDATA elements (e.g., script) are still raw text.

In XML mode, an end tag closes the nearest open element with its name. In HTML mode, and in documents read like in HTML mode in Universal mode (see Reader.EffectiveMode), elements are closed like the HTML Standard tree construction stage closes them: end tags are implied where HTML parsers imply them (e.g., a div start tag closes an open p, and a tr start tag an open td), an end tag closes the nearest open element with its name unless an element which limits its scope (e.g., table or td) is open inside it, and SVG and MathML elements are opened and closed like foreign content. The Reader doesn't build a tree, and differs from tree construction mostly in the steps which add elements, move content or ignore tags:

• html, head and body elements are only opened by their start tags. Their start tags inside other elements (except head in html, and body in html or head), which HTML parsers ignore, don't open an element, and their attributes aren't merged into the open html or body element. A body start tag after content which HTML parsers put in an implied body (e.g., the text in <html><head></head>x<body>) opens an element too, while HTML parsers ignore it. A head opened by a start tag is closed by the start tags and text which HTML parsers put after it.
• The end tags of body and html close them, while HTML parsers keep them open and put the content after them in the body.
• There are no implied tbody, tr or colgroup elements, and no foster parenting: content in a table outside a cell stays in the table.
• There is no list of active formatting elements: formatting elements closed by a misnested end tag are not reopened, there is no adoption agency algorithm, and an a or nobr start tag closes an open a or nobr only if no special element (e.g., div) is open inside it.
• An end tag closes the elements open inside its element even where HTML parsers ignore it (e.g., </span> with a div open in the span). </br> (unless it closes a br opened with XML syntax), and </p> without an open p in scope, are stray, while HTML parsers handle them as <br> and <p></p>.
• Start tags which HTML parsers ignore or handle as an end tag, such as nested form and select elements and table parts outside a table, open an element.
• Template contents are read like other content, while HTML parsers use the template insertion modes (e.g., parsing them like table rows).
• Quirks mode isn't detected: a table start tag closes an open p in quirks mode documents too.
• With scripting disabled (the default), noscript in a head without a start tag is read like in the body (see Reader.SetScripting).

github.com/pgaskin/htmlstream/htmlutil.Document tracks the html, head and body elements, including the ones HTML parsers imply, and quirks mode. github.com/pgaskin/htmlstream/polyglot writes the implied elements, reopens misnested formatting elements with a simplified adoption agency algorithm, and warns about the other differences which change its output.

Example

A Transformer copies the input to the output, applying the edits made to each token. Everything else, including malformed markup, is copied unchanged.

package main

import (
	"bytes"
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<P>See <a href="https://example.com/">example.com</a>,
<A HREF=/about class=nav>about</A> and <a href="https://example.org/" rel=nofollow>example.org</a>.
<!-- unclosed <b>bold`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Find(htmlstream.Tag("a", htmlstream.AttrFunc("href", func(v []byte) bool {
		return bytes.HasPrefix(v, []byte("https://"))
	}))) {
		rel := "noopener"
		if a, ok := x.Token().Attr("rel"); ok {
			rel = a.Value() + " " + rel
		}
		if err := x.SetAttr("rel", rel); err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<P>See <a href="https://example.com/" rel="noopener">example.com</a>,
<A HREF=/about class=nav>about</A> and <a href="https://example.org/" rel="nofollow noopener">example.org</a>.
<!-- unclosed <b>bold

Index

• Constants
• Variables
• func AdjustMathMLAttrName(lname string) string
• func AdjustSVGAttrName(lname string) string
• func AdjustSVGTagName(lname string) string
• func AppendCDATA[S ~string | ~[]byte](dst []byte, text S) []byte
• func AppendComment[S ~string | ~[]byte](dst []byte, text S) []byte
• func AppendElement[S ~string | ~[]byte](dst []byte, name string, text S, attrs ...string) []byte
• func AppendEndTag(dst []byte, name string) []byte
• func AppendEscapeAttr[S ~string | ~[]byte](dst []byte, s S) []byte
• func AppendEscapeText[S ~string | ~[]byte](dst []byte, s S) []byte
• func AppendProcInst[S ~string | ~[]byte](dst []byte, target string, data S) []byte
• func AppendStartTag(dst []byte, name string, selfClosing bool, attrs ...string) []byte
• func AppendXMLChars[S ~string | ~[]byte](dst []byte, s S) ([]byte, bool)
• func ValidXMLChars[S ~string | ~[]byte](s S) bool
• func ValidXMLName(name string) bool
• type Attr
  • func (a Attr) AppendName(dst []byte) []byte
  • func (a Attr) AppendValue(dst []byte) []byte
  • func (a Attr) HasValue() bool
  • func (a Attr) Is(name string) bool
  • func (a Attr) Local() []byte
  • func (a Attr) Name() []byte
  • func (a Attr) Prefix() []byte
  • func (a Attr) Quote() byte
  • func (a Attr) Raw() []byte
  • func (a Attr) RawValue() []byte
  • func (a Attr) Value() string
• type Element
  • func (e Element) AppendLocalName(dst []byte) []byte
  • func (e Element) Attr(name string) (Attr, bool)
  • func (e Element) AttrAt(i int) Attr
  • func (e Element) AttrNS(ns, local string) (Attr, bool)
  • func (e Element) AttrNamespace(a Attr) string
  • func (e Element) Attrs() iter.Seq[Attr]
  • func (e Element) HTMLRules() bool
  • func (e Element) HasAttr(name string) bool
  • func (e Element) Is(name string) bool
  • func (e Element) Local() []byte
  • func (e Element) LocalIs(name string) bool
  • func (e Element) NS() string
  • func (e Element) Name() []byte
  • func (e Element) NumAttrs() int
  • func (e Element) Pos() Pos
  • func (e Element) Prefix() []byte
• type Hold
• type Kind
  • func (k Kind) String() string
• type Matcher
  • func And(ms ...Matcher) Matcher
  • func AttrEquals(name, value string) Matcher
  • func AttrFunc(name string, fn func(value []byte) bool) Matcher
  • func AttrHasWord(name, word string) Matcher
  • func AttrNSFunc(ns, local string, fn func(value []byte) bool) Matcher
  • func HasAttr(name string) Matcher
  • func HasAttrNS(ns, local string) Matcher
  • func HasClass(class string) Matcher
  • func Not(m Matcher) Matcher
  • func Or(ms ...Matcher) Matcher
  • func Tag(name string, ms ...Matcher) Matcher
  • func TagNS(ns, local string, ms ...Matcher) Matcher
• type Mode
  • func (m Mode) String() string
• type Pos
• type Reader
  • func NewReader(r io.Reader) *Reader
  • func (r *Reader) All() iter.Seq[*Token]
  • func (r *Reader) Children() iter.Seq[*Token]
  • func (r *Reader) Depth() int
  • func (r *Reader) EffectiveMode() Mode
  • func (r *Reader) Err() error
  • func (r *Reader) Find(m Matcher) bool
  • func (r *Reader) LookupPrefix(prefix string) (string, bool)
  • func (r *Reader) Mode() Mode
  • func (r *Reader) Next() bool
  • func (r *Reader) ReadText(dst []byte, limit int) (b []byte, truncated bool)
  • func (r *Reader) Scripting() bool
  • func (r *Reader) SetChunkSize(n int)
  • func (r *Reader) SetKeepAttrs(v bool)
  • func (r *Reader) SetMaxDepth(n int)
  • func (r *Reader) SetMaxStackBytes(n int)
  • func (r *Reader) SetMaxTokenSize(n int)
  • func (r *Reader) SetMode(m Mode)
  • func (r *Reader) SetScripting(v bool)
  • func (r *Reader) SetStrict(v bool)
  • func (r *Reader) SetXMLSyntax(v bool)
  • func (r *Reader) Skip() error
  • func (r *Reader) Stack() iter.Seq[Element]
  • func (r *Reader) Token() *Token
• type SyntaxError
  • func (e *SyntaxError) Error() string
• type Token
  • func (t *Token) AppendLocalName(dst []byte) []byte
  • func (t *Token) AppendText(dst []byte) []byte
  • func (t *Token) Attr(name string) (Attr, bool)
  • func (t *Token) AttrAt(i int) Attr
  • func (t *Token) AttrNS(ns, local string) (Attr, bool)
  • func (t *Token) AttrNamespace(a Attr) string
  • func (t *Token) Attrs() iter.Seq[Attr]
  • func (t *Token) Bogus() bool
  • func (t *Token) Clone() *Token
  • func (t *Token) Continued() bool
  • func (t *Token) Data() []byte
  • func (t *Token) HTMLRules() bool
  • func (t *Token) HasAttr(name string) bool
  • func (t *Token) Implied() bool
  • func (t *Token) InRawText() bool
  • func (t *Token) Incomplete() bool
  • func (t *Token) Is(name string) bool
  • func (t *Token) Kind() Kind
  • func (t *Token) Local() []byte
  • func (t *Token) LocalIs(name string) bool
  • func (t *Token) NS() string
  • func (t *Token) Name() []byte
  • func (t *Token) NumAttrs() int
  • func (t *Token) Opens() bool
  • func (t *Token) Partial() bool
  • func (t *Token) Pos() Pos
  • func (t *Token) Prefix() []byte
  • func (t *Token) Raw() []byte
  • func (t *Token) SelfClosing() bool
  • func (t *Token) StartPos() Pos
  • func (t *Token) Stray() bool
  • func (t *Token) String() string
  • func (t *Token) Unsound() bool
  • func (t *Token) UnsoundError() error
  • func (t *Token) Untracked() bool
• type TokenBuffer
  • func (b *TokenBuffer) All() iter.Seq[*Token]
  • func (b *TokenBuffer) Append(t *Token)
  • func (b *TokenBuffer) At(i int) *Token
  • func (b *TokenBuffer) Len() int
  • func (b *TokenBuffer) Reset()
  • func (b *TokenBuffer) Size() int
• type Transformer
  • func NewTransformer(w io.Writer, r io.Reader) *Transformer
  • func (t *Transformer) All() iter.Seq[*Token]
  • func (t *Transformer) Children() iter.Seq[*Token]
  • func (t *Transformer) Close() error
  • func (t *Transformer) Depth() int
  • func (t *Transformer) EffectiveMode() Mode
  • func (t *Transformer) Empty() error
  • func (t *Transformer) Err() error
  • func (t *Transformer) ExpandSelfClosing() error
  • func (t *Transformer) Find(m Matcher) bool
  • func (t *Transformer) Held(h Hold) []byte
  • func (t *Transformer) Hold() (Hold, error)
  • func (t *Transformer) HoldDiscard(h Hold, raw []byte) error
  • func (t *Transformer) HoldRelease(h Hold) error
  • func (t *Transformer) Holding() int
  • func (t *Transformer) InsertAfter(raw []byte) error
  • func (t *Transformer) InsertAtEnd(raw []byte)
  • func (t *Transformer) InsertBefore(raw []byte) error
  • func (t *Transformer) LookupPrefix(prefix string) (string, bool)
  • func (t *Transformer) Mode() Mode
  • func (t *Transformer) Next() bool
  • func (t *Transformer) ReadText(dst []byte, limit int) ([]byte, bool)
  • func (t *Transformer) Reader() *Reader
  • func (t *Transformer) Remove() error
  • func (t *Transformer) RemoveAttr(name string) error
  • func (t *Transformer) RemoveAttrs(fn func(Attr) bool) error
  • func (t *Transformer) Rename(name string) error
  • func (t *Transformer) RenameAttr(name, newName string) error
  • func (t *Transformer) RenameEnd(name string) error
  • func (t *Transformer) Replace(raw []byte) error
  • func (t *Transformer) Scripting() bool
  • func (t *Transformer) SetAttr(name, value string) error
  • func (t *Transformer) SetAttrAt(name, value string, at int) error
  • func (t *Transformer) SetChunkSize(n int)
  • func (t *Transformer) SetKeepAttrs(v bool)
  • func (t *Transformer) SetMaxDepth(n int)
  • func (t *Transformer) SetMaxHold(n int)
  • func (t *Transformer) SetMaxStackBytes(n int)
  • func (t *Transformer) SetMaxTokenSize(n int)
  • func (t *Transformer) SetMode(m Mode)
  • func (t *Transformer) SetScripting(v bool)
  • func (t *Transformer) SetStrict(v bool)
  • func (t *Transformer) SetXMLSyntax(v bool)
  • func (t *Transformer) Skip() error
  • func (t *Transformer) Stack() iter.Seq[Element]
  • func (t *Transformer) Token() *Token
  • func (t *Transformer) Unwrap() error
  • func (t *Transformer) Wrap(start, end []byte) error

Examples

• Package
• AppendComment
• AppendElement
• Reader
• Reader.Children
• Reader.Find
• Reader.ReadText
• Reader.SetMode
• Reader.SetStrict
• Reader.Skip
• Reader.Stack
• Token.Attrs
• Token.NS
• TokenBuffer
• Transformer (Xml)
• Transformer.Empty
• Transformer.Hold
• Transformer.HoldDiscard
• Transformer.InsertBefore
• Transformer.ReadText
• Transformer.Remove
• Transformer.Rename
• Transformer.SetAttr
• Transformer.Unwrap
• Transformer.Wrap
• ValidXMLName

Constants

const (
	// DefaultMaxTokenSize is the default maximum size of a tag, doctype or
	// processing instruction, 32 MiB.
	DefaultMaxTokenSize = 32 << 20

	// DefaultMaxDepth is the default maximum number of open elements
	// tracked.
	DefaultMaxDepth = 8192

	// DefaultMaxStackBytes is the default maximum number of bytes kept for
	// the open elements, 8 MiB.
	DefaultMaxStackBytes = 8 << 20

	// DefaultChunkSize is the default size at which text, comments and
	// CDATA sections are split into multiple tokens, 64 KiB.
	DefaultChunkSize = 64 << 10
)

The defaults of the Reader's limits and chunk size (see Reader.SetMaxTokenSize, Reader.SetMaxDepth, Reader.SetMaxStackBytes and Reader.SetChunkSize).

const (
	NamespaceXHTML  = "http://www.w3.org/1999/xhtml"
	NamespaceSVG    = "http://www.w3.org/2000/svg"
	NamespaceMathML = "http://www.w3.org/1998/Math/MathML"
	NamespaceXLink  = "http://www.w3.org/1999/xlink"
	NamespaceXML    = "http://www.w3.org/XML/1998/namespace"
	NamespaceXMLNS  = "http://www.w3.org/2000/xmlns/"
)

Namespace URIs, as returned by Token.NS, Token.AttrNamespace and Reader.LookupPrefix.

const DefaultMaxHold = 64 << 20

DefaultMaxHold is the default maximum size of the output held back by a Transformer (see Transformer.SetMaxHold) and by a polyglot Writer (see github.com/pgaskin/htmlstream/polyglot.Options.MaxHold), 64 MiB.

Variables

var (
	// ErrTokenTooLarge is returned for a tag, doctype or processing
	// instruction larger than the maximum token size (see
	// [Reader.SetMaxTokenSize]). The returned error wraps it with the maximum
	// size.
	ErrTokenTooLarge = errors.New("htmlstream: token too large")

	// ErrUnsupportedEncoding is returned for input starting with a UTF-16 or
	// UTF-32 byte order mark, or with "<" followed by another ASCII character
	// in UTF-16.
	ErrUnsupportedEncoding = errors.New("htmlstream: unsupported encoding (utf-16 or utf-32)")

	// ErrUntrackedLimit is returned when an operation depends on the end of
	// an untracked element (see [Token.Untracked]) while as many untracked
	// elements as the maximum depth (see [Reader.SetMaxDepth]) already have
	// operations pending on their ends. These operations are the
	// Transformer's element operations, and in strict mode, Skip and
	// Children. It is sticky. The returned error wraps it with the maximum
	// depth.
	ErrUntrackedLimit = errors.New("htmlstream: too many untracked elements (MaxDepth) with pending operations")
)

Errors returned by a Reader, and by a Transformer reading with it.

var ErrHeldOpen = errors.New("htmlstream: can't discard held output containing the start tag of an open element (its end tag has edits)")

ErrHeldOpen is returned by Transformer.HoldDiscard when the output held back has the start tag of an element which is still open and whose end tag has edits (from Wrap, Rename or Unwrap). The hold doesn't end, and the Transformer continues.

var ErrHoldLimit = errors.New("htmlstream: held output exceeds MaxHold")

ErrHoldLimit is returned when the output held back (see Transformer.Hold) exceeds the limit set by Transformer.SetMaxHold. The returned error wraps it with the sizes.

var ErrHoldReleased = errors.New("htmlstream: hold already released")

ErrHoldReleased is returned by Transformer.HoldRelease and Transformer.HoldDiscard for a hold which has already ended: released, discarded, or ended by Close. Like ErrHeldOpen, it doesn't stop the Transformer. The holds of github.com/pgaskin/htmlstream/polyglot.Writer return it too.

var ErrIncompleteTag = errors.New("htmlstream: can't edit an incomplete start tag")

ErrIncompleteTag is returned when editing a start tag cut off by the end of the input. Unlike other errors, it doesn't stop the Transformer, and the tag is written unchanged.

var ErrInvalidOp = errors.New("htmlstream: invalid operation")

ErrInvalidOp is returned by a Transformer method called without a current token, on a kind of token it doesn't accept, or in a state which doesn't allow it (e.g., HoldRelease of a hold with an open hold inside it). The returned error wraps it with details. Unlike other errors, it doesn't stop the Transformer, and the operation isn't done. The polyglot package's github.com/pgaskin/htmlstream/polyglot.ErrInvalidOp differs: it stops the conversion.

Functions

func AdjustMathMLAttrName

func AdjustMathMLAttrName(lname string) string

AdjustMathMLAttrName returns the case HTML parsers give the MathML attribute with the lowercase name lname, or lname if it isn't adjusted: definitionurl becomes definitionURL, as in the steps to adjust MathML attributes of the HTML Standard.

func AdjustSVGAttrName

func AdjustSVGAttrName(lname string) string

AdjustSVGAttrName returns the case HTML parsers give the SVG attribute with the lowercase name lname (e.g., "viewbox" becomes "viewBox"), or lname if it isn't adjusted, as in the steps to adjust SVG attributes of the HTML Standard.

func AdjustSVGTagName

func AdjustSVGTagName(lname string) string

AdjustSVGTagName returns the case HTML parsers give the SVG element with the lowercase name lname (e.g., "foreignobject" becomes "foreignObject"), or lname if it isn't adjusted. The names are those of the table in the rules for parsing tokens in foreign content of the HTML Standard.

func AppendCDATA

func AppendCDATA[S ~string | ~[]byte](dst []byte, text S) []byte

AppendCDATA appends a CDATA section containing text to dst. Where the text contains "]]>", which would end the section, the section is ended after "]]" and another one started before ">", so XML parsers read the text unchanged, except for carriage returns, which they convert to newlines everywhere (end-of-line handling) and which can't be escaped in a CDATA section. Characters XML doesn't allow aren't replaced (see AppendXMLChars).

The output may not be valid in HTML documents: HTML parsers only read CDATA sections in SVG and MathML content (elsewhere, they read a bogus comment ending at the first ">").

func AppendComment

func AppendComment[S ~string | ~[]byte](dst []byte, text S) []byte

AppendComment appends a comment containing text to dst, changing the text so the comment is valid in both HTML and XML: a space is inserted where the text starts with ">" or "->", between two hyphens, and after a hyphen at the end. The text then follows the comment syntax of the HTML Standard (it doesn't start with ">" or "->", contain "<!--", "-->" or "--!>", or end with "<!-") and the Comment production of XML 1.0 (no "--", and no "-" at the end). Characters XML doesn't allow aren't replaced (see AppendXMLChars).

Example

AppendComment changes the text where it would end the comment early or isn't allowed in XML comments.

package main

import (
	"fmt"

	"github.com/pgaskin/htmlstream"
)

func main() {
	for _, text := range []string{"note", "a -- b", "->x", "x-", "<!-- nested -->"} {
		fmt.Println(string(htmlstream.AppendComment(nil, text)))
	}
}

Output:
<!--note-->
<!--a - - b-->
<!-- ->x-->
<!--x- -->
<!--<!- - nested - ->-->

func AppendElement

func AppendElement[S ~string | ~[]byte](dst []byte, name string, text S, attrs ...string) []byte

AppendElement appends an element named name to dst, with the escaped text and the attributes given as name and value pairs. It panics if attrs has an odd number of elements. For an element without content written as <name />, use AppendStartTag with selfClosing.

The text is escaped with AppendEscapeText, so name must not be a raw text element like script or style, where HTML parsers don't decode character references, or a void element like br, which has no end tag (HTML parsers read </br> as another br).

Example

The Append functions write markup which HTML and XML parsers read the same way, for inserting with a Transformer.

package main

import (
	"fmt"

	"github.com/pgaskin/htmlstream"
)

func main() {
	var b []byte
	b = htmlstream.AppendElement(b, "a", "Fish & chips <3", "href", "/search?q=fish&chips", "title", `"fish"`)
	b = append(b, '\n')
	b = htmlstream.AppendStartTag(b, "img", true, "src", "a.png", "alt", "")
	b = append(b, '\n')
	b = htmlstream.AppendStartTag(b, "p", false)
	b = htmlstream.AppendEscapeText(b, "1 < 2\r\n")
	b = htmlstream.AppendEndTag(b, "p")
	b = append(b, '\n')
	b = htmlstream.AppendComment(b, "built with --release")
	fmt.Println(string(b))
}

Output:
<a href="/search?q=fish&amp;chips" title="&quot;fish&quot;">Fish &amp; chips &lt;3</a>
<img src="a.png" alt="" />
<p>1 &lt; 2&#13;
</p>
<!--built with - -release-->

func AppendEndTag

func AppendEndTag(dst []byte, name string) []byte

AppendEndTag appends an end tag named name to dst.

func AppendEscapeAttr

func AppendEscapeAttr[S ~string | ~[]byte](dst []byte, s S) []byte

AppendEscapeAttr appends s to dst, escaping it for use as a quoted attribute value in HTML or XML: &, <, >, " and ' are written as character references, and so are tabs, newlines and carriage returns, which XML parsers would change to spaces (attribute-value normalization). Characters XML doesn't allow aren't replaced (see AppendXMLChars). NEL and U+2028, which are line ends in XML 1.1 documents, aren't escaped (see Modes in the package documentation).

func AppendEscapeText

func AppendEscapeText[S ~string | ~[]byte](dst []byte, s S) []byte

AppendEscapeText appends s to dst, escaping it for use as text in HTML or XML: &, < and > are written as character references, and so are carriage returns, which HTML parsers (preprocessing the input stream) and XML parsers (end-of-line handling) convert to newlines. It isn't for the contents of raw text elements like script and style, where HTML parsers don't decode character references. Characters XML doesn't allow aren't replaced (see AppendXMLChars). NEL and U+2028, which are line ends in XML 1.1 documents, aren't escaped (see Modes in the package documentation).

func AppendProcInst

func AppendProcInst[S ~string | ~[]byte](dst []byte, target string, data S) []byte

AppendProcInst appends a processing instruction with the target and data to dst, as <?target data?>, or <?target?> if data is empty. A space is inserted between "?" and ">" where the data contains "?>", which would end the processing instruction. XML parsers skip the whitespace after the target, so data starting with whitespace isn't read unchanged. The target is written as-is and isn't checked: it must be a name (see ValidXMLName) other than "xml" in any case, which is reserved for the XML declaration. Characters XML doesn't allow aren't replaced (see AppendXMLChars).

The output may not be valid in HTML documents: HTML parsers read a processing instruction as a bogus comment ending at the first ">" (see Token.Bogus).

func AppendStartTag

func AppendStartTag(dst []byte, name string, selfClosing bool, attrs ...string) []byte

AppendStartTag appends a start tag named name to dst, with attributes given as name and value pairs. Values are escaped (see AppendEscapeAttr); names are written as-is and aren't checked (see ValidXMLName). If selfClosing, it is written as <name />, which HTML parsers ignore on HTML elements other than void elements (e.g., <div /> opens a div; see Transformer.Empty). It panics if attrs has an odd number of elements.

func AppendXMLChars

func AppendXMLChars[S ~string | ~[]byte](dst []byte, s S) ([]byte, bool)

AppendXMLChars appends s to dst, replacing the characters XML 1.0 doesn't allow (control characters other than tab, newline and carriage return, U+FFFE, U+FFFF, and invalid UTF-8, including encoded surrogates) with U+FFFD, and form feeds, which are whitespace in HTML, with spaces. It reports whether anything was replaced. XML 1.1 only allows U+007F to U+009F other than NEL as character references, which AppendXMLChars doesn't write, unlike a Transformer in the attribute values it sets (see Modes in the package documentation).

func ValidXMLChars

func ValidXMLChars[S ~string | ~[]byte](s S) bool

ValidXMLChars reports whether s is valid UTF-8 and only contains the characters allowed in XML 1.0 documents (the Char production; see AppendXMLChars). XML 1.1 only allows U+007F to U+009F other than NEL as character references (see Modes in the package documentation).

func ValidXMLName

func ValidXMLName(name string) bool

ValidXMLName reports whether name is a valid XML name with namespaces (a QName in Namespaces in XML 1.0): at most one colon, which isn't at the start or end, and the prefix and the local name each start with a letter or underscore. Non-ASCII characters must be name characters under the XML 1.0 Fourth Edition rules, which encoding/xml implements and all XML 1.0 parsers accept (the Fifth Edition allows more). Only the syntax is checked: names with the reserved xml and xmlns prefixes are valid.

Example

The XML helpers check and fix names and text for XML output.

package main

import (
	"fmt"

	"github.com/pgaskin/htmlstream"
)

func main() {
	for _, name := range []string{"p", "xlink:href", "data-1", "1st", "a:b:c", "@click"} {
		fmt.Println(name, htmlstream.ValidXMLName(name))
	}
	text, replaced := htmlstream.AppendXMLChars(nil, "form\ffeed and \x01 control")
	fmt.Printf("%q %t\n", text, replaced)
	fmt.Println(htmlstream.AdjustSVGTagName("foreignobject"), htmlstream.AdjustSVGAttrName("viewbox"))
}

Output:
p true
xlink:href true
data-1 true
1st false
a:b:c false
@click false
"form feed and � control" true
foreignObject viewBox

Types

type Attr

type Attr struct {
	// contains filtered or unexported fields
}

Attr is an attribute of a tag. It is only valid as long as the token it came from.

func (Attr) AppendName

func (a Attr) AppendName(dst []byte) []byte

AppendName appends the attribute name to dst. Under HTML rules (see Token.HTMLRules), it is the name HTML parsers give the attribute before adjusting the case of SVG and MathML attribute names (see AdjustSVGAttrName): ASCII-lowercased, with NUL replaced with U+FFFD, like the HTML tokenizer does in the attribute name state. Otherwise, it is the name as written.

func (Attr) AppendValue

func (a Attr) AppendValue(dst []byte) []byte

AppendValue appends the decoded value to dst. Character references are decoded (under HTML rules, with the rules for attribute values; see XML for those decoded under XML rules) and newlines are normalized. In HTML, NUL is replaced with U+FFFD, like the HTML tokenizer does. In XML, whitespace is normalized to spaces like XML parsers do for attributes without a declared type (attribute-value normalization), and so are NEL, U+2028 and CR NEL, which are line ends in documents declared as XML 1.1 (see Modes in the package documentation).

func (Attr) HasValue

func (a Attr) HasValue() bool

HasValue reports whether the attribute has a value (possibly empty).

func (Attr) Is

func (a Attr) Is(name string) bool

Is reports whether the attribute is named name. Under HTML rules (see Attr.AppendName), the name is read like the HTML tokenizer reads it in the attribute name state, with NUL replaced with U+FFFD, and compared ASCII case-insensitively; otherwise, names are compared exactly.

func (Attr) Local

func (a Attr) Local() []byte

Local returns the attribute name as written without the namespace prefix.

func (Attr) Name

func (a Attr) Name() []byte

Name returns the attribute name as written (see Attr.AppendName).

func (Attr) Prefix

func (a Attr) Prefix() []byte

Prefix returns the namespace prefix of the attribute name as written, or nil. A name starting with a colon has none, as in XML parsers, and other names are split at the first colon (see Token.Prefix).

func (Attr) Quote

func (a Attr) Quote() byte

Quote returns the quote character around the value, or 0 if the value is unquoted or missing.

func (Attr) Raw

func (a Attr) Raw() []byte

Raw returns the exact bytes of the attribute, from the start of the name to the end of the value (including the closing quote).

func (Attr) RawValue

func (a Attr) RawValue() []byte

RawValue returns the undecoded value without quotes.

func (Attr) Value

func (a Attr) Value() string

Value returns the decoded value.

type Element

type Element struct {
	// contains filtered or unexported fields
}

Element is an open element. It is only valid until the next call to Reader.Next.

func (Element) AppendLocalName

func (e Element) AppendLocalName(dst []byte) []byte

AppendLocalName appends the element name without the namespace prefix to dst, as Token.AppendLocalName does.

func (Element) Attr

func (e Element) Attr(name string) (Attr, bool)

Attr returns the element's first attribute named name. It requires Reader.SetKeepAttrs: without it, it reports false.

func (Element) AttrAt

func (e Element) AttrAt(i int) Attr

AttrAt returns the element's attribute at index i in source order (see Element.Attrs). It panics if i is out of the range [0, NumAttrs).

func (Element) AttrNS

func (e Element) AttrNS(ns, local string) (Attr, bool)

AttrNS returns the element's first attribute with the local name local in the namespace ns, like Token.AttrNS. It requires Reader.SetKeepAttrs: without it, it reports false.

func (Element) AttrNamespace

func (e Element) AttrNamespace(a Attr) string

AttrNamespace returns the namespace URI of an attribute of the element, like Token.AttrNamespace, using the namespace declarations in scope at the element.

func (Element) Attrs

func (e Element) Attrs() iter.Seq[Attr]

Attrs returns an iterator over the element's attributes. It is empty unless Reader.SetKeepAttrs was enabled. Like Token.Attrs, ranging over it only avoids allocating if the compiler inlines the iterator; Element.NumAttrs and Element.AttrAt never allocate.

func (Element) HTMLRules

func (e Element) HTMLRules() bool

HTMLRules reports whether the element follows HTML rules (see Token.HTMLRules).

func (Element) HasAttr

func (e Element) HasAttr(name string) bool

HasAttr reports whether the element has an attribute named name (see Attr.Is). It requires Reader.SetKeepAttrs: without it, it reports false.

func (Element) Is

func (e Element) Is(name string) bool

Is reports whether the element is named name (see Token.Is).

func (Element) Local

func (e Element) Local() []byte

Local returns the element name as written without the namespace prefix (see Token.Local).

func (Element) LocalIs

func (e Element) LocalIs(name string) bool

LocalIs reports whether the local name of the element (see Element.Local) is name, compared like Element.Is.

func (Element) NS

func (e Element) NS() string

NS returns the namespace URI of the element (see Token.NS).

func (Element) Name

func (e Element) Name() []byte

Name returns the element name as written in the start tag (see Token.Name).

func (Element) NumAttrs

func (e Element) NumAttrs() int

NumAttrs returns the number of the element's attributes, including duplicates. It is zero unless Reader.SetKeepAttrs was enabled.

func (Element) Pos

func (e Element) Pos() Pos

Pos returns the position of the element's start tag.

func (Element) Prefix

func (e Element) Prefix() []byte

Prefix returns the namespace prefix of the element name as written, or nil (see Token.Prefix).

type Hold

type Hold struct {
	// contains filtered or unexported fields
}

A Hold identifies a hold started by Transformer.Hold, for Transformer.HoldRelease, Transformer.HoldDiscard and Transformer.Held. It is only valid with the Transformer which returned it. The zero Hold is no hold.

type Kind

type Kind uint8

Kind is the kind of a Token.

const (
	// Text is character data, including the contents of raw text and RCDATA
	// elements.
	Text Kind = iota + 1

	// StartTag is a start tag, including self-closing tags.
	StartTag

	// EndTag is an end tag. A Reader also emits an implied EndTag with an
	// empty Raw for every element which is closed without an end tag.
	EndTag

	// Comment is a comment, including bogus comments like <!foo>, </ foo>,
	// and <?foo> in HTML (see [Token.Bogus]).
	Comment

	// CDATA is a CDATA section.
	CDATA

	// Doctype is a document type declaration.
	Doctype

	// ProcInst is a processing instruction, including the XML declaration.
	ProcInst
)

func (Kind) String

func (k Kind) String() string

String returns the name of the constant for k, e.g., "StartTag".

type Matcher

type Matcher func(*Token) bool

Matcher reports whether a token matches, for Reader.Find. Matchers can be combined with And, Or and Not.

The attribute matchers (HasAttr, AttrEquals, AttrFunc, etc.) also match end tags with attributes (e.g., </p class="x">), whose attributes HTML parsers ignore. Reader.Find only returns start tags.

func And

func And(ms ...Matcher) Matcher

And matches tokens which match all of ms.

func AttrEquals

func AttrEquals(name, value string) Matcher

AttrEquals matches tags with an attribute named name whose decoded value is value.

func AttrFunc

func AttrFunc(name string, fn func(value []byte) bool) Matcher

AttrFunc matches tags with an attribute named name whose decoded value matches fn. The value is only valid during the call. It is in a buffer of the token shared with the other matchers of decoded values (AttrEquals, AttrHasWord, HasClass, AttrNSFunc and HasAttrNS), so fn must not call them on the same token.

func AttrHasWord

func AttrHasWord(name, word string) Matcher

AttrHasWord matches tags with word in the whitespace-separated value of the attribute named name (e.g., class or rel). Words are compared exactly; for attributes whose keywords are ASCII case-insensitive, like rel, use AttrFunc.

func AttrNSFunc

func AttrNSFunc(ns, local string, fn func(value []byte) bool) Matcher

AttrNSFunc matches tags with an attribute with the local name local (compared like Attr.Is) in the namespace ns (see Token.AttrNamespace) whose decoded value matches fn. The value is only valid during the call, and fn must not call the other matchers of decoded values on the same token, as for AttrFunc.

func HasAttr

func HasAttr(name string) Matcher

HasAttr matches tags with an attribute named name.

func HasAttrNS

func HasAttrNS(ns, local string) Matcher

HasAttrNS matches tags with an attribute with the local name local in the namespace ns (see Token.AttrNamespace).

func HasClass

func HasClass(class string) Matcher

HasClass matches tags with class in their whitespace-separated class attribute. Classes are compared exactly, as browsers do in no-quirks and limited-quirks mode documents. In quirks mode, browsers compare classes ASCII case-insensitively.

func Not

func Not(m Matcher) Matcher

Not matches tokens which don't match m.

func Or

func Or(ms ...Matcher) Matcher

Or matches tokens which match any of ms.

func Tag

func Tag(name string, ms ...Matcher) Matcher

Tag matches start and end tags named name (see Token.Is) which match all of ms.

func TagNS

func TagNS(ns, local string, ms ...Matcher) Matcher

TagNS matches start and end tags with the local name local in the namespace ns (see Token.NS) which match all of ms.

type Mode

type Mode uint8

Mode selects the markup syntax accepted by a Reader.

const (
	// HTML tokenizes like the [HTML Standard] tokenizer does for text/html,
	// which browsers use for all HTML documents, including HTML 4 ones. Use
	// it for documents which browsers parse as HTML: anything served as
	// text/html, whatever the markup looks like.
	//
	// Raw text elements (script, style, etc.) and RCDATA elements (title,
	// textarea) are recognized, self-closing syntax only closes elements
	// inside svg and math (elsewhere the slash is ignored: void elements have
	// no contents anyway, and other elements are opened), and CDATA sections
	// are only recognized inside svg and math.
	//
	// [HTML Standard]: https://html.spec.whatwg.org/multipage/parsing.html#tokenization
	HTML Mode = iota

	// XML tokenizes XML syntax, including XHTML. Use it for documents which
	// XML parsers read: XML formats, and XHTML served as an XML media type
	// (e.g., application/xhtml+xml).
	//
	// There are no raw text elements, CDATA sections and processing
	// instructions are recognized everywhere, self-closing syntax is always
	// honored, names are case-sensitive, and doctypes may contain an internal
	// subset, which isn't processed. Comments end at the first "-->" (HTML
	// parsers also end them at "--!>"), and the internal subset ends at the
	// first "]" outside its markup declarations, comments, processing
	// instructions and conditional sections.
	//
	// Character references (see [Token.AppendText] and [Attr.AppendValue])
	// are decoded if they end with a semicolon: numeric ones, and all the
	// named character references of HTML (e.g.,  ), not only those of
	// the entities XML predefines (e.g., &), which are the only ones XML
	// parsers accept without a declaration in the doctype (most other XML
	// parsers throw an error in this case). Numeric references to U+0080 to
	// U+009F aren't replaced with windows-1252 characters like in HTML. Other
	// references (e.g., to entities declared in the doctype) are left as they
	// are.
	//
	// A Reader in XML mode follows [XML 1.0], with the [end-of-line handling]
	// of XML 1.1 in documents declared as XML 1.1 (see Modes in the package
	// documentation).
	//
	// [XML 1.0]: https://www.w3.org/TR/xml/
	// [end-of-line handling]: https://www.w3.org/TR/xml11/#sec-line-ends
	XML

	// Universal reads HTML or XHTML of unknown type, choosing the rules of
	// each document at its start (see [Reader.EffectiveMode]): a document
	// starting with an XML declaration (after an optional UTF-8 byte order
	// mark) whose first start tag declares the XHTML namespace for its own
	// name on that tag (e.g., <html xmlns="http://www.w3.org/1999/xhtml">, or
	// <h:html xmlns:h="http://www.w3.org/1999/xhtml">) is read like in [XML]
	// mode, and every other document like in [HTML] mode with
	// [Reader.SetXMLSyntax]. XHTML files usually start with an XML
	// declaration, and XHTML 1.x pages served as text/html usually don't. The
	// XML declaration is recognized as a processing instruction starting with
	// <?xml followed by XML whitespace or ?>, without validating it (e.g.,
	// <?xml?> and the text declaration <?xml encoding="utf-8"?> are recognized
	// too).
	//
	// It is a fallback for HTML or XHTML of unknown type: if the parser which
	// reads the document is known (HTML for text/html, XML for XML media
	// types), use [HTML] or [XML] mode (see Modes in the package
	// documentation). Never use it for documents which aren't HTML: XML
	// formats (e.g., feeds, EPUB package documents and SVG files) go to XML
	// mode, as Universal mode reads them as HTML (e.g., the link element of
	// RSS is void).
	//
	// In documents read like in XML mode, the HTML void elements in the XHTML
	// namespace (with their names in lowercase, e.g., br and meta, but not
	// image, which only HTML parsers turn into img) are closed like with XML
	// syntax in HTML mode: one without self-closing syntax is closed by its
	// end tag if it is the next token or follows whitespace, and otherwise by
	// an implied end tag before the next token (see Element model in the
	// package documentation), so <br>, <br></br>, <br> </br> and <br/> are all
	// a br element. XML parsers read <br> as the start of an element
	// containing the rest of its parent.
	//
	// The rules are chosen at the first token, or with an XML declaration, at
	// the first token after it other than whitespace, a comment, a processing
	// instruction or a doctype. The tokens before that are read like in HTML
	// mode with XML syntax, except that processing instructions and doctypes
	// after the XML declaration are read like in XML mode whatever the rules
	// chosen (e.g., <?pi a>b?> is one processing instruction), with the line
	// ends of XML 1.1 in documents declared as XML 1.1. One of them which
	// reaches the end of the input that way ends at the first ">" in it after
	// all, as in HTML mode with XML syntax (a processing instruction is then a
	// bogus comment, and an incomplete one without a ">"), and the rules are
	// chosen after it as usual. Before the rules are chosen, and in documents
	// read like in HTML mode, only the first doctype of the input is read with
	// an internal subset or quoted literals, as with XML syntax; in documents
	// read like in XML mode, the doctypes after the first start tag are read
	// like in XML mode. As in HTML mode, comments also end at "--!>", and a
	// start tag whose name doesn't start with an ASCII letter (e.g., <_x>) is
	// text.
	Universal
)

func (Mode) String

func (m Mode) String() string

String returns the name of the constant for m, e.g., "HTML".

type Pos

type Pos int64

Pos is the byte offset of a token in the input. It identifies a token across multiple passes over the same input. Implied end tags have the position of the token which caused them, or the input length at EOF. A line and column, if needed, are counted from the input by the caller.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads tokens from HTML, XHTML or XML markup. It isn't safe for concurrent use, and reads a single input: it can't be reset to read another one.

Example

Every element opened by a start tag is closed by exactly one end tag. End tags omitted in the input are implied, and end tags which don't close an element are stray.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<ul><li>One<li>Two</ul></span><p>Three`

	r := htmlstream.NewReader(strings.NewReader(page))
	for tok := range r.All() {
		var flags string
		if tok.Implied() {
			flags += " (implied)"
		}
		if tok.Stray() {
			flags += " (stray)"
		}
		fmt.Printf("%d %-8s %q%s\n", r.Depth(), tok.Kind(), tok.Raw(), flags)
	}
	if err := r.Err(); err != nil {
		panic(err)
	}
}

Output:
1 StartTag "<ul>"
2 StartTag "<li>"
2 Text     "One"
2 EndTag   "" (implied)
2 StartTag "<li>"
2 Text     "Two"
2 EndTag   "" (implied)
1 EndTag   "</ul>"
0 EndTag   "</span>" (stray)
1 StartTag "<p>"
1 Text     "Three"
1 EndTag   "" (implied)

func NewReader

func NewReader(r io.Reader) *Reader

NewReader returns a Reader reading from r in HTML mode. It reads bytes without decoding them, which works for UTF-8 and other encodings in which ASCII characters have their ASCII bytes, and returns ErrUnsupportedEncoding for UTF-16 and UTF-32 input (see Input in the package documentation).

func (*Reader) All

func (r *Reader) All() iter.Seq[*Token]

All returns an iterator over the remaining tokens. Each token is valid until the next iteration. Check Err after iterating.

func (*Reader) Children

func (r *Reader) Children() iter.Seq[*Token]

Children returns an iterator over the tokens inside the element opened by the current start tag, which may be untracked (see Token.Untracked). After the iteration completes, the current token is the element's end tag. If the current token doesn't open an element, the iterator yields nothing. Check Reader.Err after iterating.

Example

Children iterates over the tokens inside an element, including those of nested elements. Depth gives the nesting level.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<nav><ul>
<li><a href=/>Home</a>
<li><a href=/docs>Docs</a><ul><li><a href=/docs/api>API</a></ul>
<li><a href=/blog>Blog</a>
</ul></nav>`

	r := htmlstream.NewReader(strings.NewReader(page))
	if !r.Find(htmlstream.Tag("ul")) {
		panic("no list")
	}
	depth := r.Depth()
	for tok := range r.Children() {
		if tok.Kind() == htmlstream.StartTag && tok.Is("a") {
			href, _ := tok.Attr("href")
			fmt.Printf("level %d: %s\n", (r.Depth()-depth)/2, href.Value())
		}
	}
	if err := r.Err(); err != nil {
		panic(err)
	}
	fmt.Println(r.Token().Kind(), string(r.Token().Raw()))
}

Output:
level 1: /
level 1: /docs
level 2: /docs/api
level 1: /blog
EndTag </ul>

func (*Reader) Depth

func (r *Reader) Depth() int

Depth returns the number of open elements, including the current start or end tag's element. A stray end tag (see Token.Stray) has the same depth as the enclosing element's end tag would; use Token.StartPos to identify the element an end tag closes.

func (*Reader) EffectiveMode

func (r *Reader) EffectiveMode() Mode

EffectiveMode returns the mode whose rules the Reader follows. In HTML and XML modes, it is the mode. In Universal mode, it is Universal until the Reader chooses the rules of the document, and then HTML (read like in HTML mode with Reader.SetXMLSyntax) or XML. The choice is made at a token before or at the first start or end tag (see Universal), and applies to that token.

func (*Reader) Err

func (r *Reader) Err() error

Err returns the first error encountered, or nil at the end of the input. An error of the io.Reader is returned as it is, after the tokens for the input read before it, and without implied end tags for the open elements. Errors are sticky: Next, Find, Skip and the other methods stop at them.

func (*Reader) Find

func (r *Reader) Find(m Matcher) bool

Find advances to the next start tag matching m. It returns false at the end of the input or on error.

Example

Find advances to the next start tag matching a Matcher. Tag names and attribute names are compared ASCII case-insensitively in HTML mode.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<div class="card featured"><IMG SRC=a.png alt=A></div>
<div class=card><img src=b.png><img alt="no source"></div>
<img class=featured src=c.png>`

	r := htmlstream.NewReader(strings.NewReader(page))
	for r.Find(htmlstream.Or(
		htmlstream.HasClass("featured"),
		htmlstream.Tag("img", htmlstream.HasAttr("src"), htmlstream.Not(htmlstream.HasAttr("alt"))),
	)) {
		tok := r.Token()
		fmt.Printf("%s %s\n", tok.Name(), tok.Raw())
	}
	if err := r.Err(); err != nil {
		panic(err)
	}
}

Output:
div <div class="card featured">
img <img src=b.png>
img <img class=featured src=c.png>

func (*Reader) LookupPrefix

func (r *Reader) LookupPrefix(prefix string) (string, bool)

LookupPrefix resolves a namespace prefix ("" for the default namespace) using the namespace declarations in scope, including those on the current start tag. It works in all modes, even though namespace declarations are otherwise ignored under HTML rules (see Token.HTMLRules), where prefixes and declarations are matched ASCII case-insensitively, like attribute names. It only reports declarations: Token.AttrNamespace ignores them for xlink:href and the other xlink attributes HTML parsers put in the XLink namespace on SVG and MathML elements.

func (*Reader) Mode

func (r *Reader) Mode() Mode

Mode returns the markup syntax (see Reader.SetMode).

func (*Reader) Next

func (r *Reader) Next() bool

Next advances to the next token. It returns false at the end of the input or on error.

func (*Reader) ReadText

func (r *Reader) ReadText(dst []byte, limit int) (b []byte, truncated bool)

ReadText appends the decoded text (see Token.AppendText) of the current Text, Comment, CDATA or ProcInst token to dst and returns the result. At other tokens (e.g., a start tag), it appends nothing: github.com/pgaskin/htmlstream/htmlutil.TextContent and github.com/pgaskin/htmlstream/htmlutil.InnerText read the text of an element. If limit is positive, at most limit bytes are appended, without splitting a UTF-8 sequence; the rest of the text is skipped and truncated is true.

If the text is split into several tokens (see Token.Partial), ReadText calls Next until the last of them, which becomes the current token. Check Err afterwards, since Next can fail. With a Transformer, each token advanced over is written as by Next: edits made before ReadText apply to the first token, and edits made after it to the last one. To replace all of the text, hold back the output from the first token (see Transformer.Hold), replace the last token, and end the hold with Transformer.HoldDiscard.

Example

Long text is split into several tokens at about the chunk size. ReadText reads the decoded text of all of them, up to a limit.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p>Fish &amp; chips, ` + `fish &amp; chips, ` + `fish &amp; chips.</p><p>Next</p>`

	r := htmlstream.NewReader(strings.NewReader(page))
	r.SetChunkSize(16)
	for r.Next() {
		tok := r.Token()
		if tok.Kind() == htmlstream.Text {
			partial := tok.Partial()
			text, truncated := r.ReadText(nil, 24)
			fmt.Printf("%q partial=%t truncated=%t\n", text, partial, truncated)
		}
	}
	if err := r.Err(); err != nil {
		panic(err)
	}
}

Output:
"Fish & chips, fish & chi" partial=true truncated=true
"Next" partial=false truncated=false

func (*Reader) Scripting

func (r *Reader) Scripting() bool

Scripting reports whether scripting is enabled (see Reader.SetScripting).

func (*Reader) SetChunkSize

func (r *Reader) SetChunkSize(n int)

SetChunkSize sets the size at which text, comments and CDATA sections are split into multiple tokens (see Token.Partial). A text token can be longer, up to the maximum token size, to keep a numeric character reference whole (see Limits and strict mode in the package documentation). It may be called at any time and applies from the next token. The default is DefaultChunkSize. Values below 1 are treated as 1, and it is limited to the maximum token size.

func (*Reader) SetKeepAttrs

func (r *Reader) SetKeepAttrs(v bool)

SetKeepAttrs sets whether the attributes of open elements are kept, so they are available from Element.Attrs. The start tags count against the stack size limit (see Reader.SetMaxStackBytes), so with large attributes, fewer elements are tracked. It must be called before the first call to Next, and panics otherwise.

func (*Reader) SetMaxDepth

func (r *Reader) SetMaxDepth(n int)

SetMaxDepth sets the maximum number of open elements tracked. Elements nested deeper are not tracked (see Token.Untracked): their names and attributes aren't kept, but their ends are still known. Elements are also untracked past the stack size limit (see Reader.SetMaxStackBytes). At most that many untracked elements can have operations pending on their ends (see ErrUntrackedLimit). The default is DefaultMaxDepth, and values below 1 are treated as 1. It must be called before the first call to Next, and panics otherwise.

In HTML mode, and in documents read like in HTML mode in Universal mode, end tags and some start tags (e.g., li, dd, dt, a, nobr, the ruby elements, and start tags closing a p) check the open elements, which can take time proportional to the number of open elements. For untrusted input, set it lower.

func (*Reader) SetMaxStackBytes

func (r *Reader) SetMaxStackBytes(n int)

SetMaxStackBytes sets the maximum number of bytes kept for the open elements: their names, namespace declarations, and with Reader.SetKeepAttrs, their start tags. Namespace URIs count twice, as they are also kept as the strings returned by Token.NS and the other methods returning namespace URIs. An element which would exceed it is not tracked (see Token.Untracked), like one nested past the maximum depth. The default is DefaultMaxStackBytes, and negative values are treated as 0. It must be called before the first call to Next, and panics otherwise.

func (*Reader) SetMaxTokenSize

func (r *Reader) SetMaxTokenSize(n int)

SetMaxTokenSize sets the maximum size of a tag, doctype or processing instruction. Character data (text, comments and CDATA sections) is split into multiple tokens instead. The default is DefaultMaxTokenSize, and values below 64 are treated as 64. It must be called before the first call to Next, and panics otherwise.

The memory used by the Reader is proportional to it, and a tag of many short attributes uses many times its size (see Limits and strict mode in the package documentation). For untrusted input, set it as low as the largest tags expected allow.

func (*Reader) SetMode

func (r *Reader) SetMode(m Mode)

SetMode sets the markup syntax. It must be called before the first call to Next, and panics otherwise.

Example

The modes read the same markup differently. In HTML mode, title contents are text and br is a void element. In XML mode, names are case-sensitive and every start tag opens an element unless self-closing. Universal mode reads a document without an XML declaration like HTML mode with XML syntax, so the br element is closed by an implied end tag before the text after it.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<title>A <b>bold</b> title</title><P>a<br>b</p>`

	for _, mode := range []htmlstream.Mode{htmlstream.HTML, htmlstream.XML, htmlstream.Universal} {
		r := htmlstream.NewReader(strings.NewReader(page))
		r.SetMode(mode)
		var toks []string
		for tok := range r.All() {
			s := string(tok.Raw())
			switch {
			case tok.Implied():
				s = "(implied </" + string(tok.Name()) + ">)"
			case tok.Stray():
				s += "(stray)"
			}
			toks = append(toks, s)
		}
		if err := r.Err(); err != nil {
			panic(err)
		}
		fmt.Printf("%s: %s\n", mode, strings.Join(toks, "|"))
	}
}

Output:
HTML: <title>|A <b>bold</b> title|</title>|<P>|a|<br>|b|</p>
XML: <title>|A |<b>|bold|</b>| title|</title>|<P>|a|<br>|b|</p>(stray)|(implied </br>)|(implied </P>)
Universal: <title>|A <b>bold</b> title|</title>|<P>|a|<br>|(implied </br>)|b|</p>

func (*Reader) SetScripting

func (r *Reader) SetScripting(v bool)

SetScripting sets whether scripting is enabled, like the parser scripting mode of HTML parsers. The default is false. With scripting enabled, noscript elements contain raw text, as in browsers running scripts. With scripting disabled, their contents are markup, and a noscript element in a head element opened by a start tag is closed by content other than link, meta, style, etc., like in HTML parsers; in a head without a start tag, which the Reader doesn't track, it is parsed like in the body. Under XML rules (see Token.HTMLRules), noscript contents are markup whatever the setting. It must be called before the first call to Next, and panics otherwise.

Scripting is generally better left disabled. With scripting enabled, the contents of a noscript element are raw text, so Reader.Find, the Transformer, github.com/pgaskin/htmlstream/sax and github.com/pgaskin/htmlstream/htmlutil don't see the elements in them (e.g., the img fallback of a lazily loaded image), while XML parsers, readers which don't run scripts and github.com/pgaskin/htmlstream/polyglot.Convert read them as markup. Enable it to read documents as a browser running scripts does, e.g., for github.com/pgaskin/htmlstream/htmlutil.InnerText and github.com/pgaskin/htmlstream/htmlutil.IsHidden to follow what such a browser renders.

func (*Reader) SetStrict

func (r *Reader) SetStrict(v bool)

SetStrict sets whether operations which depend on structurally unsound markup (see Token.Unsound) return a SyntaxError instead of guessing, which also stops the Reader. The default is false, and it may be called at any time. These operations are:

• Reader.Find, at an unsound start tag it matches;
• Reader.Skip and Reader.Children, at an unsound start tag which opens an element, and at the unsound implied end tag of the element;
• the Transformer's methods which require a start tag (e.g., Transformer.SetAttr), at an unsound start tag;
• the Transformer's element operations (Transformer.Remove, Transformer.Unwrap, Transformer.Wrap, Transformer.Rename and Transformer.Empty), like Skip; for those which don't advance to the end of the element, Next returns the error at its implied end tag.

Otherwise, unsound tokens are read like the others.

Strict mode doesn't check that the input is structurally sound according to the HTML or XML specifications, only that the operations used on it are unambiguous.

Example

In strict mode, operations which depend on structurally unsound markup, like skipping an element closed without its required end tag, return a SyntaxError.

package main

import (
	"errors"
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<div><span>unclosed</div><p>after</p>`

	for _, strict := range []bool{false, true} {
		r := htmlstream.NewReader(strings.NewReader(page))
		r.SetStrict(strict)
		if !r.Find(htmlstream.Tag("span")) {
			panic("no span")
		}
		if err := r.Skip(); err != nil {
			var serr *htmlstream.SyntaxError
			if errors.As(err, &serr) {
				fmt.Printf("strict=%t: %s at offset %d\n", strict, serr.Msg, serr.Pos)
				continue
			}
			panic(err)
		}
		fmt.Printf("strict=%t: skipped to %q, implied=%t unsound=%t\n", strict, r.Token().Raw(), r.Token().Implied(), r.Token().Unsound())
	}
}

Output:
strict=false: skipped to "", implied=true unsound=true
strict=true: element "span" at offset 5 closed without its end tag at offset 19

func (*Reader) SetXMLSyntax

func (r *Reader) SetXMLSyntax(v bool)

SetXMLSyntax makes HTML mode accept XML syntax: self-closing tags on non-void elements, CDATA sections (outside raw text and RCDATA elements, e.g., script and title, whose contents are text up to their end tag, as in HTML), processing instructions, and doctypes with an internal subset (which isn't processed, see XML). A processing instruction ends at the first ">", where HTML parsers end the bogus comment they read it as, and is one only if that ">" follows "?": <?php $a->b ?> and Word's <?xml:namespace ... /> are bogus comments ending at that ">". The ">" is looked for only up to the maximum token size (see Reader.SetMaxTokenSize), so a processing instruction whose first ">" is past it is a bogus comment too, and in Universal mode, an XML declaration this long isn't recognized, so the document is read like in HTML mode. A doctype ends at the first ">" outside its internal subset, also in a quoted literal, as in HTML parsers, and one whose internal subset reaches the end of the input at the first ">" in it. Only the first doctype of the input is read with an internal subset: later ones, which HTML parsers ignore, end at the first ">". Void elements without self-closing syntax open an element, closed by their end tag if it is the next token or follows whitespace, as in XML, and otherwise by an implied end tag before the next token (see Element model in the package documentation): <br>, <br></br>, <br> </br> and <br/> are all one br element, while HTML parsers read </br> as another one. It has no effect in XML mode, which always accepts XML syntax, and in Universal mode, which applies it to the documents it reads like in HTML mode. It must be called before the first call to Next, and panics otherwise.

Enable it for documents written as XHTML (e.g., with an XML declaration or the XHTML namespace declaration, and self-closing tags meant as empty elements) or read as XML. Leave it disabled to read HTML as browsers read text/html, where <div/> is a start tag, CDATA sections are bogus comments and processing instructions are comments (see Modes in the package documentation).

func (*Reader) Skip

func (r *Reader) Skip() error

Skip advances to the end tag of the element opened by the current start tag, which may be untracked (see Token.Untracked). If the current token doesn't open an element, Skip does nothing. It returns the error which stopped the Reader, if any (see Reader.Err), including the SyntaxError of strict mode.

Example

Skip advances to the end tag of the current element, here to leave out the contents of script and style elements while collecting the text.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p>Hello<script>document.write("<p>scripted</p>")</script>, <style>p { color: red }</style>world</p>`

	r := htmlstream.NewReader(strings.NewReader(page))
	var text []byte
	for r.Next() {
		tok := r.Token()
		switch tok.Kind() {
		case htmlstream.StartTag:
			if tok.Is("script") || tok.Is("style") {
				if err := r.Skip(); err != nil {
					panic(err)
				}
			}
		case htmlstream.Text:
			text = tok.AppendText(text)
		}
	}
	if err := r.Err(); err != nil {
		panic(err)
	}
	fmt.Println(string(text))
}

Output:
Hello, world

func (*Reader) Stack

func (r *Reader) Stack() iter.Seq[Element]

Stack returns an iterator over the open elements, innermost first. While the current token is a start tag which opens a tracked element, or that element's end tag, it is the innermost element. Elements nested past the stack limits (see Token.Untracked) aren't included, though Reader.Depth counts them.

Example

Stack returns the open elements. With SetKeepAttrs, their attributes are kept too.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<body><main id=content><section class=intro><p>Hello, <em>world</em>!</p></section></main></body>`

	r := htmlstream.NewReader(strings.NewReader(page))
	r.SetKeepAttrs(true)
	if !r.Find(htmlstream.Tag("em")) {
		panic("no em")
	}
	var path []string
	for e := range r.Stack() {
		s := string(e.Name())
		if id, ok := e.Attr("id"); ok {
			s += "#" + id.Value()
		}
		if class, ok := e.Attr("class"); ok {
			s += "." + class.Value()
		}
		path = append([]string{s}, path...)
	}
	fmt.Println(strings.Join(path, " > "))
}

Output:
body > main#content > section.intro > p > em

func (*Reader) Token

func (r *Reader) Token() *Token

Token returns the current token. It is valid until the next call to Next. It returns nil before the first call to Next and after Next returns false.

type SyntaxError

type SyntaxError struct {
	Pos Pos    // the position of the token
	Msg string // the problem, e.g., why the token is unsound
}

SyntaxError is a syntax error at a position in the input. The Reader returns one in strict mode when an operation depends on structurally unsound markup (see Reader.SetStrict); other packages also use it in their strict modes, e.g., for input they would drop (see github.com/pgaskin/htmlstream/sax.Reader.SetStrict).

func (*SyntaxError) Error

func (e *SyntaxError) Error() string

Error returns the message with the position, as "htmlstream: syntax error at offset Pos: Msg".

type Token

type Token struct {
	// contains filtered or unexported fields
}

Token is a token read by a Reader. It is only valid until the next call to Reader.Next, and the byte slices it returns alias the Reader's buffer (see Token.Clone and TokenBuffer for copies which stay valid).

func (*Token) AppendLocalName

func (t *Token) AppendLocalName(dst []byte) []byte

AppendLocalName appends the tag name without the namespace prefix to dst. If the tag follows HTML rules (see Token.HTMLRules), it is the name HTML parsers give the element: ASCII-lowercased and with NUL replaced with U+FFFD, like the HTML tokenizer does in the tag name state. This includes SVG and MathML elements; HTML parsers then give some SVG elements a name with uppercase letters (e.g., foreignObject), which AdjustSVGTagName returns. This avoids allocating to switch on tag names:

var buf [16]byte
switch string(tok.AppendLocalName(buf[:0])) {
case "p":
}

func (*Token) AppendText

func (t *Token) AppendText(dst []byte) []byte

AppendText appends the decoded contents of a Text, Comment, CDATA or ProcInst token to dst, like the HTML tokenizer (or an XML parser) reads them: character references are decoded in text (except in raw text elements like script; see XML for those decoded under XML rules), and newlines are normalized: CR and CR LF become LF, and so do NEL, U+2028 and CR NEL under XML rules in documents declared as XML 1.1 (see Modes in the package documentation). NUL is replaced with U+FFFD in RCDATA, raw text and comments (except in XML mode, see Reader.EffectiveMode), like the tokenizer does; in other text and in CDATA sections, it is kept, as the tokenizer emits it (tree construction drops it from text in HTML content, and replaces it in SVG and MathML content). XML doesn't allow NUL anywhere.

For a ProcInst token, it appends the data like an XML parser reads it (processing instruction): the contents after the target, without the whitespace separating them, with newlines normalized. Character references aren't decoded, and NUL is kept.

func (*Token) Attr

func (t *Token) Attr(name string) (Attr, bool)

Attr returns the first attribute named name, compared like Attr.Is. HTML parsers also keep the first of the attributes with the same name and drop the others (see the attribute name state); in XML, duplicate attributes are an error.

func (*Token) AttrAt

func (t *Token) AttrAt(i int) Attr

AttrAt returns the attribute of a tag at index i in source order (see Token.Attrs). It panics if i is out of the range [0, NumAttrs).

func (*Token) AttrNS

func (t *Token) AttrNS(ns, local string) (Attr, bool)

AttrNS returns the first attribute with the local name local (compared like Attr.Is) in the namespace ns (see Token.AttrNamespace); ns is empty for unprefixed attributes.

func (*Token) AttrNamespace

func (t *Token) AttrNamespace(a Attr) string

AttrNamespace returns the namespace URI of an attribute of the token, which is empty for unprefixed attributes other than xmlns.

The xml and xmlns prefixes are always bound, and xmlns is in the XMLNS namespace. On SVG and MathML elements under HTML rules (see Token.HTMLRules), xlink:actuate, xlink:arcrole, xlink:href, xlink:role, xlink:show, xlink:title and xlink:type are in the XLink namespace whatever the namespace declarations say, as in HTML parsers (see adjust foreign attributes). Under HTML rules, these prefixes and names are matched ASCII case-insensitively, like attribute names.

Other prefixed attributes, including other xlink ones, are resolved using the namespace declarations in scope, in all modes. Under HTML rules, this is an extension: HTML parsers put them in no namespace. Likewise, HTML parsers only put xml:lang and xml:space (in the XML namespace) and xmlns and xmlns:xlink (in the XMLNS namespace) in a namespace on SVG and MathML elements, and other xml and xmlns attributes, and those of HTML elements, in no namespace. Under HTML rules, prefixes and namespace declarations are matched ASCII case-insensitively, like attribute names. On a copy (see Token.Clone), the declarations in scope when the copy was made are used.

func (*Token) Attrs

func (t *Token) Attrs() iter.Seq[Attr]

Attrs returns an iterator over the attributes of a tag in source order, including duplicates. Ranging over it only avoids allocating if the compiler inlines the iterator, which it doesn't do in large functions; Token.NumAttrs and Token.AttrAt never allocate.

Example

Attributes are decoded like HTML parsers decode them. Raw gives the exact bytes.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<input type=checkbox Checked value="fish &amp; chips" title='it&#39;s' type=radio>`

	r := htmlstream.NewReader(strings.NewReader(page))
	r.Next()
	tok := r.Token()
	for a := range tok.Attrs() {
		fmt.Printf("%-8s %-24s value=%q has=%t quote=%q\n", a.Name(), a.Raw(), a.Value(), a.HasValue(), a.Quote())
	}
	typ, _ := tok.Attr("TYPE")
	fmt.Println("type:", typ.Value())
}

Output:
type     type=checkbox            value="checkbox" has=true quote='\x00'
Checked  Checked                  value="" has=false quote='\x00'
value    value="fish & chips" value="fish & chips" has=true quote='"'
title    title='it's'         value="it's" has=true quote='\''
type     type=radio               value="radio" has=true quote='\x00'
type: checkbox

func (*Token) Bogus

func (t *Token) Bogus() bool

Bogus reports whether the token is a bogus comment, like <!foo> or <?foo> in HTML, which the HTML tokenizer reads in the bogus comment state.

The HTML Standard has since added processing instructions to the HTML tokenizer: <?target data> with a target other than xml and xml-stylesheet is a processing instruction ending at the first ">" (see the processing instruction open state). In HTML mode, the Reader still reads all of them as bogus comments, as the HTML tokenizer did before, except that with XML syntax, those whose first ">" follows "?" are processing instructions (see Reader.SetXMLSyntax).

</> is a bogus comment too, with no data and a Raw of "</>", so a Transformer writes it unchanged, but HTML parsers read it as nothing: the HTML tokenizer ignores it in the end tag open state, and gives no comment. The last chunk of a longer bogus comment (see Token.Continued) can have the same Raw.

func (*Token) Clone

func (t *Token) Clone() *Token

Clone returns a copy of the token which stays valid after the Reader advances: its byte slices don't alias the Reader's buffer, and NS and AttrNamespace use the namespace declarations in scope when the copy was made. Everything else observable about the copy is the same as for the token, including Pos and StartPos, so copies can be given to consumers of tokens later (e.g., held back until the tokens after them are known). A TokenBuffer copies tokens without allocating for each.

func (*Token) Continued

func (t *Token) Continued() bool

Continued reports whether the token continues the text, comment or CDATA section of the previous token.

func (*Token) Data

func (t *Token) Data() []byte

Data returns the undecoded contents of a Text, Comment, CDATA, Doctype or ProcInst token (excluding the delimiters). For a ProcInst token, it is the contents after the target (see Token.Name), including the whitespace separating them, which XML parsers skip (see Token.AppendText).

func (*Token) HTMLRules

func (t *Token) HTMLRules() bool

HTMLRules reports whether a start or end tag follows HTML rules: names are compared ASCII case-insensitively, and the HTML rules for void, raw text and RCDATA elements, optional end tags and character references apply. This is the case for the tokens read in HTML mode, and in Universal mode for the tokens of documents read like in HTML mode (see Reader.EffectiveMode), whatever the kind of token: it reports false for the tokens read in Universal mode before the rules are chosen.

func (*Token) HasAttr

func (t *Token) HasAttr(name string) bool

HasAttr reports whether the tag has an attribute named name (see Attr.Is).

func (*Token) Implied

func (t *Token) Implied() bool

Implied reports whether the token is an implied end tag, which closes an element without an end tag in the input. Its Raw is empty and its Pos is the position of the token which caused it, or the end of the input.

func (*Token) InRawText

func (t *Token) InRawText() bool

InRawText reports whether a Text token is the contents of a raw text element (script, style, xmp, iframe, noembed, noframes, plaintext, and noscript if scripting is enabled), where character references aren't decoded.

func (*Token) Incomplete

func (t *Token) Incomplete() bool

Incomplete reports whether the token was cut off by the end of the input. An incomplete start tag opens no element, and leaves out the attribute being read when the input ended (e.g., <b class=x has no attributes).

func (*Token) Is

func (t *Token) Is(name string) bool

Is reports whether the token is a start or end tag named name. Under HTML rules (see Token.HTMLRules), the tag name is read like the HTML tokenizer reads it in the tag name state, with NUL replaced with U+FFFD, and compared ASCII case-insensitively; otherwise, names are compared exactly. HTML parsers ASCII-lowercase tag names, so a lowercase name matches the name they give the element.

func (*Token) Kind

func (t *Token) Kind() Kind

Kind returns the kind of token.

func (*Token) Local

func (t *Token) Local() []byte

Local returns the tag name as written without the namespace prefix (see Token.AppendLocalName). Under HTML rules (see Token.HTMLRules), where HTML parsers treat a name like "o:p" as a whole, it is the whole name, and Prefix returns nil.

func (*Token) LocalIs

func (t *Token) LocalIs(name string) bool

LocalIs reports whether the local name of a start or end tag (see Token.Local) is name, compared like Token.Is.

func (*Token) NS

func (t *Token) NS() string

NS returns the namespace URI of a start or end tag, or an empty string.

Under HTML rules (see Token.HTMLRules), this is the namespace an HTML parser would put the element in (XHTML, SVG or MathML); namespace declarations are ignored (see Reader.LookupPrefix). Under XML rules, it is resolved from the namespace declarations.

Example

In HTML mode, NS gives the namespace HTML parsers put an element in. AppendLocalName gives a name to switch on without allocating, lowercased in HTML mode, where names are case-insensitive, and AdjustSVGTagName the case HTML parsers give SVG names.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<DIV><svg viewBox="0 0 10 10"><FOREIGNOBJECT><p>text</p></FOREIGNOBJECT></svg><math><mi>x</mi></math></DIV>`

	r := htmlstream.NewReader(strings.NewReader(page))
	var buf [16]byte
	for r.Next() {
		tok := r.Token()
		if tok.Kind() != htmlstream.StartTag {
			continue
		}
		local := tok.AppendLocalName(buf[:0])
		if tok.NS() == htmlstream.NamespaceSVG {
			local = []byte(htmlstream.AdjustSVGTagName(string(local)))
		}
		fmt.Printf("%-13s %-13s %s\n", tok.Name(), local, tok.NS())
	}
}

Output:
DIV           div           http://www.w3.org/1999/xhtml
svg           svg           http://www.w3.org/2000/svg
FOREIGNOBJECT foreignObject http://www.w3.org/2000/svg
p             p             http://www.w3.org/1999/xhtml
math          math          http://www.w3.org/1998/Math/MathML
mi            mi            http://www.w3.org/1998/Math/MathML

func (*Token) Name

func (t *Token) Name() []byte

Name returns the tag name or processing instruction target as written. Under HTML rules (see Token.HTMLRules), HTML parsers ASCII-lowercase tag names and replace NUL with U+FFFD in them: Token.Is compares names like they do, and Token.AppendLocalName returns the name they give the element.

func (*Token) NumAttrs

func (t *Token) NumAttrs() int

NumAttrs returns the number of attributes of a tag, including duplicates.

func (*Token) Opens

func (t *Token) Opens() bool

Opens reports whether a start tag opened an element, which is closed by a later (possibly implied) EndTag token. Void elements, and self-closing tags where self-closing syntax is honored, don't open an element, except for void elements in HTML mode with Reader.SetXMLSyntax and in documents read like in XML mode in Universal mode (see Element model in the package documentation).

func (*Token) Partial

func (t *Token) Partial() bool

Partial reports whether the text, comment or CDATA section continues in the next token.

func (*Token) Pos

func (t *Token) Pos() Pos

Pos returns the position of the token in the input.

func (*Token) Prefix

func (t *Token) Prefix() []byte

Prefix returns the namespace prefix of a tag name as written, or nil (always under HTML rules, see Local). A name starting with a colon (e.g., :x, under XML rules) has none, as in XML parsers. Other names are split at the first colon, also where they aren't namespace-well-formed, which encoding/xml reads as local names without a prefix: <x:/> has the prefix x and an empty local name, and <a:b:c> the prefix a and the local name b:c.

func (*Token) Raw

func (t *Token) Raw() []byte

Raw returns the exact bytes of the token in the input. It is empty for implied end tags.

func (*Token) SelfClosing

func (t *Token) SelfClosing() bool

SelfClosing reports whether a start tag uses self-closing syntax. In HTML mode, it only closes elements inside svg and math, like in HTML parsers, unless Reader.SetXMLSyntax is enabled (see Token.Opens).

func (*Token) StartPos

func (t *Token) StartPos() Pos

StartPos returns the position of the start tag of the element closed by an end tag (explicit or implied), or -1 for stray end tags and other tokens.

func (*Token) Stray

func (t *Token) Stray() bool

Stray reports whether the token is an end tag without a matching open element.

func (*Token) String

func (t *Token) String() string

String returns a description of the token for debugging.

func (*Token) Unsound

func (t *Token) Unsound() bool

Unsound reports whether the token is structurally unsound: an implied end tag for an element whose end tag may not be omitted (it was closed by an ancestor's end tag, a misnested end tag, or the end of the input), a tag which was cut off or contains quotes or '<' in attribute names or unquoted values (usually a missing quote), a construct cut off by the end of the input, or the text of a raw text element without an end tag.

func (*Token) UnsoundError

func (t *Token) UnsoundError() error

UnsoundError returns a SyntaxError saying why the token is structurally unsound (see Token.Unsound), or nil if it isn't. These are the errors the Reader and the Transformer return in strict mode.

func (*Token) Untracked

func (t *Token) Untracked() bool

Untracked reports whether a start tag opened an element nested past the stack limits (see Reader.SetMaxDepth and Reader.SetMaxStackBytes). An untracked element is counted in Reader.Depth but its name isn't kept: it is closed by the next end tag (whatever its name) when it is the innermost open element (a tracked element can be open inside it when it is untracked past the stack size limit), by the end tag of a tracked element below it, or at the end of the input. A void element under XML rules or with XML syntax is closed like a tracked one instead: by its end tag if it is the next token or follows whitespace, and otherwise by an implied end tag before the next token, which isn't unsound (see Element model in the package documentation). Its EndTag has a StartPos of -1 and isn't stray; when implied, it also has an empty Name. Its end is still known, so Reader.Skip, Reader.Children and the Transformer's element operations work on it: it is the first EndTag which isn't stray with the same Reader.Depth as the start tag. In strict mode, they return a SyntaxError if it is closed without an end tag (except for a void element), since whether its end tag may be omitted isn't known. Its contents are read as if they were in the innermost tracked element (its namespace and rules), without the namespace declarations of the untracked elements, except for the contents of raw text and RCDATA elements (see Element model in the package documentation).

type TokenBuffer

type TokenBuffer struct {
	// contains filtered or unexported fields
}

A TokenBuffer holds copies of tokens (see Token.Clone), e.g., to decide what to do with them after seeing the tokens which follow. The copies share the buffer's memory, which is reused after Reset, so appending to a buffer allocates only when it grows. The zero value is an empty buffer ready to use.

Example

A TokenBuffer keeps copies of tokens after the Reader advances, here to list the links after reading the whole document.

package main

import (
	"fmt"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p><a href=/a>A</a>, <a href=/b>B</a> and <a href=/c>C</a>.</p>`

	r := htmlstream.NewReader(strings.NewReader(page))
	var links htmlstream.TokenBuffer
	for r.Find(htmlstream.Tag("a", htmlstream.HasAttr("href"))) {
		links.Append(r.Token())
	}
	if err := r.Err(); err != nil {
		panic(err)
	}
	fmt.Println(links.Len(), "links,", links.Size(), "bytes")
	for tok := range links.All() {
		href, _ := tok.Attr("href")
		fmt.Printf("%d: %s\n", tok.Pos(), href.Value())
	}
}

Output:
3 links, 33 bytes
3: /a
21: /b
42: /c

func (*TokenBuffer) All

func (b *TokenBuffer) All() iter.Seq[*Token]

All returns an iterator over the tokens, in order.

func (*TokenBuffer) Append

func (b *TokenBuffer) Append(t *Token)

Append appends a copy of t.

func (*TokenBuffer) At

func (b *TokenBuffer) At(i int) *Token

At returns the i-th token (0 is the first appended). It is valid until the buffer is reset. It panics if i is out of range [0, Len()).

func (*TokenBuffer) Len

func (b *TokenBuffer) Len() int

Len returns the number of tokens in the buffer.

func (*TokenBuffer) Reset

func (b *TokenBuffer) Reset()

Reset removes all tokens, keeping the memory. The tokens returned before are no longer valid.

func (*TokenBuffer) Size

func (b *TokenBuffer) Size() int

Size returns the number of bytes of raw markup in the buffer.

type Transformer

type Transformer struct {
	// contains filtered or unexported fields
}

Transformer reads tokens like a Reader and writes them to an io.Writer, applying edits made to the current token. Tokens without edits are written unchanged, so if no edits are made, the output is identical to the input. It has the methods of a Reader, and Transformer.Reader returns the Reader it reads with.

Each token is written when the Transformer advances past it, including tokens skipped by Transformer.Skip, Transformer.Children, Transformer.Find and Transformer.ReadText. Edits apply to the current token, and edits to a start tag can also affect the element's end tag.

Attribute edits apply to the start tag as edited so far: attributes are matched by the names given by RenameAttr, and attributes added by SetAttr and SetAttrAt can be changed, renamed and removed like the original ones, keeping their position.

The values set are written in double quotes, with &, <, >, ", ', tabs, newlines and carriage returns written as character references, so XML parsers don't change the whitespace to spaces (attribute-value normalization) and HTML parsers don't change carriage returns to newlines (preprocessing the input stream). On elements read under XML rules (those for which Token.HTMLRules reports false: in XML mode, and in documents read like in XML mode in Universal mode) in documents declared as XML 1.1, NEL and U+2028, which are line ends in XML 1.1, and the control characters other than tab, newline, carriage return and NUL, which it only allows as character references, are written as character references too (see Modes in the package documentation). HTML and XML parsers read the values as given, except for the characters XML doesn't allow (in XML 1.1, NUL), which aren't replaced (see AppendXMLChars). Names passed to the editing methods are written as given, without checks (see ValidXMLName). Markup passed as a []byte (e.g., to Replace, InsertBefore, InsertAfter, Wrap and HoldDiscard) is copied, so the caller can reuse the slice.

Output can be held back (see Transformer.Hold) to decide what to do with it after seeing the tokens which follow.

The output is buffered: call Transformer.Close after the last token, or the output can be incomplete. Like a Reader, a Transformer isn't safe for concurrent use, and transforms a single input.

The editing methods apply to the current token, and each one documents the kinds of token it accepts. Called without a current token or on another kind of token, they return an error wrapping ErrInvalidOp and do nothing else.

Errors are sticky and can be retrieved with Err or Close, except ErrInvalidOp, ErrIncompleteTag, ErrHeldOpen and ErrHoldReleased, which only mean the operation wasn't done. On error, the output is incomplete.

Example (Xml)

In XML mode, a Transformer edits XML documents, keeping everything it doesn't edit as written: here, the XML declaration, a doctype with an internal subset, processing instructions, CDATA sections, entity references and namespace prefixes. Elements named like HTML elements, like script and br, are ordinary elements.

package main

import (
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const doc = `<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE feed [<!ENTITY ver "2.1">]>
<?xml-stylesheet href="feed.xsl" type="text/xsl"?>
<f:feed xmlns:f="urn:feed" f:version='&ver;'>
  <f:entry id="1"><script><![CDATA[a < b]]></script></f:entry><f:entry id="2" draft="yes"><br>x</br></f:entry>
  <f:entry id="3"><br/></f:entry>
</f:feed>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(doc))
	x.SetMode(htmlstream.XML)
	for x.Next() {
		tok := x.Token()
		var err error
		switch {
		case tok.Kind() == htmlstream.ProcInst && string(tok.Name()) == "xml-stylesheet":
			err = x.Replace(htmlstream.AppendProcInst(nil, "xml-stylesheet", `href="new.xsl" type="text/xsl"`))
		case tok.Kind() == htmlstream.StartTag && tok.Is("f:entry") && tok.HasAttr("draft"):
			err = x.Remove()
		case tok.Kind() == htmlstream.CDATA:
			err = x.Replace(htmlstream.AppendCDATA(nil, "if (a[b[0]]>c) {}"))
		}
		if err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE feed [<!ENTITY ver "2.1">]>
<?xml-stylesheet href="new.xsl" type="text/xsl"?>
<f:feed xmlns:f="urn:feed" f:version='&ver;'>
  <f:entry id="1"><script><![CDATA[if (a[b[0]]]]><![CDATA[>c) {}]]></script></f:entry>
  <f:entry id="3"><br/></f:entry>
</f:feed>

func NewTransformer

func NewTransformer(w io.Writer, r io.Reader) *Transformer

NewTransformer returns a Transformer reading from r in HTML mode and writing to w. For XML documents, set XML mode with SetMode (see Modes in the package documentation).

func (*Transformer) All

func (t *Transformer) All() iter.Seq[*Token]

See Reader.All.

func (*Transformer) Children

func (t *Transformer) Children() iter.Seq[*Token]

See Reader.Children.

func (*Transformer) Close

func (t *Transformer) Close() error

Close writes the remaining tokens, the output still held back (see Transformer.Hold) and anything inserted with InsertAtEnd, flushes the output and returns the first error encountered. It doesn't close the io.Writer passed to NewTransformer. Calling Close again returns the same error, and writes nothing more unless InsertAtEnd was called since.

func (*Transformer) Depth

func (t *Transformer) Depth() int

See Reader.Depth.

func (*Transformer) EffectiveMode

func (t *Transformer) EffectiveMode() Mode

See Reader.EffectiveMode.

func (*Transformer) Empty

func (t *Transformer) Empty() error

Empty removes the contents of the element opened by the current start tag and its end tag, and writes it as an empty element. The current token becomes the element's end tag. Markup inserted after the start tag (see Transformer.InsertAfter) stays inside the element, and markup inserted before or after the end tag is written after the element.

HTML elements under HTML rules (see Token.HTMLRules) other than void elements are written as <x ...></x>, since HTML parsers ignore self-closing syntax on them. Other elements are written with self-closing syntax, like <x .../>, which XML parsers and, for SVG and MathML elements, HTML parsers read as empty, or as <x ...></x> to keep markup inserted after the start tag inside the element. Void elements are always written with self-closing syntax, with the markup inserted after the start tag following them. The stray end tag of a void element under XML rules or with XML syntax (e.g., the </br> in <br>x</br>, see Transformer.Remove) stays in the output, after the contents following the element.

The current token must be a start tag.

Example

Empty removes the contents of an element. HTML elements keep their end tag, and SVG elements are written with self-closing syntax.

package main

import (
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p>Text</p><script>tracker()</script><svg><circle r=1><title>x</title></circle></svg>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Next() {
		tok := x.Token()
		if tok.Kind() == htmlstream.StartTag && (tok.Is("script") || tok.Is("circle")) {
			if err := x.Empty(); err != nil {
				panic(err)
			}
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<p>Text</p><script></script><svg><circle r=1 /></svg>

func (*Transformer) Err

func (t *Transformer) Err() error

See Reader.Err.

func (*Transformer) ExpandSelfClosing

func (t *Transformer) ExpandSelfClosing() error

ExpandSelfClosing rewrites a self-closing start tag like <x/> as <x></x>. It removes all the "/" characters directly before the ">", since HTML parsers read <x//> as self-closing too. If the self-closing syntax isn't honored (e.g., <div/> in HTML mode, which opens an element), or the element is an HTML void element under HTML rules (e.g., <br/>), only these "/" characters are removed. The end tag is part of the start tag's edits: Remove, Replace and Unwrap remove it with the start tag.

The current token must be a start tag.

func (*Transformer) Find

func (t *Transformer) Find(m Matcher) bool

See Reader.Find.

func (*Transformer) Held

func (t *Transformer) Held(h Hold) []byte

Held returns the output held back by the hold h so far, including that of the holds inside it, or nil if h isn't open. It doesn't include the current token. It is valid until the Transformer advances or a hold ends.

func (*Transformer) Hold

func (t *Transformer) Hold() (Hold, error)

Hold starts holding back the output from the current token on: the current token and the following ones are written with their edits, like otherwise, but to a buffer instead of the output, until the hold ends with Transformer.HoldRelease or Transformer.HoldDiscard. The output of element operations (Remove, Empty, Skip, Children) on held tokens is held too.

A hold started while another is open is inside it, and must end first: its output then stays in the enclosing hold. HoldRelease and HoldDiscard of a hold with an open hold inside it return an error wrapping ErrInvalidOp, and so do those of the zero Hold and of a hold of another Transformer. Those of a hold which has ended return ErrHoldReleased. Holds can end after the end of the input (when Next returns false), before Close. Close ends the holds still open, writing the output they hold back.

The edits of a held token can't be changed after the Transformer advances past it, but the edits of an element's end tag (from Wrap, Rename and Unwrap of its start tag) apply when the end tag is written, which may be after the hold ends.

The held output is limited to DefaultMaxHold by default (see Transformer.SetMaxHold): exceeding it is an error (ErrHoldLimit). To give up on a hold at a smaller size instead, check the size of Transformer.Held and end the hold with HoldRelease or HoldDiscard.

The current token can be of any kind.

Example

Hold holds back the output until the tokens which follow decide what to do with it, here to remove the paragraphs without text.

package main

import (
	"bytes"
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p>One</p><p> <!-- empty --> </p><p><br></p><p>Two <b>three</b></p>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	var h htmlstream.Hold // the hold of the current paragraph, if not zero
	for x.Next() {
		tok := x.Token()
		var err error
		switch {
		case tok.Kind() == htmlstream.StartTag && tok.Is("p"):
			h, err = x.Hold()
		case h == htmlstream.Hold{}:
		case tok.Kind() == htmlstream.Text && len(bytes.TrimSpace(tok.Raw())) != 0:
			err = x.HoldRelease(h)
			h = htmlstream.Hold{}
		case tok.Kind() == htmlstream.EndTag && tok.Is("p"):
			// The end tag isn't held back: remove it, then the held output.
			if err = x.Remove(); err == nil {
				err = x.HoldDiscard(h, nil)
			}
			h = htmlstream.Hold{}
		}
		if err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<p>One</p><p>Two <b>three</b></p>

func (*Transformer) HoldDiscard

func (t *Transformer) HoldDiscard(h Hold, raw []byte) error

HoldDiscard ends the hold h, writing raw (which may be nil) instead of the output held back. The current token isn't part of the held output, so markup to put before the held output is passed as part of raw, e.g., HoldDiscard(h, append(prefix, t.Held(h)...)); InsertBefore on the current token writes after the held output. If a hold encloses h, raw is held back in it, and counts toward the limit (see Transformer.SetMaxHold).

If the output held back contains the start tag of an element which is still open and whose end tag has edits (from Wrap, Rename or Unwrap), the end tag would no longer match the start tag written, so HoldDiscard returns ErrHeldOpen and the hold doesn't end. The end tags of elements without such edits are written as usual, like after Replace(nil) on the start tag: use Remove on the end tag to drop it.

See Transformer.Hold for the other errors.

Example

HoldDiscard ends a hold, replacing the output held back, here to put runs of images in a gallery element once the token after them is known, or after the end of the input.

package main

import (
	"bytes"
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p>One</p><img src=a.png><img src=b.png>
<img src=c.png><p>Two</p><img src=d.png><p>Three</p><img src=e.png><img src=f.png>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	var h htmlstream.Hold
	images := 0
	// endRun ends the hold of a run of images, putting them in a gallery if
	// there are several.
	endRun := func() error {
		n := images
		images = 0
		if n == 1 {
			return x.HoldRelease(h)
		}
		gallery := append([]byte(`<div class="gallery">`), x.Held(h)...)
		return x.HoldDiscard(h, append(gallery, "</div>"...))
	}
	for x.Next() {
		tok := x.Token()
		var err error
		switch {
		case tok.Kind() == htmlstream.StartTag && tok.Is("img"):
			if images == 0 {
				h, err = x.Hold()
			}
			images++
		case images == 0:
		case tok.Kind() == htmlstream.Text && len(bytes.TrimSpace(tok.Raw())) == 0:
		default:
			// The current token isn't held back: it is written after.
			err = endRun()
		}
		if err != nil {
			panic(err)
		}
	}
	if images != 0 {
		if err := endRun(); err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<p>One</p><div class="gallery"><img src=a.png><img src=b.png>
<img src=c.png></div><p>Two</p><img src=d.png><p>Three</p><div class="gallery"><img src=e.png><img src=f.png></div>

func (*Transformer) HoldRelease

func (t *Transformer) HoldRelease(h Hold) error

HoldRelease ends the hold h, writing the output held back (or keeping it in the enclosing hold). See Transformer.Hold for the errors.

func (*Transformer) Holding

func (t *Transformer) Holding() int

Holding returns the number of holds which haven't ended.

func (*Transformer) InsertAfter

func (t *Transformer) InsertAfter(raw []byte) error

InsertAfter writes raw after the current token. For a start tag which opens an element, this is inside the element, before its contents: Transformer.Remove removes it with the element, and Transformer.Empty keeps it in the element.

An implied end tag (see Token.Implied) writes nothing, so HTML parsers read markup inserted after it as inside the element, which they close later (e.g., after the implied end tag of the p in <div><p>x</div>). To close the element first, replace the implied end tag with an end tag (see AppendEndTag).

The current token can be of any kind.

func (*Transformer) InsertAtEnd

func (t *Transformer) InsertAtEnd(raw []byte)

InsertAtEnd writes raw at the end of the output, after the last token, when the Transformer is closed. After an error, which Close returns, it does nothing.

func (*Transformer) InsertBefore

func (t *Transformer) InsertBefore(raw []byte) error

InsertBefore writes raw before the current token, outside the wraps of a start tag (see Transformer.Wrap).

The current token can be of any kind.

Example

InsertBefore and InsertAfter write markup around a token. Markup written after a start tag is inside the element. InsertAtEnd writes markup at the end of the output.

package main

import (
	"errors"
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<h1>Title</h1><p>Text<br>more text.</p>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Next() {
		tok := x.Token()
		if tok.Kind() != htmlstream.StartTag {
			continue
		}
		var err error
		switch {
		case tok.Is("h1"):
			err = errors.Join(
				x.InsertBefore([]byte(`<a id="top"></a>`)),
				x.InsertAfter([]byte("&sect; ")), // inside the element
			)
		case tok.Is("br"):
			err = x.InsertAfter([]byte("\n")) // a void element has no contents
		}
		if err != nil {
			panic(err)
		}
	}
	x.InsertAtEnd([]byte(`<footer><a href="#top">Top</a></footer>`))
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<a id="top"></a><h1>&sect; Title</h1><p>Text<br>
more text.</p><footer><a href="#top">Top</a></footer>

func (*Transformer) LookupPrefix

func (t *Transformer) LookupPrefix(prefix string) (string, bool)

See Reader.LookupPrefix.

func (*Transformer) Mode

func (t *Transformer) Mode() Mode

See Reader.Mode.

func (*Transformer) Next

func (t *Transformer) Next() bool

See Reader.Next.

func (*Transformer) ReadText

func (t *Transformer) ReadText(dst []byte, limit int) ([]byte, bool)

See Reader.ReadText.

Example

Text longer than the chunk size is split into several tokens (see Token.Partial), here with a small chunk size. To replace all of it, hold back the output from the first token, read the text, replace the last token, and discard the held output.

package main

import (
	"bytes"
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p>This text &amp; its character reference are longer than the chunk size.</p><p>Short.</p>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	x.SetChunkSize(8)
	for x.Next() {
		if x.Token().Kind() != htmlstream.Text {
			continue
		}
		h, err := x.Hold()
		if err != nil {
			panic(err)
		}
		text, _ := x.ReadText(nil, 0) // the current token is now the last one
		if err := x.Replace(htmlstream.AppendEscapeText(nil, bytes.ToUpper(text))); err != nil {
			panic(err)
		}
		if err := x.HoldDiscard(h, nil); err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<p>THIS TEXT &amp; ITS CHARACTER REFERENCE ARE LONGER THAN THE CHUNK SIZE.</p><p>SHORT.</p>

func (*Transformer) Reader

func (t *Transformer) Reader() *Reader

Reader returns the Reader the Transformer reads with, for functions taking a Reader (e.g., htmlutil.InnerText). Tokens read with it are written like those read with the Transformer's methods, and edits apply to its current token.

func (*Transformer) Remove

func (t *Transformer) Remove() error

Remove removes the current token. If it is a start tag which opens an element, the element's contents and end tag are also removed, and the current token becomes the element's end tag. Markup inserted before the start tag and the markup passed to Transformer.Wrap are still written; markup inserted after the start tag is inside the element and is removed with it.

A void element under XML rules or with XML syntax (e.g., <br>, see Element model in the package documentation) opens an element only up to its end tag if that follows it directly or after whitespace, so only then is the end tag removed with it: in <br>x</br>, the end tag is stray, and stays in the output, as it does for Transformer.Unwrap, Transformer.Rename and Transformer.Empty.

The current token can be of any kind.

Example

Element operations on a start tag also apply to the element's end tag and contents.

package main

import (
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<div class=ad>Buy <b>now</b>!</div><p>Text<span class=ad>sponsored</span>.</p><!-- comment -->`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Next() {
		tok := x.Token()
		var err error
		switch {
		case tok.Kind() == htmlstream.Comment:
			err = x.Remove()
		case htmlstream.HasClass("ad")(tok):
			err = x.Remove() // removes the contents and the end tag too
		}
		if err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<p>Text.</p>

func (*Transformer) RemoveAttr

func (t *Transformer) RemoveAttr(name string) error

RemoveAttr removes all attributes named name (see Attr.Is) from the current start tag.

The current token must be a start tag.

func (*Transformer) RemoveAttrs

func (t *Transformer) RemoveAttrs(fn func(Attr) bool) error

RemoveAttrs removes the attributes matching fn from the current start tag. fn is called with each attribute as edited so far, including added ones. An edited or added attribute passed to fn is only valid during the call.

The current token must be a start tag.

func (*Transformer) Rename

func (t *Transformer) Rename(name string) error

Rename renames the current start tag and the end tag of the element it opens, if any. Implied end tags are not written. A name with a namespace prefix needs a declaration of the prefix in scope, which Rename doesn't add (e.g., add an xmlns:prefix attribute with SetAttr). The stray end tag of a void element under XML rules or with XML syntax (e.g., the </br> in <br>x</br>, see Transformer.Remove) isn't renamed.

The current token must be a start tag.

Example

Rename renames the start tag of an element and its end tag. Implied end tags aren't written.

package main

import (
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p><b>Bold</b> and <i>italic</I>.<p><b>Second`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Next() {
		tok := x.Token()
		if tok.Kind() != htmlstream.StartTag {
			continue
		}
		var err error
		switch {
		case tok.Is("b"):
			err = x.Rename("strong")
		case tok.Is("i"):
			err = x.Rename("em")
		}
		if err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<p><strong>Bold</strong> and <em>italic</em>.<p><strong>Second

func (*Transformer) RenameAttr

func (t *Transformer) RenameAttr(name, newName string) error

RenameAttr renames the attributes named name (see Attr.Is) on the current start tag to newName, keeping their position and value.

The current token must be a start tag.

func (*Transformer) RenameEnd

func (t *Transformer) RenameEnd(name string) error

RenameEnd renames the current end tag. Implied end tags are not written.

The current token must be an end tag.

func (*Transformer) Replace

func (t *Transformer) Replace(raw []byte) error

Replace replaces the current token with raw. If the current token is a start tag which opens an element, only the start tag is replaced.

The current token can be of any kind.

func (*Transformer) Scripting

func (t *Transformer) Scripting() bool

See Reader.Scripting.

func (*Transformer) SetAttr

func (t *Transformer) SetAttr(name, value string) error

SetAttr sets the value of the first attribute named name (see Attr.Is) on the current start tag, keeping its position, or adds one at the end like Transformer.SetAttrAt with position -1. The value is escaped as described in Transformer.

The current token must be a start tag.

Example

Attribute edits keep the other attributes as they are written. Positions passed to SetAttrAt count the attributes of the input.

package main

import (
	"bytes"
	"errors"
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<img src=cat.jpg WIDTH=100 data-x=1 data-y=2 alt='A cat'>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Find(htmlstream.Tag("img")) {
		edits := []error{
			x.SetAttr("alt", `A "cat"`),     // replaces the value, keeping the position
			x.SetAttr("loading", "lazy"),    // adds an attribute at the end
			x.SetAttrAt("id", "photo", 0),   // adds an attribute before the first one
			x.RenameAttr("width", "height"), // keeps the value
			x.RemoveAttrs(func(a htmlstream.Attr) bool {
				return bytes.HasPrefix(a.Name(), []byte("data-"))
			}),
		}
		if err := errors.Join(edits...); err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<img id="photo" src=cat.jpg height=100 alt="A &quot;cat&quot;" loading="lazy">

func (*Transformer) SetAttrAt

func (t *Transformer) SetAttrAt(name, value string, at int) error

SetAttrAt removes the attributes named name (see Attr.Is) from the current start tag and adds one with the value at position at in the original attributes: 0 is before the first, 1 before the second, -1 after the last, -2 before the last. Attributes added at the same position are written in the order they were added. The value is escaped as described in Transformer.

Positions count the attributes of the start tag as Token.Attrs returns them, including duplicates, so they don't change with other edits to the tag. With n attributes, at must be between -(n+1) and n; otherwise, SetAttrAt returns an error wrapping ErrInvalidOp.

The current token must be a start tag.

func (*Transformer) SetChunkSize

func (t *Transformer) SetChunkSize(n int)

See Reader.SetChunkSize.

func (*Transformer) SetKeepAttrs

func (t *Transformer) SetKeepAttrs(v bool)

See Reader.SetKeepAttrs.

func (*Transformer) SetMaxDepth

func (t *Transformer) SetMaxDepth(n int)

See Reader.SetMaxDepth.

func (*Transformer) SetMaxHold

func (t *Transformer) SetMaxHold(n int)

SetMaxHold sets the maximum size of the output held back (see Transformer.Hold). The default is DefaultMaxHold. Negative values are treated as 0, with which holding back any output is an error.

func (*Transformer) SetMaxStackBytes

func (t *Transformer) SetMaxStackBytes(n int)

See Reader.SetMaxStackBytes.

func (*Transformer) SetMaxTokenSize

func (t *Transformer) SetMaxTokenSize(n int)

See Reader.SetMaxTokenSize.

func (*Transformer) SetMode

func (t *Transformer) SetMode(m Mode)

See Reader.SetMode.

func (*Transformer) SetScripting

func (t *Transformer) SetScripting(v bool)

See Reader.SetScripting.

func (*Transformer) SetStrict

func (t *Transformer) SetStrict(v bool)

See Reader.SetStrict.

func (*Transformer) SetXMLSyntax

func (t *Transformer) SetXMLSyntax(v bool)

See Reader.SetXMLSyntax.

func (*Transformer) Skip

func (t *Transformer) Skip() error

See Reader.Skip.

func (*Transformer) Stack

func (t *Transformer) Stack() iter.Seq[Element]

See Reader.Stack.

func (*Transformer) Token

func (t *Transformer) Token() *Token

See Reader.Token.

func (*Transformer) Unwrap

func (t *Transformer) Unwrap() error

Unwrap removes the current start tag and, if it opens an element, the element's end tag, keeping the element's contents.

The current token must be a start tag.

Example

Unwrap removes the tags of an element, keeping its contents.

package main

import (
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<p><font color=red>Warning:</font> do not <span>touch</span>.`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Next() {
		tok := x.Token()
		if tok.Kind() == htmlstream.StartTag && (tok.Is("font") || tok.Is("span")) {
			if err := x.Unwrap(); err != nil {
				panic(err)
			}
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<p>Warning: do not touch.

func (*Transformer) Wrap

func (t *Transformer) Wrap(start, end []byte) error

Wrap writes start before the current start tag, and end after the end tag of the element it opens (or after the start tag if it doesn't open an element). Later wraps are inside earlier ones. Markup inserted before the start tag (see Transformer.InsertBefore), and after the element (with Transformer.InsertAfter at its end tag, or at the start tag if it doesn't open an element), is outside the wrap, whatever the order of the calls.

Under HTML rules, end is written after the element's end tag also when it is implied (see Token.Implied), where HTML parsers still have the element open, like markup inserted after it (see Transformer.InsertAfter): e.g., wrapping the p of <p>a<div>b</div> in a span gives <span><p>a</span><div>b</div>, where HTML parsers ignore the </span>, since the p is open, and put the div in the span. To close the element first, replace the implied end tag with an end tag (see AppendEndTag).

The current token must be a start tag.

Example

Wrap writes markup around an element, and Replace writes markup instead of a token.

package main

import (
	"os"
	"strings"

	"github.com/pgaskin/htmlstream"
)

func main() {
	const page = `<table><tr><td>1</td></tr></table><p>An <img src=x.png alt="image"> here.</p>`

	x := htmlstream.NewTransformer(os.Stdout, strings.NewReader(page))
	for x.Next() {
		tok := x.Token()
		if tok.Kind() != htmlstream.StartTag {
			continue
		}
		var err error
		switch {
		case tok.Is("table"):
			err = x.Wrap([]byte(`<div class="scroll">`), []byte(`</div>`))
		case tok.Is("img"):
			alt, _ := tok.Attr("alt")
			err = x.Replace(htmlstream.AppendElement(nil, "em", alt.Value()))
		}
		if err != nil {
			panic(err)
		}
	}
	if err := x.Close(); err != nil {
		panic(err)
	}
}

Output:
<div class="scroll"><table><tr><td>1</td></tr></table></div><p>An <em>image</em> here.</p>