textanchor

README

textanchor

Durable references to spans of text in Markdown documents — anchors that survive edits to the document they point into.

The problem

You want to attach something — a comment, a highlight, a suggestion — to a span of text in a Markdown file, and store it outside that file. A byte offset breaks as soon as anyone inserts a line above it. A plain text search breaks as soon as the same words appear twice.

textanchor describes a span by its content and its surroundings, then relocates it later by searching and scoring. When it cannot find a confident match, it says so rather than guessing.

Install

go get github.com/vloothuis/textanchor

Usage

import "github.com/vloothuis/textanchor"

// Create an anchor for document[start:end].
a, err := textanchor.New(document, start, end, nil)
if err != nil {
    return err
}
// ... serialise and store `a` alongside your annotation ...

// Later, against a possibly-edited document:
res := textanchor.Resolve(edited, a, nil)
switch {
case res.Orphaned:
    // The text is gone or changed too much. Show a.Quote as context —
    // don't silently drop the annotation.
    log.Printf("orphaned: %s", res.OrphanReason)
default:
    fmt.Println(edited[res.Range.Start:res.Range.End], res.Confidence)
}

Resolving many anchors against one document is cheaper with a prepared document, which builds the per-document structures once:

doc := textanchor.NewDocument(document)
for _, a := range anchors {
    res := doc.Resolve(a, nil)
    // ...
}

textanchor.ResolveAll(document, anchors, nil) does the same in one call.

What an anchor stores

Field	Purpose
Quote	the exact selected text
Prefix / Suffix	~50 characters either side, to disambiguate repeats
ContainingSentence	the full sentence, for quotes under 50 characters
HeadingContext	the nearest preceding ATX heading
ParagraphIndex	paragraph index within that heading's section

The structural fields are what let an anchor survive a rewrite that defeats quote matching alone.

How resolution works

1. Candidates. Every exact occurrence of Quote. If there are none, fall back to fuzzy matching, with a 0.8 penalty applied to the resulting score:

   • Within a paragraph: a sliding window at ±25% of the quote length.
   • Across paragraphs, for a quote that spans a blank line (a heading and its body, say): the quote's first paragraph is matched against the end of a document paragraph, its last paragraph against the start of a later one, and what lies between is compared as a whole. Each paragraph inserted or removed in the middle costs 0.15. Both ends, and their length-weighted combination, must reach 0.6 similarity, so surrounding context alone can never place an anchor.

   Both fuzzy kinds compete on score.

2. Scoring. Each candidate is scored:

   0.40*quote + 0.20*prefix + 0.20*suffix + 0.10*structural + 0.10*uniqueness
   

   where structural is itself 0.7*heading + 0.3*paragraph, and uniqueness is 1/occurrences — so a quote appearing once scores higher than one appearing five times.

3. Threshold. The best candidate wins if it scores at or above MinConfidence (default 0.5). Otherwise the result is orphaned, and Confidence still reports the best score achieved.

Similarity is length-adaptive: Levenshtein ratio below 100 characters, token-based Jaccard above. Quotes shorter than 5 characters are not fuzzy-matched at all, since short generic strings match everywhere.

Selections made over rendered HTML

A browser selection gives you rendered text — no markdown syntax, whitespace collapsed, possibly spanning several blocks — so its offsets do not correspond to the source. The quotefind subpackage bridges that gap by walking the goldmark AST and building a per-byte map from rendered text back to source offsets:

import "github.com/vloothuis/textanchor/quotefind"

start, end, ok := quotefind.Find(markdownSource, selectedText)
if ok {
    a, err := textanchor.New(markdownSource, start, end, nil)
    // ...
}

Use quotefind.FindWithContext when the selection may occur more than once, and quotefind.RenderedTextWithMapping if you need the rendered text and position map to do your own matching.

To highlight a resolved range that spans several blocks, split it with quotefind.Segments and wrap each part separately. One inline element cannot cross a block boundary: the HTML parser closes it at the first block end.

for _, seg := range quotefind.Segments(markdownSource, res.Range.Start, res.Range.End) {
    // wrap markdownSource[seg.Start:seg.End]
}

Each segment is cut to one block's inline content, so block markup (## , - , [ ] ) stays outside it. A code span the range touches is included whole, backticks and all; code blocks and HTML blocks are left out. The parser has GitHub-flavoured tables, task lists and strikethrough enabled, matching a GFM renderer.

Notes

• Offsets are byte offsets, matching Go's string indexing.
• Cross-paragraph fuzzy matching needs the quote in source form, with its blank lines. A quote without them is treated as one paragraph.
• Normalise your documents consistently. If your storage layer reformats Markdown (re-wrapping paragraphs, for instance), run the same normalisation before creating an anchor and before resolving one, or anchors will orphan on a purely cosmetic change.
• Treat orphaning as normal rather than exceptional, and surface orphans to the user with Quote as context. Research on annotation systems is consistent on this point: hiding a failed anchor makes users think their work was lost, and a bad guess is worse than an honest "not found".

Provenance

Extracted from margin, a CLI for annotating Markdown without modifying the source files.

License

MIT — see LICENSE.

Documentation

Overview

Package textanchor creates and resolves durable references to spans of text in Markdown documents.

An anchor describes a span by its content and surroundings rather than by a byte offset, so it can be relocated after the document has been edited. Each anchor stores the quoted text, ~50 characters of prefix and suffix context, the containing sentence (for short quotes), and structural context (the nearest preceding heading and the paragraph index within that section).

Resolution finds candidate matches, scores each by context similarity, and returns the best one along with a confidence score. When no candidate scores above the threshold the anchor is reported as orphaned rather than guessed at — callers are expected to surface orphans to the user, not to discard them.

Typical use:

a, err := textanchor.New(document, start, end, nil)
// ... persist a ...
res := textanchor.Resolve(editedDocument, a, nil)
if res.Orphaned {
	// show the anchor's Quote as context; do not silently drop it
}

Use ResolveAll to resolve many anchors against one document; it hoists the per-document structural analysis out of the per-anchor loop.

Anchors index the document by BYTE offset, matching Go's string indexing.

The subpackage quotefind maps a selection made over RENDERED markdown back to offsets in the markdown source, which is what a browser text selection needs before an anchor can be created from it.

Index

• Variables
• type Anchor
  • func New(document string, start, end int, opts *CreateOptions) (Anchor, error)
• type CreateOptions
  • func DefaultCreateOptions() *CreateOptions
• type Document
  • func NewDocument(text string) *Document
  • func (d *Document) Resolve(anchor Anchor, opts *ResolveOptions) ResolveResult
  • func (d *Document) Text() string
• type Range
• type ResolveOptions
  • func DefaultResolveOptions() *ResolveOptions
• type ResolveResult
  • func Resolve(document string, anchor Anchor, opts *ResolveOptions) ResolveResult
  • func ResolveAll(document string, anchors []Anchor, opts *ResolveOptions) []ResolveResult

Variables

var (
	ErrInvalidRange = errors.New("invalid range: start must be less than end")
	ErrOutOfBounds  = errors.New("range out of bounds")
	ErrEmptyQuote   = errors.New("quote cannot be empty")
)

Sentinel errors for the anchor package.

Types

type Anchor

type Anchor struct {
	// Quote is the exact text that was selected/annotated.
	Quote string `toml:"quote" json:"quote"`

	// Prefix is approximately 50 characters of text immediately before
	// the quote. Used for disambiguation when the quote appears multiple
	// times in the document.
	Prefix string `toml:"prefix" json:"prefix"`

	// Suffix is approximately 50 characters of text immediately after
	// the quote. Used for disambiguation.
	Suffix string `toml:"suffix" json:"suffix"`

	// ContainingSentence holds the full sentence containing the quote,
	// useful for word/phrase-level annotations where the quote itself
	// is very short.
	ContainingSentence string `toml:"containing_sentence,omitempty" json:"containing_sentence,omitempty"`

	// HeadingContext is the text of the nearest preceding heading,
	// providing structural context for resolution.
	HeadingContext string `toml:"heading_context,omitempty" json:"heading_context,omitempty"`

	// ParagraphIndex is the 0-based index of the paragraph within the
	// section defined by HeadingContext. -1 if not applicable.
	ParagraphIndex int `toml:"paragraph_index,omitempty" json:"paragraph_index,omitempty"`
}

Anchor represents a persistent reference to a span of text in a document. It stores enough context to relocate the text even after the document has been edited.

func New

func New(document string, start, end int, opts *CreateOptions) (Anchor, error)

New creates an anchor for the given text range in a document.

Parameters:

• document: The full text of the Markdown document
• start: Start byte offset of the selection (inclusive)
• end: End byte offset of the selection (exclusive)
• opts: Optional configuration (nil for defaults)

Returns an Anchor that can be serialized and later resolved.

type CreateOptions

type CreateOptions struct {
	// PrefixLen is the number of characters to capture before the quote.
	// Default: 50
	PrefixLen int

	// SuffixLen is the number of characters to capture after the quote.
	// Default: 50
	SuffixLen int

	// IncludeContainingSentence controls whether to extract the full
	// sentence for short quotes. Default: true for quotes < 50 chars.
	IncludeContainingSentence bool

	// IncludeStructuralContext controls whether to extract heading
	// and paragraph position. Default: true
	IncludeStructuralContext bool
}

CreateOptions configures anchor creation.

func DefaultCreateOptions

func DefaultCreateOptions() *CreateOptions

DefaultCreateOptions returns the default options for anchor creation.

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document is a document prepared for resolving anchors against it.

Resolving needs several per-document structures: the whitespace-collapsed form, the paragraph table and the heading positions. Building them is linear in the document and independent of the anchor, so a caller resolving many anchors against one document builds a Document once and calls Document.Resolve for each. Resolve and ResolveAll are wrappers that do exactly that.

A Document is immutable after construction and safe for concurrent use.

func NewDocument

func NewDocument(text string) *Document

NewDocument prepares text for resolving anchors against it.

func (*Document) Resolve

func (d *Document) Resolve(anchor Anchor, opts *ResolveOptions) ResolveResult

Resolve locates anchor in the document. See the package-level Resolve for the algorithm and the meaning of the result.

func (*Document) Text

func (d *Document) Text() string

Text returns the document the Document was prepared from.

type Range

type Range struct {
	Start int // Inclusive, 0-based byte offset
	End   int // Exclusive, 0-based byte offset
}

Range represents a character range in a document.

type ResolveOptions

type ResolveOptions struct {
	// MinConfidence is the minimum confidence score required to consider
	// an anchor resolved. Below this threshold, the anchor is orphaned.
	// Default: 0.5
	MinConfidence float64

	// PreferStructuralMatch gives weight to matches that are in the
	// same structural position (same heading, similar paragraph index).
	// Default: true
	PreferStructuralMatch bool
}

ResolveOptions configures anchor resolution.

func DefaultResolveOptions

func DefaultResolveOptions() *ResolveOptions

DefaultResolveOptions returns the default options for anchor resolution.

type ResolveResult

type ResolveResult struct {
	// Range is the located character range. Nil if orphaned.
	//
	// Matching ignores whitespace differences, so this span may be a different
	// length from the anchor's Quote — a quote the document has since rewrapped
	// resolves to a span containing the newline. Slice the document with
	// Range.Start and Range.End; do not compute the end as Start + len(Quote).
	Range *Range

	// Confidence is a score from 0.0 to 1.0 indicating how confident
	// we are in this match. 1.0 = exact match with perfect context.
	Confidence float64

	// Orphaned is true if the anchor could not be resolved.
	Orphaned bool

	// OrphanReason explains why resolution failed, if applicable.
	OrphanReason string
}

ResolveResult contains the result of attempting to resolve an anchor.

func Resolve

func Resolve(document string, anchor Anchor, opts *ResolveOptions) ResolveResult

Resolve attempts to locate an anchor in a document.

The resolution algorithm:

1. Find all exact matches for anchor.Quote
2. If there are none, find fuzzy matches: within one paragraph, and for a quote spanning several paragraphs, across them (see findCrossBlock)
3. Score each candidate by prefix/suffix similarity, structural context and uniqueness
4. Return the best match above the confidence threshold

If no match meets the threshold, returns Orphaned: true.

To resolve several anchors against one document, use NewDocument once and Document.Resolve per anchor.

func ResolveAll

func ResolveAll(document string, anchors []Anchor, opts *ResolveOptions) []ResolveResult

ResolveAll resolves multiple anchors against one document, preparing the document once.