extractors

package
v0.48.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package extractors defines the language-extractor protocol, the shared behavior every extractor embeds, and the registry that resolves a language name to its extractor.

An extractor answers directives out of source code. The build pipeline hands it a directive name, the directive's attributes, its body lines, the project's declared source paths and the project's base directory; the extractor returns the Markdown that replaces the directive. It also answers the three discovery questions the coverage and quality measurements ask of a file: which symbols it exports, what one symbol's parameters and return value are, and what the module's own documentation says.

The base

Base carries everything the language extractors share: the directive dispatch table, the no-op defaults for the optional protocol methods, and the formatting helpers that turn source-derived text into Markdown (symbol headings and spans, doc-comment renesting, Google-style docstring sections, the config-file tables, brace-block and comment-block scanning). A language package embeds Base, implements Detect, and populates the dispatch table.

The registry

Each language package registers its own factory from an init function, so a consumer links a language in by importing its package. KnownLanguages is the authority on which names selfdoc ships an extractor for: a name on that list with no registered factory is a wiring mistake and is reported as an error rather than silently answered with a stub. A name that is not on the list is a language selfdoc has no extractor for, and NewStub answers it in band -- an unsupported-language directive renders an error marker on the page instead of aborting the build, which is what the unsupported-source-language lint reports at check time.

Index

Constants

View Source 
const (
	// PySpaceClass is Python's \s: the ASCII whitespace characters, the four
	// ASCII separator controls, and the Unicode whitespace code points.
	PySpaceClass = util.PythonSpaceClass

	// PyNonSpaceClass is Python's \S, the complement of PySpaceClass.
	PyNonSpaceClass = util.PythonNonSpaceClass

	// PyWordClass is Python's \w where it matches an identifier character: a
	// letter, a digit or an underscore, in any script.
	PyWordClass = util.PythonWordClass
)

The three character classes below are the exported spellings of Python's \s, \S and \w for a ported regexp. Go's versions of all three are ASCII-only, where Python's are Unicode-aware for text patterns, so a pattern copied across unchanged would quietly stop matching a no-break space or a non-Latin identifier. Every extractor ports Python regexes, so the classes are exported here rather than restated in each one.

The values are util's: one authority for what Python's classes denote, aliased under the names the extractors were written against.

Variables

This section is empty.

Functions

func CollectCommentLinesAbove 

func CollectCommentLinesAbove(lines []string, startLine int, prefix string, skipBlankLines bool) string

CollectCommentLinesAbove collects the contiguous comment lines above startLine whose trimmed form begins with prefix, walking upward and stopping at the first line that does not.

The prefix and one optional following space are removed from each line. With skipBlankLines the walk crosses blank lines between the declaration and the comment block; without it, a blank line means the declaration has no comment. startLine is a zero-based index into lines.

func ConfigFromJSON 

func ConfigFromJSON(fullPath, displayPath string, excludeKeys []string) (string, error)

ConfigFromJSON reads a JSON config file and renders it as the Key/Type/Value table.

func ConfigFromTOML 

func ConfigFromTOML(fullPath, displayPath string, excludeKeys []string) (string, error)

ConfigFromTOML reads a TOML config file and renders its leaf keys as the Key/Type/Value table, with nested tables flattened into dotted keys.

func ConfigTableFromJSON 

func ConfigTableFromJSON(value any, displayPath string, excludeKeys []string) (string, error)

ConfigTableFromJSON renders decoded JSON as the Key/Type/Value table.

A document whose top level is not an object has no keys to tabulate, so it is rendered as a fenced, indented JSON block instead.

func ConfigTableFromJSONText 

func ConfigTableFromJSONText(text []byte, displayPath string, excludeKeys []string) (string, error)

ConfigTableFromJSONText parses JSON text and renders it as the Key/Type/Value table, or the error marker when it does not parse.

func DecodeJSON 

func DecodeJSON(data []byte) (any, error)

DecodeJSON decodes JSON text into the value model this package renders: nil, bool, int64, float64, string, []any and *JSONObject.

Object key order is kept, and an integer literal stays an integer rather than becoming a float, because the config tables print those as different types.

func Dedent 

func Dedent(text string) string

Dedent removes the longest common leading whitespace from every line of text, reproducing Python's textwrap.dedent.

Lines made only of whitespace are emptied first and then ignored when the common margin is measured, so a blank line indented less than the block does not defeat the dedent.

func DemoteDocHeadings 

func DemoteDocHeadings(text string, baseLevel int) string

DemoteDocHeadings renests the headings a doc comment wrote, under the one above them.

A doc comment is written as if it owned a document -- Go's own convention is "# Usage", and a KDoc, docstring or JSDoc block can carry any markdown -- but on a generated reference page it is a subsection of a symbol, and the page already has exactly one H1: its title. Emitted verbatim, a package doc with headings puts a second H1 on the page, which is a hard error, and the symbol the doc belongs to stops being its parent in the outline.

baseLevel is the level of the heading the text sits under: the symbol's own heading, or 1 -- the page title -- for a directive that emits the doc alone. Every heading shifts by the same amount, so the doc's internal structure is kept; the shallowest one becomes the direct child of baseLevel. Nothing shifts when the doc is already nested deeply enough, and nothing goes past H6, markdown's floor.

func DetectLanguage 

func DetectLanguage(dir string) (string, error)

DetectLanguage auto-detects a project's language from the marker files in dir, returning the empty string when none is detected.

func DetectionOrder 

func DetectionOrder() []string

DetectionOrder lists the languages auto-detection tries, in priority order.

func ExcludeKeysFromAttrs 

func ExcludeKeysFromAttrs(attrs map[string]string) []string

ExcludeKeysFromAttrs reads a directive's exclude attribute as a key set. An absent or empty attribute excludes nothing.

func ExtractBraceBlock 

func ExtractBraceBlock(source string, openBracePos int) (string, bool)

ExtractBraceBlock returns the content between the brace at openBracePos and its match, exclusive of both, and reports whether the braces matched.

String literals are skipped, so a brace inside one does not change the depth, and a backslash escapes the next byte. openBracePos is a byte offset into source, which is what a regexp match index and strings.Index both report.

func FormatDocstring 

func FormatDocstring(docstring string, baseLevel int) string

FormatDocstring transforms Google-style docstring sections into markdown.

Section headers like "Args:", "Returns:" and "Raises:" followed by indented "name: description" lines become bold headers with bullet lists, so the markdown converter renders them as structured HTML instead of collapsing the whitespace.

Source-wrapped prose (Go, JSDoc and KDoc doc comments wrap at around 75 columns) is first normalized with prose.JoinWrappedLines so a soft-wrapped sentence becomes one line; blank-line paragraph breaks, indented preformatted blocks, fenced code, list items and doctest lines are left verbatim.

baseLevel is the level of the heading this text is emitted under, or 1 for the page title when it is emitted alone. Headings the doc comment wrote are renested beneath it; see DemoteDocHeadings. It has no default, because every caller knows where it is putting the text and a wrong guess puts a second H1 on the page.

func FormatError 

func FormatError(message string) string

FormatError renders a message as the block-quoted marker selfdoc leaves in place of a directive it could not resolve.

func HandleTableConfig 

func HandleTableConfig(
	path string,
	_ *string,
	_ []string,
	_ []string,
	baseDir string,
	attrs map[string]string,
) (string, error)

HandleTableConfig is the shared table-config handler. It reads a JSON or TOML config file and renders it as a key/type/value table; any other extension is shown verbatim in a fenced block, since there is nothing to tabulate.

A language that recognizes a further config format maps table-config to its own handler and delegates here for everything it does not handle -- what the TypeScript extractor does for JSONC.

func IsDir 

func IsDir(path string) bool

IsDir reports whether path is an existing directory, the question Python's os.path.isdir answers.

func IsFile 

func IsFile(path string) bool

IsFile reports whether path is an existing regular file, the question Python's os.path.isfile answers -- which every extractor asks of a candidate it is resolving.

func IsKnownLanguage 

func IsKnownLanguage(name string) bool

IsKnownLanguage reports whether selfdoc ships an extractor for name.

func JSONTypeName 

func JSONTypeName(value any) string

JSONTypeName names a decoded value's type as the config tables print it.

func JSONValueRepr 

func JSONValueRepr(value any) string

JSONValueRepr renders a decoded value compactly enough for a table cell: a scalar verbatim, a long string truncated, and a collection as its size.

func KnownLanguages 

func KnownLanguages() []string

KnownLanguages lists every language selfdoc ships an extractor for.

func ParseCommaSet 

func ParseCommaSet(value string) []string

ParseCommaSet splits a comma-separated attribute value into its distinct, trimmed, non-empty parts.

The result is sorted, where the Python set it replaces iterated in an arbitrary order. Only one caller iterates it -- ApplyExcludeKeys, to name the first excluded key a document does not carry -- so sorting turns an arbitrary choice among several missing keys into a stable one.

func ReadSource 

func ReadSource(filepath string) (string, error)

ReadSource reads a source file as text.

func Register 

func Register(name string, factory Factory)

Register records the factory for a language. Each language package calls it from an init function, so linking the package in is what makes the language available.

It panics on a name that is not a known language and on a second registration of the same name: both are mistakes in the language package itself, visible the moment the binary starts.

func Registered 

func Registered() []string

Registered lists the languages whose packages are linked into this binary, sorted.

func RenderJSONIndent2 

func RenderJSONIndent2(value any) string

RenderJSONIndent2 renders a decoded value the way Python's json.dumps(value, indent=2) does, byte for byte: two-space indentation, ": " between a key and its value, "{}" and "[]" for the empty collections, and every non-ASCII character escaped.

Keys are NOT sorted -- the call site this serves prints a JSON document whose top level is not an object, and reordering what the file said would misreport it.

func RenderTable 

func RenderTable(headers []string, rows [][]string) (string, error)

RenderTable renders a Markdown table in the one form the extractors emit: no per-column alignment and no padding. It exists so the nine extractors share a single call into the table renderer.

func SourcePaths 

func SourcePaths(config map[string]any) ([]string, error)

SourcePaths lists just the declared source paths. It is empty for a project that publishes no code.

func SymbolHeading 

func SymbolHeading(level int, name string) string

SymbolHeading is a heading naming something the extractor read out of source.

A class, a function, a struct field, a table, a module path -- every one of them is a token the generator copied from code, and it is emitted as a code span so the page says so.

That is a claim about what the text is, and two readers act on it. A human sees a symbol set in the page's code face rather than a word in its prose face. The spell checker sees a code span and does not read it: it masks inline code, so an identifier it cannot recognize as English -- JSONResponse, returncode -- stops being a misspelling the moment the generator marks it as what it is. What remains flagged is docstring prose, which is the author's writing and the author's to fix; an identifier written into a sentence is backticked by whoever wrote the sentence.

The heading's anchor is unaffected: anchors are slugified from the rendered inline form, and rendering a code span leaves the same text, so #jsonresponse still addresses the same heading.

func SymbolHeadingPattern 

func SymbolHeadingPattern(name string) *regexp.Regexp

SymbolHeadingPattern matches a heading that names name, wherever the heading came from.

The coverage measurement asks one question of a page -- does any heading on it name this symbol -- and pages come from two writers. SymbolHeading writes the code-span form; an author writing a reference page by hand writes whichever form reads well to them. The optional qualifier admits a method named by its owner (Pipeline.Execute).

func SymbolSpan 

func SymbolSpan(name string) string

SymbolSpan is a token the extractor read out of source, inline in generated prose.

The heading form of the same claim is SymbolHeading; this is what a generated list item, a table cell or a rendered label uses. A name that came from code is set in the code face and skipped by the spell checker, whichever generated structure it appears in.

Types

type Base 

type Base struct {
	// contains filtered or unexported fields
}

Base carries the behavior every language extractor shares: the directive dispatch table and the no-op answers to the optional protocol methods.

A language package embeds it, builds it with NewBase, and overrides whichever of the optional methods its language can answer. Name comes from the language it was built with, so the unknown-directive message names the right extractor without the language package restating it.

func NewBase 

func NewBase(language string, handlers map[string]Handler) Base

NewBase builds the shared part of a language extractor from the language's registry name and its directive dispatch table.

The handlers usually close over the extractor being constructed, so the normal shape is to allocate the extractor first and assign its Base second.

func (Base) Extract 

func (b Base) Extract(
	directiveName string,
	attrs map[string]string,
	body []string,
	sourcePaths []string,
	baseDir string,
) (string, error)

Extract dispatches a directive to the handler registered for it, passing the path and target attributes out as their own arguments because every handler reads them.

A directive name with no handler renders an error marker naming both the directive and the language, since the same name can be valid for another language.

func (Base) FileExtensions 

func (b Base) FileExtensions() []string

FileExtensions reports that the language claims no file extensions.

func (Base) ModuleDocstring 

func (b Base) ModuleDocstring(string) (string, error)

ModuleDocstring reports no module documentation, for a language whose extractor cannot read it.

func (Base) Name 

func (b Base) Name() string

Name is the language's registry name.

func (Base) PublicSymbols 

func (b Base) PublicSymbols(string) ([]string, error)

PublicSymbols reports no symbols, for a language whose extractor cannot read them.

func (Base) ResolvePath 

func (b Base) ResolvePath(string, []string, string) string

ResolvePath resolves nothing, for a language whose extractor has no path convention.

func (Base) SymbolDetails 

func (b Base) SymbolDetails(string, string) (*SymbolDetails, error)

SymbolDetails reports nothing, for a language whose extractor cannot read signatures.

type DetectedLanguage 

type DetectedLanguage struct {
	// Path is the directory the language was detected in.
	Path string
	// Language is the language's registry name.
	Language string
}

DetectedLanguage is one language auto-detection found in a directory.

func DetectLanguages 

func DetectLanguages(dir string) ([]DetectedLanguage, error)

DetectLanguages reports every language detected in dir, not just the first. A polyglot repository answers with several, in detection-priority order.

type DocParam 

type DocParam struct {
	// Name is the parameter name as the docstring writes it, carrying any
	// variadic or keyword prefix.
	Name string
	// Type is the parenthesized type the docstring gives, nil when it gives
	// none.
	Type *string
	// Description is the parameter's documentation, with continuation lines
	// joined on.
	Description string
}

DocParam is one documented parameter.

type DocRaise 

type DocRaise struct {
	// Type is the exception type the docstring names.
	Type string
	// Description is when it is raised.
	Description string
}

DocRaise is one documented exception.

type DocSections 

type DocSections struct {
	// Description is everything before the first section header.
	Description string
	// Params are the entries of the Args, Arguments, Keyword Args and
	// Keyword Arguments sections, in order.
	Params []DocParam
	// Returns is the text of the Returns or Yields section, nil when the
	// docstring has neither.
	Returns *string
	// Raises are the entries of the Raises section, in order.
	Raises []DocRaise
}

DocSections is a Google-style docstring parsed into its parts.

func ParseDocstringSections 

func ParseDocstringSections(text string) DocSections

ParseDocstringSections parses a Google-style docstring into its structured sections. It is how the quality measurement learns which parameters and return values a symbol's own documentation covers.

type Extractor 

type Extractor interface {
	// Name is the language's registry name, as selfdoc.json spells it.
	Name() string

	// Detect reports whether dir looks like a project in this language,
	// by the presence of the language's own marker files.
	Detect(dir string) bool

	// ResolvePath resolves a directive's path argument to a filesystem path,
	// trying each of sourcePaths as a prefix and then baseDir directly.
	// It returns the empty string when nothing resolves.
	ResolvePath(pathArg string, sourcePaths []string, baseDir string) string

	// Extract resolves one directive into the Markdown that replaces it.
	//
	// A directive that cannot be resolved -- a missing file, a syntax error, an
	// unknown directive name -- returns an error marker as its Markdown and a
	// nil error, so one bad directive degrades one region of one page instead of
	// failing the build. The error return is for a broken toolchain.
	Extract(directiveName string, attrs map[string]string, body []string, sourcePaths []string, baseDir string) (string, error)

	// FileExtensions lists the file extensions this language owns, each with
	// its leading dot.
	FileExtensions() []string

	// PublicSymbols lists the symbols file exports, in source order.
	PublicSymbols(file string) ([]string, error)

	// SymbolDetails reports what file says about symbol's parameters and return
	// value. It returns nil when the symbol is not in the file. A dotted name
	// selects a member of a type ("Pipeline.Execute").
	SymbolDetails(file, symbol string) (*SymbolDetails, error)

	// ModuleDocstring is the module- or package-level documentation at path,
	// with soft-wrapped prose joined so a wrapped first sentence reads as one.
	ModuleDocstring(path string) (string, error)
}

Extractor is the protocol every language extractor implements.

The methods split into three groups. Name and Detect identify the language. Extract resolves one directive into Markdown. The rest answer the discovery questions the coverage, quality and staleness measurements ask.

Three methods return an error where the Python protocol they replace returned an empty result: a parser that cannot run at all is a broken installation, not a file with no symbols in it. A file that cannot be read, or whose contents do not parse, still answers empty -- that is a property of the file and every extractor reports it that way.

func Lookup 

func Lookup(name string) (Extractor, bool, error)

Lookup builds the extractor registered for a language.

It reports an error for a known language whose package is not linked in, rather than answering with a stub: the project declared a language selfdoc supports, and quietly rendering "no extractor for 'python'" across its pages would blame the project for a wiring mistake in selfdoc. A language selfdoc does not support is not an error -- it returns false, and the caller substitutes NewStub.

func NewStub 

func NewStub(language string) Extractor

NewStub builds the extractor for a language selfdoc has none for.

type Factory 

type Factory func() Extractor

Factory builds an extractor.

It takes nothing: every extractor answers by reading and parsing files, and none of them needs an effects handle. The Python one used to run an interpreter through one and now parses in-process.

type Handler 

type Handler func(
	path string,
	target *string,
	body []string,
	sourcePaths []string,
	baseDir string,
	attrs map[string]string,
) (string, error)

Handler resolves one directive for one language. It is the value type of an extractor's dispatch table.

target is nil when the directive declared no target attribute, which is a different question from an empty target: a code-test directive with no target renders the whole file, while an empty one looks for a symbol with no name.

The error return is for a broken toolchain, never for an unresolvable directive -- see Extractor.Extract.

type JSONObject 

type JSONObject struct {
	// contains filtered or unexported fields
}

JSONObject is a decoded JSON or TOML table that remembers the order its keys appeared in.

Order is not a nicety here: every config table selfdoc renders lists its rows in document order, which is what a reader comparing the page against the file expects. Go's map type has no order, so the decoders in this file build this instead.

func ApplyExcludeKeys 

func ApplyExcludeKeys(data *JSONObject, excludeKeys []string, displayPath string) (*JSONObject, string)

ApplyExcludeKeys drops excludeKeys from data.

It returns the filtered object, or -- when data does not carry one of the keys -- a nil object and the error marker naming it. Excluding a key that is not there is a mistake in the directive rather than a no-op: the author believes they are hiding something, and silently rendering it would publish the very value they meant to withhold.

func NewJSONObject 

func NewJSONObject() *JSONObject

NewJSONObject builds an empty object.

func (*JSONObject) Get 

func (o *JSONObject) Get(key string) (any, bool)

Get reports the value at key. A nil object reports nothing.

func (*JSONObject) Has 

func (o *JSONObject) Has(key string) bool

Has reports whether the object carries key. A nil object carries nothing.

func (*JSONObject) Keys 

func (o *JSONObject) Keys() []string

Keys lists the object's keys in the order they appeared.

A nil object has no keys, which is what a decoder that answered "no object here" means. Has, Get, Keys and Len all read a nil receiver as the empty object, so a caller that took an object out of a document does not have to know whether the document carried one.

func (*JSONObject) Len 

func (o *JSONObject) Len() int

Len is the number of keys. A nil object has none.

func (*JSONObject) Set 

func (o *JSONObject) Set(key string, value any)

Set records key's value, appending the key when it is new and leaving its position alone when it is not -- the behavior of assigning into a Python dict, which is what the decoders this replaces did.

type SourceEntry 

type SourceEntry struct {
	// Path is the source path as selfdoc.json declares it, relative to the
	// project's base directory.
	Path string
	// Language is the language's registry name.
	Language string
	// Extractor reads this path's sources.
	Extractor Extractor
}

SourceEntry is a declared source path with its language and the extractor that reads it.

func ResolveSourceEntries 

func ResolveSourceEntries(config map[string]any) ([]SourceEntry, error)

ResolveSourceEntries resolves a config's source declarations into entries carrying their extractors.

A project that publishes no code declares no source entries, so an absent or empty source key yields none.

type SymbolDetails 

type SymbolDetails struct {
	// Params are the symbol's parameters in declaration order, with a
	// receiver, self and cls already dropped.
	Params []SymbolParam
	// ReturnType is the declared return type, nil when the symbol declares
	// none.
	ReturnType *string
	// ReturnDocumented reports whether the documentation says what the symbol
	// returns.
	ReturnDocumented bool
}

SymbolDetails is what an extractor reads out of source about one symbol's parameters and return value, together with whether the symbol's own documentation covers them. The quality measurement scores a symbol from it.

type SymbolParam 

type SymbolParam struct {
	// Name is the parameter name, carrying its variadic or keyword prefix
	// ("*args", "**kwargs") where the language writes one.
	Name string
	// Type is the declared type, nil when the parameter declares none.
	Type *string
	// Documented reports whether the symbol's documentation names this
	// parameter.
	Documented bool
}

SymbolParam is one parameter of a symbol, as SymbolDetails reports it.

Source Files

View all Source files

Directories

Path Synopsis
Package dart resolves selfdoc's directives against Dart source.
 Package dart resolves selfdoc's directives against Dart source.
Package golang resolves selfdoc's directives against Go source.
 Package golang resolves selfdoc's directives against Go source.
Package kotlin resolves selfdoc's directives against Kotlin source.
 Package kotlin resolves selfdoc's directives against Kotlin source.
Package python resolves selfdoc's directives against Python source.
 Package python resolves selfdoc's directives against Python source.
Package sql resolves selfdoc's directives against PostgreSQL DDL.
 Package sql resolves selfdoc's directives against PostgreSQL DDL.
Package svelte reads Svelte component source for selfdoc.
 Package svelte reads Svelte component source for selfdoc.
Package swift resolves selfdoc's directives against Swift source.
 Package swift resolves selfdoc's directives against Swift source.
Package typescript reads TypeScript and JavaScript source for selfdoc.
 Package typescript reads TypeScript and JavaScript source for selfdoc.
Package zig reads Zig source for selfdoc.
 Package zig reads Zig source for selfdoc.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.