boilertext

Documentation

Index

• type Extractor
• type TextBlock
  • func GenerateTextBlocks(reader io.Reader, splitStrategy bufio.SplitFunc) ([]*TextBlock, error)
  • func (t *TextBlock) LinkDensity() float64

Types

type Extractor

type Extractor interface {
	Process(blocks []*TextBlock) (string, error)
}

Extractor is an interface that processes incoming HTML and outputs text within HTML minus all the boilerplate

type TextBlock

type TextBlock struct {
	NumOfWords       int
	NumOfAnchorWords int
	Content          string
}

TextBlock represents a text block which may comprise of inline elements.

func GenerateTextBlocks

func GenerateTextBlocks(reader io.Reader, splitStrategy bufio.SplitFunc) ([]*TextBlock, error)

GenerateTextBlocks takes a reader containing HTML and generates a TextBlock array from it.

func (*TextBlock) LinkDensity

func (t *TextBlock) LinkDensity() float64

LinkDensity is the number of link text words divided by the total number of words in the block.