pdftext

Documentation

Overview

Package pdftext reads the text layer of a PDF page as markdown (headings, paragraphs, lists, code, formulas and tables), without any external service: most PDFs - reports, exports, invoices - are text-native. See PageMarkdown, and PageMarkdownWith for a host that has a layout model to find the tables with no ruling lines.

Index

• Constants
• func PageMarkdown(page pdf.Page) (md string, err error)
• func PageMarkdownWith(page pdf.Page, opts PageOptions) (md string, err error)
• func TableMarkdown(s TableStructure, words []TableWord, pageHeight float64) (string, bool)
• type PageOptions
• type Result
  • func Extract(data []byte) (*Result, error)
• type TableSource
• type TableStructure
• type TableWord

Constants

const MinCharsPerPage = 20

MinCharsPerPage is the threshold below which extracted text is treated as "not really there" - a scanned/image-only PDF will still yield a handful of stray characters from stamps or embedded metadata, but nowhere near this per page on average. Below it, callers should prefer OCR (the document reader) or the vision fallback instead of trusting this near-empty text.

Functions

func PageMarkdown

func PageMarkdown(page pdf.Page) (md string, err error)

PageMarkdown returns a page as markdown: headings, code blocks, lists, paragraphs and formulas.

func PageMarkdownWith

func PageMarkdownWith(page pdf.Page, opts PageOptions) (md string, err error)

PageMarkdownWith is PageMarkdown for a caller that has a layout model to ask about tables that have no ruling lines. The source is only asked on a page whose text looks columnar and that is not rotated, and what it returns only adds tables: what the rulings found stays, and a page is never worse for a model that failed.

func TableMarkdown

func TableMarkdown(s TableStructure, words []TableWord, pageHeight float64) (string, bool)

TableMarkdown writes one table as markdown from the text read from its picture, for a page that is a picture as a whole (a scan): it has no text of its own to take the cells from. Boxes are in points from the top left corner of a page pageHeight points high. ok is false when what was read does not make a table.

Types

type PageOptions

type PageOptions struct {
	// Tables gives the tables the models see on the page; nil means there are no models.
	Tables TableSource
	// LargeImage says the page holds a picture big enough to be a table. A page of text with columns is put to the
	// models anyway; a page with such a picture is too, because the table in it has no text to look at.
	LargeImage bool
}

PageOptions says what a page can ask of a layout model.

type Result

type Result struct {
	Text      string
	PageCount int
	// Sparse is true when the extracted text is too little relative to the
	// page count to be a real text layer - most likely a scanned PDF.
	Sparse bool
}

Result holds the outcome of a native PDF text extraction attempt.

func Extract

func Extract(data []byte) (*Result, error)

Extract reads a whole PDF's text layer as markdown, page after page. It does not attempt OCR: a scanned PDF with no text layer returns a Sparse result with little or no text, not an error - callers decide what to do next. A host that wants the pages one at a time, with the tables its layout model finds, uses PageMarkdownWith.

type TableSource

type TableSource func() ([]TableStructure, error)

TableSource gives the tables the models see on a page. It is only asked for a page whose text looks columnar, and an error or an empty answer means "no tables": a page is never lost because the models were unavailable.

type TableStructure

type TableStructure struct {
	X0, Top, X1, Bottom float64
	Score               float64
	Columns             [][4]float64
	Rows                [][4]float64
	Headers             [][4]float64

	// Words reads the text of the table from the picture, for a table that is one (a table pasted as an image has no
	// text on the page to fill its cells with). It is called only when the page has too little text of its own
	// under the table, because reading text from a picture is slow and less exact than the page's own. It may be nil.
	Words func() ([]TableWord, error)
}

TableStructure is a table as the models saw it, in points measured from the top left corner of the page. Boxes are {x0, top, x1, bottom}; the models' boxes overlap a little and come in any order.

type TableWord

type TableWord struct {
	X0, Top, X1, Bottom float64
	Text                string
}

TableWord is a piece of text read from a picture, with its box in points from the top left corner of the page.