Documentation
¶
Overview ¶
Package extract turns raw document bytes into extracted text.
Index ¶
Constants ¶
const DefaultChildMemoryLimit = 512 << 20
DefaultChildMemoryLimit caps an isolation child. Matches the 512 MB used by the reference implementation in design.md §6.
const DefaultChildTimeout = 60 * time.Second
DefaultChildTimeout bounds one isolated extraction.
const DefaultMaxDecompressedBytes = 64 << 20 // 64 MiB
DefaultMaxDecompressedBytes bounds what one archive-backed document may expand to. Generous for real documents, ruinous for a zip bomb.
const DefaultTimeout = 120 * time.Second
DefaultTimeout bounds one extraction call. Generous because OCR on a scanned multi-page PDF is genuinely slow.
Variables ¶
ErrUnavailable marks a transport/deployment failure — connection refused, DNS, timeout, or a 5xx from the xberg service itself. It is the only case worth retrying or falling back on; an extraction xberg *rejected* (4xx) is a verdict on the document and is returned as a plain error instead.
var ErrUnsupportedFormat = errors.New("extract: unsupported format")
ErrUnsupportedFormat marks a document LocalExtractor has no parser for. It is a verdict on the document, not an availability failure, so a Chain stops on it rather than falling through — there is nothing further down the chain that would do better, since local parsing is already the last layer.
Functions ¶
func IsInIsolatedChild ¶
func IsInIsolatedChild() bool
IsInIsolatedChild reports whether this process was spawned as an extraction child. Useful for a host application that wants to skip expensive startup work (opening database pools, joining a cluster) in a process that is only going to parse one file and exit.
func RunIsolatedChildIfInvoked ¶
func RunIsolatedChildIfInvoked()
RunIsolatedChildIfInvoked must be the first statement in a host application's main() if it wants IsolatedExtractor to work:
func main() {
extract.RunIsolatedChildIfInvoked()
// ... normal startup
}
In the normal case it returns immediately and does nothing. When the process was spawned as an extraction child it reads one document from stdin, parses it with a LocalExtractor, writes the result to stdout, and **exits without returning** — so nothing after the call runs in a child.
A library cannot arrange this for itself: ragit does not own main(), so it cannot intercept startup the way an application re-invoking its own hidden subcommand can. This one-line requirement is the cost of that.
func SupportedExtensions ¶
func SupportedExtensions() []string
SupportedExtensions lists what LocalExtractor can parse. Deliberately much narrower than xberg's ~101 formats — this is a fallback, not a replacement.
Types ¶
type Chain ¶
type Chain struct {
// contains filtered or unexported fields
}
Chain tries each extractor in order, falling back to the next one only when the current one was *unavailable*.
The narrowness of that rule is the entire point, and it is the most expensive lesson in docs/design.md §6. A fallback fires on a transport or deployment failure — connection refused, DNS, timeout, a 5xx, a child process that could not be spawned. It does NOT fire when an extractor rejected the document itself: a corrupt PDF, an unsupported type, a clean parser error. Those are verdicts, and a verdict is final.
Retrying a rejected document down the chain means feeding bytes that already broke one parser into progressively less-contained code paths — which is precisely how the OOM incident in §6 happened. A bad document must never buy its way back in.
func NewChain ¶
NewChain builds a Chain from extractors in order of preference. Nil extractors are skipped, so a caller can write
NewChain(xbergOrNil, isolated, local)
without branching on whether the sidecar is configured.
type Extractor ¶
type Extractor interface {
Extract(ctx context.Context, data []byte, filename string) (*Result, error)
}
Extractor turns document bytes into text. Implementations decide what a terminal, non-retryable failure looks like versus a transient one — see XbergExtractor and ErrUnavailable.
type IsolatedExtractor ¶
type IsolatedExtractor struct {
// MemoryLimit caps the child. Zero means DefaultChildMemoryLimit.
MemoryLimit int64
// Timeout bounds one extraction. Zero means DefaultChildTimeout.
Timeout time.Duration
// contains filtered or unexported fields
}
IsolatedExtractor runs local parsing in a short-lived child process with a memory ceiling and a timeout.
This is the containment layer from design.md §6. Its value is structural rather than parser-specific: a blow-up in *any* parser — including one not yet audited, or a future dependency upgrade — kills a child process that owns nothing, and the parent marks that one document failed and carries on. Without it, the kernel's OOM killer picks a victim host-wide, and the application is merely the most likely one, not the only eligible one.
Wiring requirement ¶
A library cannot re-invoke itself the way an application can: ragit does not own main(). The host application must call RunIsolatedChildIfInvoked as the first statement in main(), which is what makes its binary able to serve as its own extraction child. Without that call, IsolatedExtractor's children start the host's normal startup path instead of parsing, so Extract fails with ErrUnavailable and a Chain falls through to the next layer — degraded, but not broken.
func NewIsolatedExtractor ¶
func NewIsolatedExtractor() *IsolatedExtractor
NewIsolatedExtractor builds an IsolatedExtractor with default limits.
func (*IsolatedExtractor) Extract ¶
func (e *IsolatedExtractor) Extract(ctx context.Context, data []byte, filename string) (*Result, error)
Extract parses data in a capped child process.
Failures are classified per design.md §6's fallback rule:
- failing to spawn, or a child that died without a verdict (OOM kill, timeout, missing RunIsolatedChildIfInvoked wiring) → ErrUnavailable, so a Chain moves on;
- a child that parsed and rejected the document → a plain error, so a Chain stops. A bad document does not buy its way into a less-contained parser.
type LocalExtractor ¶
type LocalExtractor struct {
// MaxDecompressedBytes caps the total bytes read out of a container
// format (docx's zip). Zero means DefaultMaxDecompressedBytes. This is
// the zip-bomb guard: a 200 kB .docx can legitimately declare gigabytes
// of entry content.
MaxDecompressedBytes int64
}
LocalExtractor parses documents in-process, with no sidecar.
It exists so a deployment that has not stood up an xberg sidecar still works, at a smaller supported format set and with no OCR (design.md §4). That convenience comes with real risk: these parsers run untrusted bytes through code that allocates based on what the file *claims*. A 212 kB PDF driving a parser to ~5 GB is the documented incident behind design.md §6.
So: do not hand a LocalExtractor untrusted uploads directly. Wrap it in an IsolatedExtractor, which runs exactly this code in a memory-capped child process. LocalExtractor is the thing being contained, not the containment.
func NewLocalExtractor ¶
func NewLocalExtractor() *LocalExtractor
NewLocalExtractor builds a LocalExtractor with default limits.
type Result ¶
type Result struct {
// Text is the extracted content, in Markdown.
Text string
PageCount int
// Metadata is the extractor's own structured output (detected tables,
// warnings, source language, ...), stored verbatim rather than parsed
// field-by-field so new fields don't require a schema migration.
Metadata json.RawMessage
}
Result is what an Extractor produces from one document.
type XbergExtractor ¶
type XbergExtractor struct {
// BaseURL is the xberg server root, e.g. http://xberg:8000.
BaseURL string
// HTTPClient is used for every call. Nil builds a client from Timeout.
HTTPClient *http.Client
// Timeout bounds a single extraction. Zero means DefaultTimeout. Ignored
// when HTTPClient is set.
Timeout time.Duration
}
XbergExtractor extracts documents through an xberg REST server (`xberg serve`). See https://docs.xberg.io.
func NewXbergExtractor ¶
func NewXbergExtractor(baseURL string, timeout time.Duration) *XbergExtractor
NewXbergExtractor builds an extractor with production defaults.
Source Files