Documentation
¶
Overview ¶
Package document converts provider-neutral source evidence into deterministic normalized units, headings, spans, and chunks.
The package does not perform filesystem, network, storage, database, queue, daemon, vault, or application work.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ValidateNormalizedDocument ¶ added in v0.14.0
func ValidateNormalizedDocument(normalized NormalizedDocument) error
ValidateNormalizedDocument verifies that a normalized document is a structurally complete, internally consistent version-3 normalization result. It detects stale identities after callers deserialize or copy the public evidence structs.
Types ¶
type Chunk ¶
type Chunk struct {
Key string
Ordinal int
Text string
HeadingPath []string
CharCount int
Checksum string
Truncated bool
Spans []ChunkSpan
}
Chunk is a stable publication and provenance unit.
type HeadingMark ¶
HeadingMark records the active heading path at a character offset.
type NormalizePolicy ¶
type NormalizePolicy struct {
// contains filtered or unexported fields
}
NormalizePolicy fixes the version-3 normalization behavior. Its structural values are private because changing any of them requires a new version.
func NewNormalizePolicy ¶
func NewNormalizePolicy(maxDocumentChars int) (NormalizePolicy, error)
NewNormalizePolicy returns the fixed version-3 policy for a document limit.
func (NormalizePolicy) Identity ¶
func (p NormalizePolicy) Identity() NormalizePolicyIdentity
Identity returns a copy of every value that defines normalization output.
type NormalizePolicyIdentity ¶
type NormalizePolicyIdentity struct {
Version int `json:"version"`
MaxDocumentChars int `json:"max_document_chars"`
MaxUnitChars int `json:"max_unit_chars"`
MaxSourceUnitBytes int `json:"max_source_unit_bytes"`
MaxMetadataSourceBytes int `json:"max_metadata_source_bytes"`
MaxLinkChars int `json:"max_link_chars"`
MaxChunkRunes int `json:"max_chunk_runes"`
ChunkOverlap int `json:"chunk_overlap"`
MaxChunks int `json:"max_chunks"`
}
NormalizePolicyIdentity is a read-only copy of every effective policy value.
type NormalizedDocument ¶
type NormalizedDocument struct {
PolicyVersion int
Family string
UnitKind string
Units []NormalizedUnit
Chunks []Chunk
Checksum string
Truncated bool
}
NormalizedDocument is deterministic, inert document evidence.
func NormalizeDocument ¶
func NormalizeDocument(source SourceDocument, policy NormalizePolicy) (NormalizedDocument, error)
NormalizeDocument converts transient provider Markdown into deterministic, inert canonical text plus exact unit spans. It never retains raw responses.
type NormalizedUnit ¶
type NormalizedUnit struct {
Index int
SourceKey string
Kind string
Text string
Header string
Dimensions UnitDimensions
CharCount int
Checksum string
Truncated bool
HeadingMarks []HeadingMark
}
NormalizedUnit is one canonical source unit and its provenance.
type SourceDocument ¶
type SourceDocument struct {
Family string
UnitKind string
Units []SourceUnit
}
SourceDocument contains provider-neutral evidence in source order.
type SourceUnit ¶
type SourceUnit struct {
Index int
Markdown string
Header string
Dimensions UnitDimensions
}
SourceUnit contains the transient evidence for one source unit.
type UnitDimensions ¶
UnitDimensions records source dimensions when the provider reports them.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package embedding prepares deterministic, provider-neutral document inputs for text embedding and optional pre-embedding distillation.
|
Package embedding prepares deterministic, provider-neutral document inputs for text embedding and optional pre-embedding distillation. |
|
eval
Package eval evaluates document retrieval recipes against versioned public or synthetic corpora with graded relevance judgments.
|
Package eval evaluates document retrieval recipes against versioned public or synthetic corpora with graded relevance judgments. |
|
internal
|
|
|
compattest
Package compattest loads the frozen cross-repository compatibility evidence.
|
Package compattest loads the frozen cross-repository compatibility evidence. |
|
manifestjson
Package manifestjson provides strict JSON checks shared by provider capability manifests and provider responses.
|
Package manifestjson provides strict JSON checks shared by provider capability manifests and provider responses. |
|
Package media detects still images, animated images, and video from bytes and evaluates them against a bounded eligibility policy.
|
Package media detects still images, animated images, and video from bytes and evaluates them against a bounded eligibility policy. |
|
mediatest
Package mediatest builds small deterministic synthetic media containers for tests.
|
Package mediatest builds small deterministic synthetic media containers for tests. |
|
Package mistral provides bounded, stateless document extraction through the Mistral OCR API.
|
Package mistral provides bounded, stateless document extraction through the Mistral OCR API. |
|
internal/probecontract
Package probecontract owns the private serialized request identity shared by production probing and consumer test support.
|
Package probecontract owns the private serialized request identity shared by production probing and consumer test support. |
|
internal/testfixture
Package testfixture contains deterministic synthetic document bytes shared by Docbank's tests and its consumer test-support package.
|
Package testfixture contains deterministic synthetic document bytes shared by Docbank's tests and its consumer test-support package. |
|
mistraltest
Package mistraltest provides deterministic synthetic documents and capability evidence for applications that test Mistral integrations.
|
Package mistraltest provides deterministic synthetic documents and capability evidence for applications that test Mistral integrations. |
|
Package voyage provides bounded, stateless multimodal embedding of images and video through the Voyage AI API.
|
Package voyage provides bounded, stateless multimodal embedding of images and video through the Voyage AI API. |
|
internal/probecontract
Package probecontract pins the request identity shared by the Voyage capability probe and synthetic test manifests.
|
Package probecontract pins the request identity shared by the Voyage capability probe and synthetic test manifests. |
|
voyagetest
Package voyagetest provides synthetic capability evidence for applications that test Voyage integrations.
|
Package voyagetest provides synthetic capability evidence for applications that test Voyage integrations. |