Documentation
¶
Overview ¶
Package ftest is a probe harness for measuring how well the local LLM extracts storable facts from everyday conversation. It is deliberately separate from the live assistant: the point is to see, on a curated set of conversations, what the model pulls out (entities, temporal facts, placeholders for unknown people) and whether it flags the high-value gaps — names and familial relationships — for an immediate follow-up.
The data model mirrors the fact-database design in docs/fact-database-research.md: entities carry a resolution status (resolved vs. NIL placeholder), statements are reified facts with bi-temporal validity and provenance, and follow-ups are the knowledge gaps the model wants filled, each with a priority (immediate | banked).
Extraction emits structured JSON (this package's types), NOT AGE Cypher, so the model's job is pure semantics; a deterministic compiler (compile.go) turns the JSON into openCypher for Apache AGE. That split keeps "did it understand the facts" measurable independently of "did it emit valid Cypher".
Index ¶
Constants ¶
const ( PriorityImmediate = "immediate" PriorityBanked = "banked" )
Priority values.
const ( ResolutionResolved = "resolved" ResolutionPlaceholder = "nil_placeholder" )
Resolution values.
Variables ¶
This section is empty.
Functions ¶
func Compile ¶
Compile turns an extraction Result into deterministic openCypher for Apache AGE.
The output is the inner Cypher (the body you'd wrap in `SELECT * FROM cypher('graph', $$ ... $$)`), one statement per line, so it is easy to eyeball and to golden-test. Design choices, all from docs/fact-database-research.md §7:
- MERGE (never CREATE) on a stable key `k`, so replaying a conversation is idempotent and never duplicates a person node.
- Facts are reified as Statement nodes carrying provenance/confidence/temporal properties, wired subject-[:SUBJECT_OF]->stmt-[:ABOUT]->object.
- Deterministic ordering (entities by handle, statements by key) and escaped string literals, so the output is reproducible and diff-friendly.
Follow-ups are emitted as trailing comments, not graph writes: they are the caller's question queue, not facts about the world.
func SaveFixture ¶
SaveFixture writes a fixture as pretty JSON (stable, diff-friendly on disk).
Types ¶
type Entity ¶
type Entity struct {
Handle string `json:"handle"` // "e1"
Type string `json:"type"` // Person | Group | Event | Activity
Mention string `json:"mention"` // "my husband"
Resolution string `json:"resolution"` // "resolved" | "nil_placeholder"
Name string `json:"name,omitempty"` // known canonical name, if any
Attrs map[string]string `json:"attrs,omitempty"` // e.g. {"rel":"spouse_of:user"}
}
Entity is a person/group/event/activity mentioned in the conversation. Handle is a conversation-local id ("e1", "e2") the model assigns and reuses across turns; the compiler maps handles to stable MERGE keys so the model never invents graph ids. Resolution records whether the mention was pinned to a known entity or left as a placeholder to be filled by a later answer.
func KnownEntities ¶
KnownEntities flattens the entities recorded across prior turns, so the next turn's prompt can tell the model which handles already exist to reuse.
type Extractor ¶
type Extractor interface {
Extract(ctx context.Context, priorTurns []Turn, known []Entity, utterance string) (*Result, error)
}
Extractor produces a structured Result for one user turn, given the prior turns and the entity handles already known, so the caller can extract incrementally.
type Fixture ¶
type Fixture struct {
Name string `json:"name"`
Notes string `json:"notes,omitempty"`
Model string `json:"model,omitempty"` // model that produced the Recorded fields
Turns []Turn `json:"turns"`
Gold *Result `json:"gold,omitempty"` // hand-authored expected consolidated facts
}
Fixture is one recorded conversation used as an extraction test case. `record` mode produces it live; `run` mode replays Turns through the extractor and, when Gold is present, grades the consolidated result against it.
func LoadFixture ¶
LoadFixture reads a fixture JSON file.
type Followup ¶
type Followup struct {
Gap string `json:"gap"` // "e1.name"
Priority string `json:"priority"` // "immediate" | "banked"
Reason string `json:"reason,omitempty"`
Question string `json:"question"`
}
Followup is a knowledge gap the model wants filled. Priority is the crux of the probe: "immediate" means the caller should ask this turn (high-value — names, familial relationships), "banked" means queue it for later so she isn't an interrogation.
type LLMExtractor ¶
type LLMExtractor struct {
URL string // .../v1/chat/completions
Model string
MaxTokens int
Client *http.Client
Ontology Ontology
}
LLMExtractor calls an OpenAI-compatible /v1/chat/completions endpoint. It is bound to an Ontology, which supplies the system prompt and the schema that constrain generation. Point URL/Model at whatever OpenAI-format server you run; a smaller "fast" model and a larger reasoning model can be compared by swapping them.
func NewLLMExtractor ¶
func NewLLMExtractor(ont Ontology) *LLMExtractor
NewLLMExtractor builds an extractor for the given ontology against an OpenAI-format endpoint. Honors AATOOLKIT_FTEST_URL / AATOOLKIT_FTEST_MODEL overrides (e.g. to point at a different model on the same server); the defaults assume a local mlx-serve on port 1234 that speaks OpenAI format directly.
type Ontology ¶
type Ontology struct {
Predicates []string `json:"predicates"`
EntityTypes []string `json:"entity_types"`
SystemPrompt string `json:"system_prompt"`
}
Ontology is the domain-specific extraction contract a caller supplies to the harness: the controlled predicate vocabulary, the entity-type enum, and the system prompt that instructs the model. Constraining the model to these values (via the response_format JSON schema AND the prompt) is the fix for vocabulary drift — a model understands the facts but invents predicate names ("schedule", "attends") instead of the graph's canonical ones unless it is pinned. Keep the predicate list and the gold fixtures in sync: a fact can only be graded if both sides name the predicate the same way.
func LoadOntology ¶
LoadOntology reads an Ontology from a JSON file (predicates, entity_types, system_prompt) and validates that none of the three is empty.
func (Ontology) BuildMessages ¶
func (o Ontology) BuildMessages(priorTurns []Turn, known []Entity, utterance string) ([]byte, error)
BuildMessages assembles the OpenAI-style messages array for one extraction call: the ontology's system prompt (with its predicate vocabulary appended), then a user message carrying prior turns as context, the known entity handles to reuse, and the current utterance to extract. The system prompt must instruct the model to emit ONLY the JSON this package parses (entities/statements/followups).
func (Ontology) ResultSchema ¶
ResultSchema returns the JSON Schema for a Result, with the key fields locked to enums: predicate to the ontology's controlled vocabulary, entity type, resolution, and follow-up priority. Emitted as response_format.json_schema so the model's output is constrained at generation time rather than fuzzy-matched afterward.
type Report ¶
type Report struct {
StmtGold int // statements in gold
StmtFound int // gold statements the model produced (recall numerator)
StmtExtra int // model statements with no gold match (over-extraction)
ImmediateGold int // gold follow-ups marked immediate
ImmediateHit int // gold immediate follow-ups the model also marked immediate
Missed []string
Extra []string
}
Report scores a consolidated extraction against a gold Result. It is deliberately forgiving on surface form and strict on the things the probe cares about: did the model find the facts (statements), and did it flag the right high-value gaps for an immediate follow-up (§6.1)?
type Result ¶
type Result struct {
Entities []Entity `json:"entities"`
Statements []Statement `json:"statements"`
Followups []Followup `json:"followups"`
}
Result is the structured extraction for one turn — or, after Consolidate, for a whole conversation. It is exactly what the model is asked to emit as JSON.
func Consolidate ¶
Consolidate merges a sequence of per-turn results into one conversation-level Result — the view grading compares against gold, and the view the compiler turns into a graph. Entities union by handle (later mentions win on name/resolution and merge attrs, modeling a placeholder being filled by a later turn). Statements dedup by key. Follow-ups dedup by gap, keeping the strongest priority and dropping any gap that a later turn resolved into a named entity.
func ParseResult ¶
ParseResult extracts the JSON object from a model reply and unmarshals it into a Result. It tolerates code fences and surrounding prose by slicing from the first '{' to the last '}' — small models often wrap JSON in chatter despite instructions.
type Statement ¶
type Statement struct {
Subject string `json:"subject"` // entity handle
Predicate string `json:"predicate"` // "birthday", "has_activity", "prefers_pronoun"...
Value string `json:"value,omitempty"` // literal object
Object string `json:"object,omitempty"` // entity-handle object (relational fact)
ValidFrom string `json:"valid_from,omitempty"` // ISO-8601 or ""; "" = unknown/open
ValidUntil string `json:"valid_until,omitempty"` // when the fact expires (e.g. Christmas)
Recurrence string `json:"recurrence,omitempty"` // iCalendar RRULE, e.g. "FREQ=WEEKLY;BYDAY=MO"
Confidence float64 `json:"confidence,omitempty"`
SourceSpan string `json:"source_span,omitempty"`
}
Statement is a reified fact: subject --predicate--> (value | object), with optional bi-temporal validity, recurrence, confidence, and the source text span it came from. Object names an entity handle when the object is itself an entity (n-ary/relational facts); Value holds a literal when it is not.
type Turn ¶
type Turn struct {
Speaker string `json:"speaker"` // "user" | "assistant"
Text string `json:"text"`
Recorded *Result `json:"recorded,omitempty"`
}
Turn is one utterance. For user turns, Recorded holds the extraction the model produced at record time, given all prior turns as context (incremental). Assistant turns carry no extraction — they exist so the model sees realistic dialogue context.
Source Files