Documentation
¶
Overview ¶
Package classify judges every document of a topic once on a fixed set of facets (spec §8) and writes them as kb-owned frontmatter: genre, depth, relevance, quality, concepts, plus the generated literals summary, entities, questions and, for articles, criterion and aliases. It also owns the shared gate judgment (relevance + quality in one request) that ingest gates, the contract impact preview and classify all ask, the backfill review queues (recapture, remove), concept proposals, and the vocabulary bootstrap for topics without articles (spec §5.2).
Index ¶
- Constants
- Variables
- func AcceptVocabulary(s *session.Session) ([]string, error)
- func ArticleBanks() []*questions.Bank
- func CodeFlags(doc *corpus.Document, hostLines map[string]map[string]int) []quality.Flag
- func CreateStub(s *session.Session, title, criterion string, now time.Time) (string, error)
- func CreateStubArticle(topicRoot, domain, title, criterion string, now time.Time) (string, error)
- func DraftVocabulary(ctx context.Context, s *session.Session) ([]contract.VocabularyItem, error)
- func FacetBank() *questions.Bank
- func GateBank() *questions.Bank
- func IsTranscriptKind(sourceKind string) bool
- func SourceBanks() []*questions.Bank
- type GateJudgment
- type GateOptions
- type Options
- type Report
Constants ¶
const ( // MinNoneForProposals is the number of sources with primary concept // `none` that triggers concept proposals. MinNoneForProposals = 5 // MaxProposals caps the concepts proposed in one run. MaxProposals = 5 )
Proposal limits (spec §5.2 "New concepts").
const ( // FactPrimaryConcept is the fact name. FactPrimaryConcept = "primary_concept" // PrimaryNone records a decided primary concept `none`. PrimaryNone = "none" // PrimaryMatched records a decided primary concept other than `none`. PrimaryMatched = "concept" )
State-row fact recording the primary concept outcome of a source, so the concept proposal trigger counts across the topic, not only this run.
const ( // MaxConceptOptions is the number of articles offered to primary_concept; // with the `none` exit the choice stays within Jev's 255 options. MaxConceptOptions = 254 // MaxConceptCandidates caps the mentions_concept_{id} nouls per document. MaxConceptCandidates = 20 )
Concept-question limits (spec §8).
const ( RoleCore = "core" RoleAdjacent = "adjacent" RoleCollectedOnPurpose = "collected_on_purpose" RoleGeneral = "general" RoleOffTopic = "off_topic" RoleUnknown = "unknown" )
Relevance roles (the `relevance` values).
const ( NoulPaywall = "paywall_or_login" NoulErrorPage = "error_or_placeholder_page" NoulThin = "thin_or_boilerplate" NoulNoSpeech = "no_speech_content" )
Quality nouls of the quality bank.
const ( ReasonPaywall = "paywall" ReasonErrorPage = quality.ErrorPage ReasonThin = quality.Thin ReasonNotAnArticle = quality.NotAnArticle ReasonNoSpeech = "no_speech" )
Quality reasons written to `quality` (and used as `triage_reason`).
const ( MaxSummaryChars = 400 MaxCriterionChars = 300 MaxEntities = 15 MinQuestions = 3 MaxQuestions = 6 MaxAliases = 8 MaxAliasWords = 6 MaxTitleChars = 120 )
Limits of generated literals (spec §4.2, §5.2, §6, §8).
const ( KindLiterals = "summary" KindCriterion = "criterion" KindAliases = "aliases" KindProposals = "concept_proposals" KindVocabulary = "vocabulary" )
Generation kinds (receipts record purpose "generate:<kind>").
const ( MinVocabulary = 12 MaxVocabulary = 40 )
Vocabulary bootstrap limits (spec §5.2).
const ConceptsDir = "wiki/concepts"
ConceptsDir is the topic-relative directory of wiki concept articles.
const ExcerptTokens = 12000
ExcerptTokens bounds the document excerpt of every classify and gate state (spec §7 stage 5: a section-aware excerpt of at most 12k tokens).
Variables ¶
var ErrArticleExists = errors.New("classify: article already exists")
ErrArticleExists is returned by CreateStubArticle when the article file is already there; it is never overwritten.
var ErrNoVocabularyDraft = errors.New("classify: topic.yaml has no vocabulary_draft; run `kb topic vocabulary <topic> --draft` first")
ErrNoVocabularyDraft is returned by AcceptVocabulary without a draft.
Functions ¶
func AcceptVocabulary ¶
AcceptVocabulary creates one stub article per `vocabulary_draft:` item (skipping articles that already exist), clears the draft and returns the topic-relative paths it created.
func ArticleBanks ¶
ArticleBanks are the banks recorded in the state row of a classified article (plan: `{classify, concept}`).
func CodeFlags ¶
CodeFlags runs the code quality checks (spec §7 stage 3) over a stored source; hostLines comes from quality.HostLineCounts over the topic's sources. The thin rule only applies to web captures (an http(s) source_url, not a transcript or a bookmark cluster): the length of a local note or a transcript is not a capture failure. `kb classify` and the impact preview share it so their bands agree.
func CreateStub ¶
CreateStub creates a stub article through CreateStubArticle and then writes its criterion through the session's owned-key writer, so the state record knows kb wrote it: a later regeneration may refresh it, and a user edit is detected and kept (spec §5.2, §6). Accepting a vocabulary draft or a concept proposal both go through here.
func CreateStubArticle ¶
CreateStubArticle writes `wiki/concepts/<Title>.md` (file name = title sanitized for the filesystem) with the stub frontmatter of the plan: title, type wiki, stage stub, domain, tags [domain, wiki, stub], created, updated, sources [] and criterion (when non-empty; prefer CreateStub, which records the criterion as kb-written). It returns the topic-relative path, and ErrArticleExists (with that path) when the file is already there.
func DraftVocabulary ¶
DraftVocabulary asks the generation model for a closed list of 12–40 concepts (title + criterion) from the contract and one line per source (title + summary), validates it by code (unique titles, criterion rules, no collision with existing articles), counts for each concept how many source summaries mention its title (code, no call) and saves the list under topic.yaml `vocabulary_draft:`.
func FacetBank ¶
FacetBank is the composite bank (classify + concept) of the classification request.
func IsTranscriptKind ¶
IsTranscriptKind reports whether a source kind is a transcript, the only documents asked `no_speech_content`.
func SourceBanks ¶
SourceBanks are the banks recorded in the state row of a classified source (plan: `{relevance, quality, classify, concept}`).
Types ¶
type GateJudgment ¶
type GateJudgment struct {
// Role is the argmax role, RoleCollectedOnPurpose when the path matches
// `collected_on_purpose_paths` (no call), or "" when relevance is off or
// the role answer is undecided.
Role string `json:"role,omitempty"`
// RoleAsked reports whether the `role` question was in the request.
RoleAsked bool `json:"role_asked"`
// RoleByPath reports that Role came from the path globs.
RoleByPath bool `json:"role_by_path,omitempty"`
// RoleProbs is the role distribution when decided.
RoleProbs map[string]float64 `json:"role_probs,omitempty"`
// POffTopic is P(off_topic).
POffTopic float64 `json:"p_off_topic"`
// PKept is P(core) + P(adjacent) + P(collected_on_purpose).
PKept float64 `json:"p_kept"`
// RoleDecided reports a usable role (decided answer or path glob).
RoleDecided bool `json:"role_decided"`
// Quality maps each decided quality noul id to P(yes).
Quality map[string]float64 `json:"quality"`
// QualityDecided reports that every quality noul was decided.
QualityDecided bool `json:"quality_decided"`
// Flags are the code flags passed in GateOptions.
Flags []quality.Flag `json:"flags,omitempty"`
// Undecided lists "<question>:<reason>" for every answer that was not
// decided (not-checked answers read "<question>:not_checked:<reason>").
Undecided []string `json:"undecided,omitempty"`
// ReceiptKey is the receipts key of the request.
ReceiptKey string `json:"receipt_key,omitempty"`
}
GateJudgment is the outcome of the shared gate request.
func JudgeGate ¶
func JudgeGate(ctx context.Context, s *session.Session, doc *corpus.Document, opts GateOptions) (GateJudgment, error)
JudgeGate asks the shared gate request for one document: the `role` choice against the contract (omitted when relevance is off in topic.yaml or the path matches `collected_on_purpose_paths`) plus the quality nouls (`no_speech_content` only for transcripts), over the state `{contract, document}` built by corpus.DocumentState. The contract is only in the state when `role` is asked; without an accepted contract it is `{}` (relevance then runs in shadow, decided by the caller's mode). Ingest gates, the impact preview and classify all call this, so the same content reuses one receipt. The error is fatal only (auth, cancellation, invalid request); failed answers are listed in Undecided.
func (GateJudgment) QualityQuestion ¶
func (g GateJudgment) QualityQuestion(reason string) string
QualityQuestion returns the question id behind a reason ("code:<flag>" for a code flag), for review items.
func (GateJudgment) QualityReason ¶
func (g GateJudgment) QualityReason(threshold float64) (string, float64)
QualityReason returns the reason code would act on at threshold: the first code flag (probability 1), else the quality noul with the highest P(yes) ≥ threshold. It returns "" when nothing fired.
type GateOptions ¶
type GateOptions struct {
// IsTranscript adds the `no_speech_content` noul.
IsTranscript bool
// Flags are the code quality flags already computed by the caller
// (quality.Check); they are carried into the judgment and win over the
// nouls in QualityReason.
Flags []quality.Flag
// Contract overrides the session's active contract (the impact preview
// judges a draft). Its hash replaces the session's in the cache key, so
// the receipts are reused once the same draft is accepted.
Contract *contract.Contract
}
GateOptions configures JudgeGate.
type Options ¶
type Options struct {
// OnlyMissing judges only documents without a state row or with a
// missing facet key (the bootstrap follow-up after vocabulary accept).
OnlyMissing bool
// All re-judges every document regardless of the state record.
All bool
// Paths restricts the run to these topic-relative paths (ingest).
Paths []string
}
Options configures Run.
type Report ¶
type Report struct {
// Documents is the number of documents considered (after Paths).
Documents int `json:"documents"`
// Judged is the number of documents that went through the facet pass.
Judged int `json:"judged"`
// Decided is the number of judged documents whose every answer and
// literal was decided (the rest stay unclassified for the next run).
Decided int `json:"decided"`
// UndecidedDocuments lists, sorted, the documents left with an
// undecided answer or literal (spec §2.3: named in the run summary).
UndecidedDocuments []string `json:"undecided_documents"`
// Skipped counts documents not judged, by reason: unchanged (state row
// current), complete (--only-missing and nothing missing), locked, stub.
Skipped map[string]int `json:"skipped"`
// Facets counts written keys (a key rewritten with the same value is not
// counted).
Facets map[string]int `json:"facets"`
// Writes counts document write outcomes: written, unchanged,
// skipped:locked, skipped:changed.
Writes map[string]int `json:"writes"`
// SkippedUserKeys counts keys left alone because the user owns them
// (skipped:user-key), by key.
SkippedUserKeys map[string]int `json:"skipped_user_keys"`
// Undecided counts answers and literals that were not decided, by
// "<question>:<reason>" (not-checked answers read
// "<question>:not_checked:<reason>"). Undecided answers never write.
Undecided map[string]int `json:"undecided"`
// Queued counts review items added, by queue.
Queued map[string]int `json:"queued"`
// DroppedEntities counts generated entity names dropped by the verbatim
// check or the cap.
DroppedEntities int `json:"dropped_entities"`
// InvalidLiterals counts generated literals rejected by code, by key.
InvalidLiterals map[string]int `json:"invalid_literals"`
// CriterionMissing lists articles without `criterion` whose option text
// fell back to their summary or head.
CriterionMissing []string `json:"criterion_missing"`
// ArticlesChanged lists articles that gained aliases (they trigger the
// link reverse pass).
ArticlesChanged []string `json:"articles_changed"`
// PrimaryNone counts sources whose primary concept was `none` in this
// run.
PrimaryNone int `json:"primary_none"`
// PrimaryNoneTotal counts the topic's sources whose primary concept is
// `none` (this run plus earlier runs), the concept proposal trigger.
PrimaryNoneTotal int `json:"primary_none_total"`
// NoVocabulary reports a topic without articles: no concept questions
// were asked.
NoVocabulary bool `json:"no_vocabulary"`
// RelevanceOff reports decisions.relevance: off (no role questions).
RelevanceOff bool `json:"relevance_off"`
}
Report summarizes one classify run. Counts are per document unless named otherwise.
func Run ¶
Run classifies the documents of the session's topic (spec §8). Per document it asks two requests: the shared gate judgment (JudgeGate: role + quality nouls; sources only) and one classification request (composite classify + concept bank: kind, depth, enough_text_to_judge, primary_concept, mentions_concept_{id}), plus one generation call for summary/entities/questions when they are missing or kb wrote them and the body changed. Articles first get their criterion and aliases generated, so the concept options of the same run use them. Code maps answers to frontmatter and writes through the session's writer; undecided answers never write and leave the document unclassified for the next run. Sources above the review band go to the recapture or remove review queues (never quarantined), and ≥5 sources with primary concept `none` produce concept proposals. Incremental by body hash, contract and bank versions unless opts.All. The error is fatal only (auth, cancellation, I/O); the report is returned alongside it with what was done.
func (Report) UndecidedTotal ¶
UndecidedTotal is the number of undecided answers and literals.
Source Files