calibration

Documentation

Index

• func ApplyThresholds(results []sarif.Result, thresholds map[string]ThresholdOverride) []sarif.Result
• func FormatCalibrationExamples(examples []FewShotExample) string
• func SuppressedResults(results []sarif.Result, thresholds map[string]ThresholdOverride) []sarif.Result
• type AnalysisPayload
• type CalibrationResponse
• type Client
  • func NewClient(baseURL, apiKey string, timeout time.Duration) *Client
  • func (c *Client) GetCalibration(ctx context.Context, teamID string, ruleIDs []string, fileType string) (*CalibrationResponse, error)
  • func (c *Client) UploadEvents(ctx context.Context, teamID string, events []Event) error
• type CrossOrgSignal
• type Event
  • func BuildEventsFromSARIF(log *sarif.Log, resultID, persona, provider, model string, shareCode bool) []Event
• type EventType
  • func (t EventType) Valid() bool
• type FeedbackPayload
• type FewShotExample
• type FindingPayload
• type LocalQueue
  • func NewLocalQueue(dir string) *LocalQueue
  • func (q *LocalQueue) Drain() ([]QueuedBatch, error)
  • func (q *LocalQueue) Enqueue(teamID string, events []Event) error
  • func (q *LocalQueue) Remove(id string) error
• type OutcomePayload
• type QueuedBatch
• type RuleProfile
  • func (p *RuleProfile) Recalculate()
• type ThresholdOverride

Functions

func ApplyThresholds

func ApplyThresholds(results []sarif.Result, thresholds map[string]ThresholdOverride) []sarif.Result

ApplyThresholds filters out findings below their rule's suppress threshold. Results with no matching threshold entry, or a zero SuppressBelow value, are always kept. When thresholds is empty the original slice is returned unchanged.

func FormatCalibrationExamples

func FormatCalibrationExamples(examples []FewShotExample) string

FormatCalibrationExamples formats few-shot examples for prompt injection.

It renders a structured calibration context block containing labelled historical examples from team feedback. Examples with verdict "useful" are labelled "USEFUL PATTERN" to reinforce correct detection; examples with verdict "noise" or "wrong" are labelled "NOISE PATTERN" to discourage similar false positives. The returned string is intended to be prepended to an LLM analysis prompt so the model can calibrate its confidence against known outcomes for the same rule and file type.

Returns an empty string when examples is nil or empty.

func SuppressedResults

func SuppressedResults(results []sarif.Result, thresholds map[string]ThresholdOverride) []sarif.Result

SuppressedResults returns the subset of findings that would be suppressed by the given thresholds — i.e. findings whose confidence is strictly below their rule's SuppressBelow value. Returns nil when thresholds is empty.

Types

type AnalysisPayload

type AnalysisPayload struct {
	ResultID     string   `json:"result_id"`
	RuleIDs      []string `json:"rule_ids"`
	FileTypes    []string `json:"file_types"`
	FindingCount int      `json:"finding_count"`
	Provider     string   `json:"provider"`
	Model        string   `json:"model"`
	Persona      string   `json:"persona"`
}

AnalysisPayload is the payload for EventAnalysisCompleted.

type CalibrationResponse

type CalibrationResponse struct {
	// TeamThresholds maps rule IDs to per-rule threshold overrides derived from
	// this team's historical feedback.
	TeamThresholds map[string]ThresholdOverride `json:"team_thresholds"`

	// CrossOrgSignals maps rule IDs to anonymised aggregate statistics across
	// all teams, giving a global noise baseline for each rule.
	CrossOrgSignals map[string]CrossOrgSignal `json:"cross_org_signals"`

	// FewShotExamples are retrieved similar findings from the team's history,
	// ordered by descending similarity score. Omitted when empty.
	FewShotExamples []FewShotExample `json:"few_shot_examples,omitempty"`
}

CalibrationResponse is returned by GET /v1/calibration/{team_id}. It carries all data the CLI needs to tune its analysis thresholds and augment prompts for a single team.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client talks to the calibration server.

func NewClient

func NewClient(baseURL, apiKey string, timeout time.Duration) *Client

NewClient constructs a Client targeting baseURL, authenticating with apiKey, and applying timeout to every HTTP request.

A trailing slash on baseURL is trimmed so that path concatenation is always consistent regardless of how the caller configures the URL.

func (*Client) GetCalibration

func (c *Client) GetCalibration(ctx context.Context, teamID string, ruleIDs []string, fileType string) (*CalibrationResponse, error)

GetCalibration fetches team-specific threshold overrides and cross-org signals from GET /v1/calibration/{teamID}.

ruleIDs, when non-empty, restricts the response to the named rules via the "rules" query parameter (comma-separated). fileType is passed as the "file_type" query parameter so the server can filter few-shot examples.

Returns a parsed CalibrationResponse on success or an error when the request fails, times out, or the server returns a non-200 status.

func (*Client) UploadEvents

func (c *Client) UploadEvents(ctx context.Context, teamID string, events []Event) error

UploadEvents sends a batch of calibration events to POST /v1/events/batch.

The request body is a JSON object with a "team_id" string and an "events" array. The server is expected to respond with HTTP 202 Accepted; any other status code is returned as an error.

type CrossOrgSignal

type CrossOrgSignal struct {
	// RuleID is the rule these stats describe.
	RuleID string `json:"rule_id"`

	// GlobalNoiseRate is the noise rate averaged across all teams that have
	// feedback for this rule.
	GlobalNoiseRate float64 `json:"global_noise_rate"`

	// TotalTeams is the number of distinct teams contributing to this signal.
	TotalTeams int `json:"total_teams"`

	// TotalFeedbackEvents is the sum of all feedback events across teams.
	TotalFeedbackEvents int `json:"total_feedback_events"`

	// Warning carries an optional human-readable note, e.g. when TotalTeams is
	// too small for the aggregate to be statistically meaningful.
	Warning string `json:"warning,omitempty"`

	// ComputedAt is when this cross-org signal was last computed.
	ComputedAt time.Time `json:"computed_at"`
}

CrossOrgSignal aggregates anonymised calibration signals across all teams for a rule. It is used to seed new-team priors and surface global noise patterns.

type Event

type Event struct {
	Type      EventType   `json:"type"`
	TeamID    string      `json:"team_id"`
	Timestamp time.Time   `json:"timestamp"`
	Payload   interface{} `json:"payload"`
}

Event is a single calibration event uploaded by the CLI.

func BuildEventsFromSARIF

func BuildEventsFromSARIF(log *sarif.Log, resultID, persona, provider, model string, shareCode bool) []Event

BuildEventsFromSARIF creates calibration events from a SARIF log.

It returns one EventAnalysisCompleted event followed by one EventFindingCreated event per result in the first run of the log. Returns nil when the log contains no runs.

Parameters:

• log: the SARIF log produced by an analysis run.
• resultID: the store result ID that identifies this analysis.
• persona: the gavel persona used during analysis (e.g. "code-reviewer").
• provider: the LLM provider name (e.g. "openrouter", "anthropic").
• model: the LLM model name used for the analysis.
• shareCode: reserved for future use; when true callers may include code snippets in payloads (currently unused).

type EventType

type EventType string

EventType identifies the kind of calibration event.

const (
	EventAnalysisCompleted EventType = "analysis_completed"
	EventFindingCreated    EventType = "finding_created"
	EventFeedbackReceived  EventType = "feedback_received"
	EventOutcomeObserved   EventType = "outcome_observed"
)

func (EventType) Valid

func (t EventType) Valid() bool

Valid reports whether t is a recognised event type.

type FeedbackPayload

type FeedbackPayload struct {
	ResultID     string `json:"result_id"`
	FindingIndex int    `json:"finding_index"`
	RuleID       string `json:"rule_id"`
	Verdict      string `json:"verdict"`
	Reason       string `json:"reason,omitempty"`
}

FeedbackPayload is the payload for EventFeedbackReceived.

type FewShotExample

type FewShotExample struct {
	RuleID      string  `json:"rule_id"`
	FileType    string  `json:"file_type"`
	CodeSnippet string  `json:"code_snippet,omitempty"`
	Message     string  `json:"message"`
	Verdict     string  `json:"verdict"`
	Reason      string  `json:"reason,omitempty"`
	Similarity  float64 `json:"similarity"`
}

FewShotExample is a retrieved past finding used to augment LLM prompts with concrete examples of how a rule was previously evaluated for a given file type. Similarity is a [0,1] score indicating how closely the stored example matches the current finding context.

type FindingPayload

type FindingPayload struct {
	ResultID    string  `json:"result_id"`
	RuleID      string  `json:"rule_id"`
	Severity    string  `json:"severity"`
	Confidence  float64 `json:"confidence"`
	FileType    string  `json:"file_type"`
	StartLine   int     `json:"start_line"`
	EndLine     int     `json:"end_line"`
	Message     string  `json:"message"`
	Explanation string  `json:"explanation,omitempty"`
	CodeSnippet string  `json:"code_snippet,omitempty"`
}

FindingPayload is the payload for EventFindingCreated.

type LocalQueue

type LocalQueue struct {
	// contains filtered or unexported fields
}

LocalQueue is a file-backed FIFO queue of QueuedBatch values. Each call to Enqueue atomically writes a single JSON file to the queue directory; Drain reads all pending files; Remove deletes a single file by batch ID.

LocalQueue is not safe for concurrent use by multiple processes sharing the same directory without additional locking at the call site.

func NewLocalQueue

func NewLocalQueue(dir string) *LocalQueue

NewLocalQueue returns a LocalQueue that persists batches under dir. The directory is created lazily on the first call to Enqueue.

func (*LocalQueue) Drain

func (q *LocalQueue) Drain() ([]QueuedBatch, error)

Drain returns all QueuedBatch values currently stored in the queue directory. Files that cannot be read or unmarshalled are silently skipped so that a single corrupt file does not prevent delivery of valid batches. The caller is responsible for calling Remove on each batch after successful delivery.

If the queue directory does not yet exist, Drain returns nil, nil.

func (*LocalQueue) Enqueue

func (q *LocalQueue) Enqueue(teamID string, events []Event) error

Enqueue marshals events into a QueuedBatch and writes it to a JSON file in the queue directory. The filename is derived from the current time in nanoseconds, which provides a naturally ordered set of pending files.

func (*LocalQueue) Remove

func (q *LocalQueue) Remove(id string) error

Remove deletes the on-disk file for the batch identified by id. It should be called after a batch has been successfully delivered to the calibration server.

type OutcomePayload

type OutcomePayload struct {
	ResultID        string `json:"result_id"`
	FindingIndex    int    `json:"finding_index"`
	RuleID          string `json:"rule_id"`
	OutcomeType     string `json:"outcome_type"`
	TimeToResolveMs int64  `json:"time_to_resolve_ms,omitempty"`
}

OutcomePayload is the payload for EventOutcomeObserved.

type QueuedBatch

type QueuedBatch struct {
	// ID is a nanosecond-precision Unix timestamp string that also serves as
	// the on-disk filename stem.
	ID string `json:"id"`

	// TeamID is the team that owns the events in this batch.
	TeamID string `json:"team_id"`

	// Events is the list of calibration events to upload.
	Events []Event `json:"events"`
}

QueuedBatch is a persisted group of Events waiting to be flushed to the calibration server. Batches are written as individual JSON files under the queue directory so that a process crash cannot corrupt previously enqueued batches.

type RuleProfile

type RuleProfile struct {
	// TeamID identifies the owning team.
	TeamID string `json:"team_id"`

	// RuleID is the rule this profile describes.
	RuleID string `json:"rule_id"`

	// TotalFindings is the cumulative number of findings emitted for this rule.
	TotalFindings int `json:"total_findings"`

	// UsefulCount is the number of findings rated as useful (true positive) via feedback.
	UsefulCount int `json:"useful_count"`

	// NoiseCount is the number of findings rated as noise (false positive) via feedback.
	NoiseCount int `json:"noise_count"`

	// WrongCount is the number of findings rated as wrong (incorrect diagnosis) via feedback.
	WrongCount int `json:"wrong_count"`

	// NoiseRate is NoiseCount / TotalFindings, or 0 if TotalFindings == 0.
	// Populated by Recalculate.
	NoiseRate float64 `json:"noise_rate"`

	// ConfCalibration is the difference between MeanUsefulConf and MeanNoiseConf.
	// A value near 0 indicates the model's confidence score does not discriminate
	// between useful and noisy findings for this rule.
	// Populated by Recalculate; 0 when confidence data is absent.
	ConfCalibration float64 `json:"conf_calibration"`

	// MeanNoiseConf is the mean model confidence across noise-rated findings.
	MeanNoiseConf float64 `json:"mean_noise_conf,omitempty"`

	// MeanUsefulConf is the mean model confidence across useful-rated findings.
	MeanUsefulConf float64 `json:"mean_useful_conf,omitempty"`

	// SuppressBelow is a confidence threshold below which findings for this rule
	// should be suppressed in future analyses. 0 means no suppression.
	// Populated by Recalculate when NoiseRate exceeds the suppression threshold.
	SuppressBelow float64 `json:"suppress_below,omitempty"`

	// LastUpdated is when this profile was last recalculated.
	LastUpdated time.Time `json:"last_updated"`
}

RuleProfile is a materialized per-team calibration profile for a single rule. It is derived by replaying FeedbackPayload and OutcomePayload events and updated incrementally on the hot path via UpdateProfileFromFeedback.

Call Recalculate after mutating count fields to keep derived fields in sync.

func (*RuleProfile) Recalculate

func (p *RuleProfile) Recalculate()

Recalculate refreshes derived fields (NoiseRate, ConfCalibration, SuppressBelow) from the raw count and confidence fields. It must be called after any mutation to count or confidence fields to keep the profile consistent.

type ThresholdOverride

type ThresholdOverride struct {
	// SuppressBelow is the confidence threshold below which findings for the
	// associated rule should be omitted from the analysis output.
	SuppressBelow float64 `json:"suppress_below"`
}

ThresholdOverride carries per-rule confidence thresholds that the calibration server pushes to the CLI so it can suppress low-quality findings locally.