redact

Documentation

Overview

Package redact keeps secrets on the machine. What the gateway sends a vendor has its secrets (API keys, private keys, tokens, passwords in connection strings and assignments), and when asked personal data and the user's own words, swapped for placeholders like {{API_KEY_k3v9x2mq}}; what the vendor says back has them swapped back, streams included, so the agent and the user see the real thing while the vendor never does.

A placeholder is the same for the same value every time (a keyed hash of it), so a conversation's history reads the same turn after turn — what a vendor cached of it stays good, and a thinking block's signature still matches the text it was made for — and one placeholder is never two values. The values are held in memory only, by placeholder, as requests bring them; a restart loses nothing, since the next request brings them again. A value once masked is masked wherever a request has it, whatever words are around it (known.go): the vendor writes about a placeholder in words no rule knows, and the agent sends that text back with the value.

Index

• Constants
• Variables
• func Hint(kind, v string) (head string, n int)
• func KindName(k string) bool
• func Mask(s string, o Options) (string, int)
• func MaskJSON(body []byte, o Options) ([]byte, int)
• func Preview(s string, o Options) string
• func Restore(s string, escaped bool) string
• func RestoreJSON(body []byte) []byte
• func RuleKind(kind string) string
• func Scrub(s string) string
• func ScrubHeader(name, value string) string
• func ScrubJSON(body []byte) []byte
• func SecretName(name string) bool
• func SetKeyPath(path string)
• func Value(p string) (string, bool)
• type Category
• type Found
• type Hit
• type Model
• type Options
• type Rule
  • func CheckRules(in []Rule) ([]Rule, error)
• type Writer
  • func NewWriter(w http.ResponseWriter) *Writer
  • func (rw *Writer) Finish()
  • func (rw *Writer) Flush()
  • func (rw *Writer) Header() http.Header
  • func (rw *Writer) Put() map[string]int
  • func (rw *Writer) Unwrap() http.ResponseWriter
  • func (rw *Writer) Write(p []byte) (int, error)
  • func (rw *Writer) WriteHeader(code int)

Constants

const (
	ByRule  = "rule"
	ByYours = "yours"
	ByWord  = "word"
	ByModel = "model"
	ByKnown = "known"
)

Which finder found a value (Hit.By): magpie's rules, the user's own rules, the user's words, the local model (with the Chinese names it brings, which run only with it), or a value masked before (known.go).

const (
	Warn  = "warn"  // sent as written, and recorded
	Block = "block" // masked, and the request held back
)

What is done with a value of a kind (Options.Actions) other than masking it.

const (
	MaxRules = 32
)

How many rules, and how long each part of one, a user may have.

const Scrubbed = "[REDACTED]"

Scrubbed is what a field or header named for a secret holds once scrubbed.

Variables

var Categories = []Category{
	{ID: "ssn", Kinds: []string{"SSN"}, On: true},
	{ID: "passport", Kinds: []string{"PASSPORT"}, On: true},
	{ID: "iban", Kinds: []string{"IBAN"}, On: true},
	{ID: "birthday", Kinds: []string{"BIRTHDAY"}, On: true},
	{ID: "mac", Kinds: []string{"MAC"}, On: true},
	{ID: "serial", Kinds: []string{"SERIAL"}, On: true},
	{ID: "home", Kinds: []string{"USERNAME"}},
	{ID: "userhost", Kinds: []string{"USERNAME", "HOSTNAME"}},
	{ID: "ip", Kinds: []string{"IP"}},
	{ID: "bucket", Kinds: []string{"BUCKET"}},
}

Categories are those of personal data the user can turn on or off, in the order Settings lists them.

Functions

func Hint

func Hint(kind, v string) (head string, n int)

Hint is as much of v as may be kept where v itself may not: its first character (the first few of a key's, which name its kind: sk-, ghp_) and how many characters it has.

func KindName

func KindName(k string) bool

KindName says k is written as a placeholder names a kind (API_KEY, a rule's own), as a setting keyed by kind must name it.

func Mask

func Mask(s string, o Options) (string, int)

Mask swaps what o covers in s for placeholders, and says how many.

func MaskJSON

func MaskJSON(body []byte, o Options) ([]byte, int)

MaskJSON masks the strings of a request body, and says how many values it masked. A body that isn't JSON is masked as text.

What a vendor sealed with a signature (a thinking block, Gemini's thoughts, Responses' encrypted reasoning) is checked by the vendor against the text it wrote, so it goes back as written: the values masked elsewhere in the request become their placeholders again, which is what the vendor wrote where the agent reads the value (Restore), as do the secrets masked before (knownSpans), and nothing else in it is touched. Masked by the rules instead, a value the vendor wrote as a placeholder in other words around it ("the password is {{SECRET_…}}") went back as the value itself, and what the vendor wrote of its own that looks like an email or a phone number went as a placeholder: either way the text no longer matched its signature, and Anthropic answers 400 "Invalid `signature` in `thinking` block".

func Preview

func Preview(s string, o Options) string

Preview is Mask with nothing remembered: each value goes as the placeholder, or with Options.Shape the made-up key, it would have, which isn't kept for Restore or as a value seen before (known.go).

func Restore

func Restore(s string, escaped bool) string

Restore puts the values back in s. escaped is for text that is itself JSON (a tool call's arguments), where a value goes in escaped.

A model doesn't always write a placeholder back as it was given: it writes {{ PHONE_b5puzbfv }} as a template has it, drops a brace, or the braces, or changes the case. Any of those is put back too when what is left names a placeholder this magpie made: its kind and the 8 of its hash, which no other text has by chance (shensmart on Discord).

func RestoreJSON

func RestoreJSON(body []byte) []byte

RestoreJSON puts the values back in a response body.

func RuleKind

func RuleKind(kind string) string

RuleKind is kind as a placeholder can have it: upper case letters, digits and _, starting with a letter, at most 24 long; "" is CUSTOM.

func Scrub

func Scrub(s string) string

Scrub takes the secrets Mask finds out of s.

func ScrubHeader

func ScrubHeader(name, value string) string

ScrubHeader is a header's value with its secrets taken out: all of it for a header named for one or a credential (Bearer, Basic), else what Scrub finds in it.

func ScrubJSON

func ScrubJSON(body []byte) []byte

ScrubJSON is a body with its secrets taken out: a JSON one string by string, a field named for a secret wholly; a stream's events each as JSON, line by line; anything else as text. A picture's data is left as it is.

func SecretName

func SecretName(name string) bool

SecretName says a field or header called name holds a secret: a token, key, password or cookie — not a count of tokens (max_tokens).

func SetKeyPath

func SetKeyPath(path string)

SetKeyPath chooses where the placeholder key is kept across restarts; "" keeps it in memory only. The gateway's owner sets it before requests arrive. Once the key is loaded, both it and its path stay fixed.

func Value

func Value(p string) (string, bool)

Value is the value behind placeholder p, while this run of magpie remembers it.

Types

type Category

type Category struct {
	ID    string
	Kinds []string
	On    bool
}

Category is one kind of personal data the user turns on or off on its own: its id in settings (redactKinds), the kinds of its placeholders, and whether it is masked while the user hasn't chosen.

type Found

type Found struct {
	Start, End int
	Label      string
}

Found is a span the model found in a text: byte offsets, and its label.

type Hit

type Hit struct {
	Kind, By, Placeholder, Action string
	Head                          string
	Len                           int
}

Hit is one value found: its kind, the finder that found it, the placeholder it went as (or would have, for one sent as written), what was done with it ("" masked, Warn, Block) and its Hint. It never holds the value itself.

type Model

type Model func(texts []string) [][]Found

Model finds spans in each of texts, in order. It never fails: one that can't find (the model didn't start) panics, so that the request is held back rather than sent unmasked (gwmw.Safely).

type Options

type Options struct {
	Secrets  bool     // API keys, private keys, tokens, passwords
	Personal bool     // emails, phone numbers, ID and bank card numbers
	Words    []string // the user's own: names, codenames, hosts
	// WordsAnyCase matches Words in any case: Nightjar masks NIGHTJAR and
	// nightjar too. Off, a word is matched as written.
	WordsAnyCase bool
	Rules        []Rule // the user's own rules, masked with the secrets
	// Kinds turns each of the Categories of personal data on or off by its
	// id, where the user chose; one not in it is as its Category.On says.
	// None of them is masked while Personal is off.
	Kinds map[string]bool
	// Model, where set, finds more in the texts to mask (model.go)
	Model Model
	// Shape sends a key as a made-up key of its form, not a placeholder
	// (shape.go)
	Shape bool

	// Actions is what is done with a value of a kind (a Hit's Kind) other
	// than masking it, where the user chose: Warn sends it as written,
	// Block masks it and the request is held back (the gateway's). A kind
	// not in it is masked.
	Actions map[string]string
	// Allow are values the user lets go as written, whichever finder finds
	// them: each a whole value, in any case.
	Allow []string

	// Seen, where set, is told of each value masked, warned of or blocked
	Seen func(Hit)
	// contains filtered or unexported fields
}

Options says what to mask.

type Rule

type Rule struct {
	Kind   string `json:"kind"`
	Prefix string `json:"prefix,omitempty"`
	// Regex is an RE2 pattern; the value is its group 1 when it has one,
	// else all of what it matches, as with magpie's own rules.
	Regex string `json:"regex,omitempty"`
}

Rule is one the user added (Settings → Privacy) for a secret magpie's own rules don't know, a gateway's key say (#195): what its placeholders are called, and the prefix its values start with or a pattern for them. Its values are masked with the secrets, as they are: the same placeholder for the same value, put back in what the vendor answers.

func CheckRules

func CheckRules(in []Rule) ([]Rule, error)

CheckRules tidies the user's rules (the kind as a placeholder has it, spaces trimmed, the empty and the repeated dropped) and says what is wrong with one that can't be used: a pattern that doesn't compile, or matches nothing at all, a prefix too short, too many or too long.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer puts the values back in what a vendor answers, on its way to the agent. A stream is put back event by event, as it comes; anything else when it is done, at Finish.

A placeholder a stream splits between two events — {{API_ in one text delta, KEY_k3v9x2mq}} in the next — is put back whole: an event whose text ends in what may be the start of one is held until the next, and if that one goes on with the same text the start moves over to it. Nothing is added or dropped, so the text the agent puts together is the same.

func NewWriter

func NewWriter(w http.ResponseWriter) *Writer

NewWriter wraps w; Finish must be called when the handler is done.

func (*Writer) Finish

func (rw *Writer) Finish()

Finish writes what is still held.

func (*Writer) Flush

func (rw *Writer) Flush()

func (*Writer) Header

func (rw *Writer) Header() http.Header

func (*Writer) Put

func (rw *Writer) Put() map[string]int

Put is how many places of each placeholder the answer had put back, so far: for the record of what masking did.

func (*Writer) Unwrap

func (rw *Writer) Unwrap() http.ResponseWriter

Unwrap is for http.ResponseController.

func (*Writer) Write

func (rw *Writer) Write(p []byte) (int, error)

func (*Writer) WriteHeader

func (rw *Writer) WriteHeader(code int)