redact

Documentation

Overview

Package redact keeps secrets on the machine. What the gateway sends a vendor has its secrets (API keys, private keys, tokens, passwords in connection strings and assignments), and when asked personal data and the user's own words, swapped for placeholders like {{API_KEY_k3v9x2mq}}; what the vendor says back has them swapped back, streams included, so the agent and the user see the real thing while the vendor never does.

A placeholder is the same for the same value every time (a keyed hash of it), so a conversation's history reads the same turn after turn — what a vendor cached of it stays good, and a thinking block's signature still matches the text it was made for — and one placeholder is never two values. The values are held in memory only, by placeholder, as requests bring them; a restart loses nothing, since the next request brings them again. A value once masked is masked wherever a request has it, whatever words are around it (known.go): the vendor writes about a placeholder in words no rule knows, and the agent sends that text back with the value.

Index

• Constants
• Variables
• func Mask(s string, o Options) (string, int)
• func MaskJSON(body []byte, o Options) ([]byte, int)
• func Restore(s string, escaped bool) string
• func RestoreJSON(body []byte) []byte
• func RuleKind(kind string) string
• func Scrub(s string) string
• func ScrubHeader(name, value string) string
• func ScrubJSON(body []byte) []byte
• func SecretName(name string) bool
• func SetKeyPath(path string)
• type Category
• type Options
• type Rule
  • func CheckRules(in []Rule) ([]Rule, error)
• type Writer
  • func NewWriter(w http.ResponseWriter) *Writer
  • func (rw *Writer) Finish()
  • func (rw *Writer) Flush()
  • func (rw *Writer) Header() http.Header
  • func (rw *Writer) Unwrap() http.ResponseWriter
  • func (rw *Writer) Write(p []byte) (int, error)
  • func (rw *Writer) WriteHeader(code int)

Constants

const (
	MaxRules = 32
)

How many rules, and how long each part of one, a user may have.

const Scrubbed = "[REDACTED]"

Scrubbed is what a field or header named for a secret holds once scrubbed.

Variables

var Categories = []Category{
	{ID: "ssn", Kinds: []string{"SSN"}, On: true},
	{ID: "passport", Kinds: []string{"PASSPORT"}, On: true},
	{ID: "iban", Kinds: []string{"IBAN"}, On: true},
	{ID: "birthday", Kinds: []string{"BIRTHDAY"}, On: true},
	{ID: "mac", Kinds: []string{"MAC"}, On: true},
	{ID: "serial", Kinds: []string{"SERIAL"}, On: true},
	{ID: "home", Kinds: []string{"USERNAME"}},
	{ID: "userhost", Kinds: []string{"USERNAME", "HOSTNAME"}},
	{ID: "ip", Kinds: []string{"IP"}},
	{ID: "bucket", Kinds: []string{"BUCKET"}},
}

Categories are those of personal data the user can turn on or off, in the order Settings lists them.

Functions

func Mask

func Mask(s string, o Options) (string, int)

Mask swaps what o covers in s for placeholders, and says how many.

func MaskJSON

func MaskJSON(body []byte, o Options) ([]byte, int)

MaskJSON masks the strings of a request body, and says how many values it masked. A body that isn't JSON is masked as text.

What a vendor sealed with a signature (a thinking block, Gemini's thoughts, Responses' encrypted reasoning) is checked by the vendor against the text it wrote, so it goes back as written: the values masked elsewhere in the request become their placeholders again, which is what the vendor wrote where the agent reads the value (Restore), as do the secrets masked before (knownSpans), and nothing else in it is touched. Masked by the rules instead, a value the vendor wrote as a placeholder in other words around it ("the password is {{SECRET_…}}") went back as the value itself, and what the vendor wrote of its own that looks like an email or a phone number went as a placeholder: either way the text no longer matched its signature, and Anthropic answers 400 "Invalid `signature` in `thinking` block".

func Restore

func Restore(s string, escaped bool) string

Restore puts the values back in s. escaped is for text that is itself JSON (a tool call's arguments), where a value goes in escaped.

A model doesn't always write a placeholder back as it was given: it writes {{ PHONE_b5puzbfv }} as a template has it, drops a brace, or the braces, or changes the case. Any of those is put back too when what is left names a placeholder this magpie made: its kind and the 8 of its hash, which no other text has by chance (shensmart on Discord).

func RestoreJSON

func RestoreJSON(body []byte) []byte

RestoreJSON puts the values back in a response body.

func RuleKind

func RuleKind(kind string) string

RuleKind is kind as a placeholder can have it: upper case letters, digits and _, starting with a letter, at most 24 long; "" is CUSTOM.

func Scrub

func Scrub(s string) string

Scrub takes the secrets Mask finds out of s.

func ScrubHeader

func ScrubHeader(name, value string) string

ScrubHeader is a header's value with its secrets taken out: all of it for a header named for one or a credential (Bearer, Basic), else what Scrub finds in it.

func ScrubJSON

func ScrubJSON(body []byte) []byte

ScrubJSON is a body with its secrets taken out: a JSON one string by string, a field named for a secret wholly; a stream's events each as JSON, line by line; anything else as text. A picture's data is left as it is.

func SecretName

func SecretName(name string) bool

SecretName says a field or header called name holds a secret: a token, key, password or cookie — not a count of tokens (max_tokens).

func SetKeyPath

func SetKeyPath(path string)

SetKeyPath chooses where the placeholder key is kept across restarts; "" keeps it in memory only. The gateway's owner sets it before requests arrive. Once the key is loaded, both it and its path stay fixed.

Types

type Category

type Category struct {
	ID    string
	Kinds []string
	On    bool
}

Category is one kind of personal data the user turns on or off on its own: its id in settings (redactKinds), the kinds of its placeholders, and whether it is masked while the user hasn't chosen.

type Options

type Options struct {
	Secrets  bool     // API keys, private keys, tokens, passwords
	Personal bool     // emails, phone numbers, ID and bank card numbers
	Words    []string // the user's own: names, codenames, hosts
	Rules    []Rule   // the user's own rules, masked with the secrets
	// Kinds turns each of the Categories of personal data on or off by its
	// id, where the user chose; one not in it is as its Category.On says.
	// None of them is masked while Personal is off.
	Kinds map[string]bool
}

Options says what to mask.

type Rule

type Rule struct {
	Kind   string `json:"kind"`
	Prefix string `json:"prefix,omitempty"`
	// Regex is an RE2 pattern; the value is its group 1 when it has one,
	// else all of what it matches, as with magpie's own rules.
	Regex string `json:"regex,omitempty"`
}

Rule is one the user added (Settings → Privacy) for a secret magpie's own rules don't know, a gateway's key say (#195): what its placeholders are called, and the prefix its values start with or a pattern for them. Its values are masked with the secrets, as they are: the same placeholder for the same value, put back in what the vendor answers.

func CheckRules

func CheckRules(in []Rule) ([]Rule, error)

CheckRules tidies the user's rules (the kind as a placeholder has it, spaces trimmed, the empty and the repeated dropped) and says what is wrong with one that can't be used: a pattern that doesn't compile, or matches nothing at all, a prefix too short, too many or too long.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer puts the values back in what a vendor answers, on its way to the agent. A stream is put back event by event, as it comes; anything else when it is done, at Finish.

A placeholder a stream splits between two events — {{API_ in one text delta, KEY_k3v9x2mq}} in the next — is put back whole: an event whose text ends in what may be the start of one is held until the next, and if that one goes on with the same text the start moves over to it. Nothing is added or dropped, so the text the agent puts together is the same.

func NewWriter

func NewWriter(w http.ResponseWriter) *Writer

NewWriter wraps w; Finish must be called when the handler is done.

func (*Writer) Finish

func (rw *Writer) Finish()

Finish writes what is still held.

func (*Writer) Flush

func (rw *Writer) Flush()

func (*Writer) Header

func (rw *Writer) Header() http.Header

func (*Writer) Unwrap

func (rw *Writer) Unwrap() http.ResponseWriter

Unwrap is for http.ResponseController.

func (*Writer) Write

func (rw *Writer) Write(p []byte) (int, error)

func (*Writer) WriteHeader

func (rw *Writer) WriteHeader(code int)