httptools

package
v0.0.0-...-2bf6222 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 21, 2026 License: GPL-3.0 Imports: 28 Imported by: 0

Documentation

Overview

Package httptools implements the token-efficient HTTP capabilities Joro exposes to automation clients.

The premise is that a 500 KB response body costs a language model on the order of 140,000 tokens to read, which is both expensive and worse at the task than a summary would be. So no tool here returns a full body by default: they return fingerprints, byte ranges, match offsets with context, and diffs. Every one caps its own output at a row, hunk or match boundary and says how to get the rest, which keeps the output usable rather than merely small — the registry's output cap is a blunt backstop behind that, not the primary mechanism.

This package holds pure logic. It knows nothing about HTTP servers or MCP; the capability wrappers live in internal/capreg.

Index

Constants

View Source 
const (
	MaxBatchVariants      = 50
	DefaultBatchConc      = 4
	MaxBatchConc          = 10
	MaxBatchRatePerSec    = 50
	DefaultBatchTimeoutMs = 10000
	DefaultBatchBudgetMs  = 60000
	MaxBatchBudgetMs      = 120000
)

Batch caps.

Fifty and ten are chosen against the fuzzer, not arbitrarily. The fuzzer exists, has a UI, streams results, and handles a hundred threads and ten-million-item cartesian products. This is for the five-to-forty request comparisons an agent actually reasons about: header permutations, a run of object ids, a set of auth states. Past roughly fifty rows the output stops fitting a model's useful attention anyway, and the right answer becomes "drive the fuzzer" — which is what the refusal says.

View Source 
const (
	DefaultMaxHunks     = 12
	MaxMaxHunks         = 40
	DefaultContextLines = 2
	MaxContextLines     = 5
)

Diff caps.

View Source 
const (
	DefaultHistoryLimit = 50
	MaxHistoryLimit     = 200
	MaxFingerprintSeqs  = 50
)

History list caps.

View Source 
const (
	DefaultReadLength = 2048
	MaxReadLength     = 16384
)

Read caps. The default is small on purpose: 16 KB of text is already four to six thousand tokens, and a client that needs more should say so a window at a time rather than have the tool guess.

View Source 
const (
	DefaultResendTimeoutMs = 15000
	MaxResendTimeoutMs     = 60000
)

Resend timeouts.

View Source 
const (
	DefaultSearchContext = 60
	MaxSearchContext     = 200
	DefaultMatchesPerReq = 3
	DefaultMatchesSingle = 20
	DefaultTotalMatches  = 40
	DefaultSearchMaxReqs = 50
	MaxSearchMaxRequests = 200
)

Search caps.

Variables

View Source 
var EditOps = []string{
	"setHeader", "addHeader", "removeHeader",
	"setMethod", "setPath", "setQuery", "removeQuery", "setRequestTarget",
	"replaceInBody", "setBody",
}

EditOps enumerates the supported operations, for the tool schema and for the error message when a client invents one.

setVersion is deliberately absent: automation sends go through Joro's own proxy over HTTP/1.1, so there is no version for a client to choose. See proxysend.go.

Functions

func ApplyEdits 

func ApplyEdits(raw []byte, edits []Edit) ([]byte, error)

ApplyEdits rewrites raw request bytes.

The discipline mirrors internal/proxy/replace.go: split at the header terminator, rebuild the header block with canonical CRLF, and leave body bytes untouched unless a body op ran. Header names match case-insensitively and original ordering is preserved, with new headers appended at the end of the block so Host stays first — some servers care.

func Batch 

func Batch(ctx context.Context, d ResendDeps, args BatchArgs) (string, error)

Batch sends a set of edited variants of one captured request and renders a comparison table.

The execution shape mirrors fuzzer.Run — a shared ticker as the rate limiter, a channel of indices, N workers — without importing it. Nothing in internal/fuzzer is exported for reuse: executePayload is unexported and coupled to a Campaign, its broadcast channel and its matcher machinery, and extracting it would be a behavior change to the fuzzer. Duplicating the shape is the repo's existing answer to this (h2_mitm.go mirrors replace.go the same way).

func DiffMessages 

func DiffMessages(aRaw, bRaw []byte, aSeq, bSeq int, args DiffArgs) string

DiffMessages compares two captured messages and renders a compact structured diff.

Status, headers and body are three separate sections. They are different kinds of evidence, and folding them together lets a large body diff drown the header change that usually explains it.

func FingerprintSeqs 

func FingerprintSeqs(store *proxy.Store, args FingerprintArgs) (string, error)

FingerprintSeqs computes fingerprints for one or more captured responses.

func HistoryStats 

func HistoryStats(store *proxy.Store, scope *proxy.Scope, args HistoryArgs) string

HistoryStats summarizes the capture store without listing it — the cheapest way for a client to orient at the start of a session.

func ListHistory 

func ListHistory(store *proxy.Store, scope *proxy.Scope, args HistoryArgs) string

ListHistory renders a compact table of captured requests.

The handle is Seq, not the hex ID. A sequence number is one or two tokens against eight to sixteen for a hex id — over fifty rows that is hundreds of tokens of pure identifier — and it is an integer a client can compare ("the 500 is seven requests after the login") and retype without transposing a character.

func MaskHeaders 

func MaskHeaders(raw []byte) (masked []byte, names []string)

MaskHeaders overwrites the values of sensitive headers with '*', leaving header names, framing and every byte offset unchanged, and returns the names it masked.

func RedactionNote 

func RedactionNote(names []string) string

RedactionNote is the line appended to any output whose bytes were masked. Without it a masked Authorization header reads as an absent one, which is how an agent reports an authenticated endpoint as unauthenticated.

func Resend 

func Resend(ctx context.Context, d ResendDeps, args ResendArgs) (string, error)

Resend applies structural edits to a captured request and sends it through Joro's proxy, returning a fingerprint rather than a body.

func SearchCorpus 

func SearchCorpus(d SearchDeps, args SearchArgs) (string, error)

SearchCorpus greps captured traffic and returns match offsets with context.

Corpus mode is two stages, and stage one is free: the pattern goes into RequestFilter.Content, so the store's own matcher runs it over ReqRaw and RespRaw inside its read lock and never copies a non-matching body out. Stage two re-runs the pattern over the survivors, because the matcher only answers yes or no and this tool needs offsets.

Two properties of that split are traps worth naming:

  1. Stage two must be at least as permissive as stage one. The store's non-regex path lowercases both sides, so stage one is case-insensitive; a case-sensitive stage two would report zero matches for a request stage one said matched, which is a silent dead end. So case-insensitivity is the default and CaseSensitive narrows stage two only — narrowing is always safe.

  2. The store matches raw bytes, so a gzipped response will not match a plaintext pattern at stage one. Deep opts into decompressing the filter-narrowed set instead, which is a real cost and therefore never implicit.

func TargetOf 

func TargetOf(store *proxy.Store, seq int, scheme, host string, edits []Edit) (dialScheme, dialHost, method, path string, err error)

TargetOf resolves the scheme and host a resend will dial, without sending.

This is what the capability's TargetExtractor calls, so the scope guard checks the same destination the send will actually use. It reads the capture store, which is why explicit host arguments and the captured URL have to agree on precedence here and in Resend.

Types

type BatchArgs 

type BatchArgs struct {
	Seq           int            `json:"seq" alias:"ref"`
	Variants      []BatchVariant `json:"variants"`
	Scheme        string         `json:"scheme"`
	Host          string         `json:"host"`
	Concurrency   int            `json:"concurrency"`
	RatePerSec    float64        `json:"ratePerSec"`
	TimeoutMs     int            `json:"timeoutMs"`
	TotalBudgetMs int            `json:"totalBudgetMs"`
	UseContext    *bool          `json:"useContext"`
}

BatchArgs is the argument shape of http.batch.

type BatchVariant 

type BatchVariant struct {
	Label string `json:"label"`
	Edits []Edit `json:"edits"`
}

BatchVariant is one labelled set of edits.

type ContextCookie 

type ContextCookie struct {
	Host  string
	Name  string
	Value string
}

ContextCookie is one entry in a jar, as reported by context.get.

type Contexts 

type Contexts struct {
	// contains filtered or unexported fields
}

Contexts holds one cookie jar per automation principal, so a multi-request authenticated flow does not require the agent to copy cookies by hand — which it cannot do at all without credential visibility.

Jars are in-memory only and are dropped on restart, on token rotation or deletion, and on a project switch.

func NewContexts 

func NewContexts() *Contexts

func (*Contexts) Apply 

func (c *Contexts) Apply(tokenID string, u *url.URL, raw []byte, editedCookies bool) ([]byte, []string)

Apply merges jar cookies into a raw request and reports the names it supplied.

It never overrides: a cookie the request already carries wins, and if the caller edited the Cookie header at all the jar stays out entirely. Re-adding a session cookie to a request an agent deliberately stripped would manufacture a false negative, which is the worst outcome in an authorization test.

func (*Contexts) Capture 

func (c *Contexts) Capture(tokenID string, u *url.URL, respRaw []byte)

Capture records Set-Cookie headers from a response.

func (*Contexts) Clear 

func (c *Contexts) Clear(tokenID, host string) int

Clear drops the whole jar, or expires every cookie for one host. It reports how many cookies it removed.

func (*Contexts) List 

func (c *Contexts) List(tokenID string, withValues bool) []ContextCookie

List returns the jar's contents. Values are included only when the caller is permitted to see them.

func (*Contexts) Reset 

func (c *Contexts) Reset(tokenID string)

Reset drops one principal's jar. ResetAll drops every jar, which is what a project switch does: a session from a previous engagement must not be replayed into a new one.

func (*Contexts) ResetAll 

func (c *Contexts) ResetAll()

type DiffArgs 

type DiffArgs struct {
	A              int    `json:"a"`
	B              int    `json:"b"`
	Part           string `json:"part"`
	IgnoreVolatile *bool  `json:"ignoreVolatile"`
	MaxHunks       int    `json:"maxHunks"`
	ContextLines   int    `json:"contextLines"`
}

DiffArgs is the argument shape of http.diff.

type Edit 

type Edit struct {
	Op    string `json:"op"`
	Name  string `json:"name,omitempty"`
	Value string `json:"value,omitempty"`
	Find  string `json:"find,omitempty"`
	Regex bool   `json:"regex,omitempty"`
	All   bool   `json:"all,omitempty"`
	Count int    `json:"count,omitempty"`
}

Edit is one structural change to a captured request.

Edits are structural rather than "here is the whole new request" for two reasons. Shipping a full request back costs hundreds to thousands of tokens per attempt, and — more importantly — a model retyping a raw request will eventually corrupt a byte. A mangled Cookie or a dropped header turns a negative result into a false negative, which is the worst outcome in a security test because it reads as "not vulnerable". A setHeader op is about twenty-five tokens and cannot corrupt anything it does not name.

type Fingerprint 

type Fingerprint struct {
	Seq        int    `json:"seq"`
	Status     int    `json:"status"`
	Len        int    `json:"len"` // body length AFTER decompression; Content-Length lies under gzip
	DurationMs int64  `json:"ms"`
	CT         string `json:"ct"`

	// BodyHash is sha256 of the exact body, first 8 hex. StructHash folds the
	// status, the surviving header names and the normalized body — it is the field
	// that makes this tool worth building, because it collapses "same page,
	// different nonce" to a single repeated string a client compares reliably.
	BodyHash   string `json:"bhash"`
	StructHash string `json:"shash"`

	// Words and Lines use the fuzzer's exact definitions, so a fingerprint row is
	// directly comparable to a fuzzer.result row.
	Words int `json:"words"`
	Lines int `json:"lines"`

	// Note is one derived string: a redirect Location, else an HTML title, else the
	// first JSON key. This single column is what turns a table of hashes into
	// something a client can reason about.
	Note   string `json:"note,omitempty"`
	Server string `json:"server,omitempty"`

	Decoded   string `json:"decoded,omitempty"`
	NormTrunc bool   `json:"normTrunc,omitempty"`
	FullHash  string `json:"fullHash,omitempty"`
	Err       string `json:"err,omitempty"`
}

Fingerprint is the compact description of one response. Every field earns its tokens; see the field comments for what each one is for.

type FingerprintArgs 

type FingerprintArgs struct {
	Seq    int    `json:"seq" alias:"ref"`
	Seqs   []int  `json:"seqs" alias:"refs"`
	Fields string `json:"fields"` // "+fullhash"
}

FingerprintArgs is the argument shape of http.fingerprint.

type HistoryArgs 

type HistoryArgs struct {
	Host         string `json:"host"`
	Method       string `json:"method"`
	Status       string `json:"status"`
	Search       string `json:"search"`
	ContentType  string `json:"contentType"`
	Content      string `json:"content"`
	ContentRegex bool   `json:"contentRegex"`
	ContentMode  string `json:"contentMode"`
	Exclude      string `json:"exclude"`
	ExtMode      string `json:"extMode"`
	ScopeOnly    bool   `json:"scopeOnly"`
	Offset       int    `json:"offset"`
	Limit        int    `json:"limit"`
	Fields       string `json:"fields"` // "+ts,+proto"
}

HistoryArgs maps one-to-one onto proxy.RequestFilter, so this tool adds no filtering logic of its own. That is the point of routing through the store: RequestFilter.Content and ContentRegex already grep raw request and response bytes inside the store's read lock, so a content-filtered listing is a corpus-wide grep for free, with no body ever leaving the store.

type ProxySendResult 

type ProxySendResult struct {
	// Seq is the history sequence number of the resulting capture, or 0 when the
	// request could not be correlated — see SendViaProxy for when that happens.
	Seq        int
	SeqNote    string
	RespRaw    []byte
	StatusCode int
	Duration   time.Duration
	Method     string
	URL        string
}

ProxySendResult is the outcome of one automation send.

func SendViaProxy 

func SendViaProxy(ctx context.Context, raw []byte, scheme, host string, d SendDeps) (*ProxySendResult, error)

SendViaProxy writes raw request bytes through Joro's own proxy listener.

This is deliberately not proxy.SendRawRequest, which dials the target directly. That function backs POST /manipulate/send and the fuzzer, and its behavior is unchanged by this package — including its SOCKS handling, since dialH1Conn routes through TransportConfig.SOCKSDialContext.

Going through the proxy means an automation send is treated exactly like browser traffic: it is captured into History, scanned by the detect engine, entered into the site map, filtered by scope at both levels, and rewritten by Match & Replace and Custom Data. SOCKS still applies, one hop later, because the proxy's own upstream dial uses the same shared TransportConfig. The consequences an operator should expect are listed in CLAUDE.md; the notable ones are that M&R may rewrite what the client asked to send, and that an enabled request intercept will pause the send in the operator's queue until it is forwarded or this call times out.

HTTP/1.1 only. ALPN is pinned to http/1.1 rather than negotiating h2: driving the h2 MITM path as a client through a CONNECT tunnel is materially more work for no benefit to the tools built on this.

type ReadArgs 

type ReadArgs struct {
	Seq      int    `json:"seq" alias:"ref"`
	Part     string `json:"part"`     // req | resp
	Section  string `json:"section"`  // headers | body | raw
	Offset   int    `json:"offset"`   // negative reads from the end
	Length   int    `json:"length"`   //
	Decode   *bool  `json:"decode"`   // default true
	Encoding string `json:"encoding"` // auto | text | hex | base64
}

ReadArgs is the argument shape of http.read.

type ReadResult 

type ReadResult struct {
	Seq         int    `json:"seq"`
	Part        string `json:"part"`
	Section     string `json:"section"`
	Encoding    string `json:"encoding"`
	TotalLength int    `json:"totalLength"`
	Offset      int    `json:"offset"`
	Returned    int    `json:"returned"`
	Truncated   bool   `json:"truncated"`
	Decoded     string `json:"decoded,omitempty"`
	Text        string `json:"text"`

	// Redacted names the withheld header values that lie inside the returned
	// window, and only those: naming one the caller did not receive implies a
	// credential the returned bytes never carried.
	Redacted []string `json:"redacted,omitempty"`
}

ReadResult is the structured half of a read, and the value the JavaScript SDK returns: a script reads r.text and branches on r.truncated rather than parsing anything. An MCP client receives Render's text instead, chosen at that boundary. The two forms are never both sent — text may be 16 KB, and duplicating it as structuredContent would double the cost of every read.

func ReadRange 

func ReadRange(reqRaw, respRaw []byte, args ReadArgs, maskCredentials bool) (*ReadResult, error)

ReadRange extracts a byte window from a captured request or response.

Coordinates are bytes of the selected section after decoding. section "raw" is the whole dump with decoding forced off, so it stays byte-exact and matches the contract of the History Raw tab.

maskCredentials withholds sensitive header values. It is applied to the half that is actually returned, never to both, so the redaction notice cannot name a header the other half carried.

func (*ReadResult) Render 

func (r *ReadResult) Render() string

Render produces the text form: exactly one meta line, then the window, then any notes. The bytes begin immediately after the first '\n' and nothing is ever inserted ahead of them.

The fixed preamble is the point. A note above the window makes the offset of the first payload byte depend on whether that note fired, and the framing changes from '\n' to '\r\n' at the same boundary — so a reader taking the first line gets the meta line, a note and the request line welded together, with nothing in the output to say which shape arrived.

The meta line always names decoded when an encoding was unwrapped, because a decoded total disagrees with the Content-Length the client just read in the headers, and an unexplained mismatch reads as a bug. It names redacted for the same reason a note does: a withheld value the reader has not been told about is a credential it will report as absent.

type ResendArgs 

type ResendArgs struct {
	Seq                 int    `json:"seq" alias:"ref"`
	Edits               []Edit `json:"edits"`
	Scheme              string `json:"scheme"`
	Host                string `json:"host"`
	UpdateContentLength *bool  `json:"updateContentLength"`
	TimeoutMs           int    `json:"timeoutMs"`
	UseContext          *bool  `json:"useContext"`
}

ResendArgs is the argument shape of http.resend.

There is no followRedirects field, deliberately. A 302 to another host is a one-line scope bypass: the guard checked the host in these arguments, and a redirect would take the connection somewhere it never approved. A client follows a redirect by reading the Location from the result and issuing a second, separately guarded call.

type ResendDeps 

type ResendDeps struct {
	Send  SendDeps
	Store *proxy.Store

	// Contexts is the per-principal cookie jar; TokenID selects this caller's.
	// Both may be zero, in which case sends are stateless.
	Contexts *Contexts
	TokenID  string
}

ResendDeps is what a resend needs from the host process.

type SearchArgs 

type SearchArgs struct {
	Pattern string `json:"pattern"`
	Regex   bool   `json:"regex"`
	Seq     int    `json:"seq" alias:"ref"`
	Part    string `json:"part"` // req | resp | both

	// Corpus narrowing, mirroring history.list. These must stay in step with the
	// historyFilterProps schema fragment in internal/capreg, which both tools
	// share; a field advertised there but missing here would be rejected by the
	// handler's DisallowUnknownFields decoder for every client that follows the
	// documented contract.
	Host        string `json:"host"`
	Method      string `json:"method"`
	Status      string `json:"status"`
	Search      string `json:"search"`
	ContentType string `json:"contentType"`
	Exclude     string `json:"exclude"`
	ExtMode     string `json:"extMode"`
	ScopeOnly   bool   `json:"scopeOnly"`

	MaxRequests   int  `json:"maxRequests"`
	MaxMatches    int  `json:"maxMatches"`
	Context       int  `json:"context"`
	CaseSensitive bool `json:"caseSensitive"`
	Deep          bool `json:"deep"`
}

SearchArgs is the argument shape of http.search. The absence of Seq selects corpus mode; its presence selects single-response mode.

type SearchDeps 

type SearchDeps struct {
	Store *proxy.Store
	Scope *proxy.Scope

	// MaskCredentials masks sensitive header values before the pattern is applied,
	// so a match cannot be reported from inside a value the caller may not see. It
	// lives here rather than in SearchArgs because it is the host's decision.
	MaskCredentials bool
}

SearchDeps is what the search capability needs from the host.

type SendDeps 

type SendDeps struct {
	// ProxyAddr is Joro's own proxy listener, e.g. "127.0.0.1:8080".
	ProxyAddr string
	// CA verifies the MITM leaf the proxy presents. We know exactly who we are
	// talking to, so this is a real verification rather than InsecureSkipVerify.
	CA    *cert.CA
	Store *proxy.Store
	// Claims is optional and only needed where sends run concurrently: it stops
	// two batch workers correlating to the same history row. A single send passes
	// nil, for which claim is a no-op.
	Claims *claimSet
}

SendDeps is what a send needs from the host process.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.