README
¶
hardiff
Compare two HTTP Archive captures and see what changed.
No dependencies.
Why
You have two HARs of the same flow. It worked yesterday and not today, or the browser gets through and your client does not, or one capture is signed in and you cannot see where that shows up. The answer is in there, in a field, somewhere among a few hundred requests.
Doing it by hand means finding the same request in both files and reading the headers side by side. Doing it with diff does not work, because the two captures never line up: requests arrive in a different order, ids and timestamps differ on every line, and one capture always has entries the other does not.
$ hardiff before.har after.har
only in before.har
GET https://example.test/only-in-a
only in after.har
GET https://example.test/only-in-b
changed
GET https://example.test/api/rules
query _ 111 -> 999
POST https://example.test/graphql
header x-cmd reserve -> (absent)
body reserveInput.tickets.0.endSeat (absent) -> "7"
body reserveInput.tickets.0.priceZone "27" -> "P5"
body reserveInput.tickets.0.secname.0 "P27" -> (absent)
body reserveInput.tickets.0.startSeat (absent) -> "8"
status 200 -> 403
2 changed, 1 only in before.har, 1 only in after.har, 0 identical
Install
go install github.com/Mootjelh/hardiff/cmd/hardiff@latest
The matching is the whole trick
Requests are matched on method and URL path, with the query string compared as a field rather than used for matching.
That one decision is what makes the output readable. Match on the full URL, which is the obvious thing to do, and a cache buster like ?_=1724876400 gives the same request two different keys. It then shows up as one request removed and one added, on every single call, and the difference you were looking for is buried under a hundred lines of noise. Above, _=111 against _=999 is reported as a query parameter that moved, which is what it is.
Several calls to the same endpoint are paired in the order they were recorded, first with first. A page that hits one endpoint repeatedly does so in a stable order, so this is right far more often than a similarity score over bodies, which would also make the output shuffle for reasons you cannot see.
JSON bodies are compared as fields
Two captures of one endpoint usually differ in a handful of keys inside several kilobytes of JSON. Comparing the bodies as text answers "they differ", which you knew. So a body that parses as JSON on both sides is flattened and compared by path:
body reserveInput.tickets.0.secname.0 "P27" -> (absent)
Anything that is not JSON is compared as text.
Absent is not empty
A header sent with an empty value and a header not sent at all are different requests, and a lot of the time that difference is the answer. So presence and value are tracked separately, and a missing field prints as (absent) rather than as a blank column. Under -json it is null against "".
A real capture is mostly other people's traffic
The first time this ran on two real 5 MB captures of one checkout flow it found what it was built to find, and then printed 105 more requests:
changed requests: 106 total fields: 890
per host
checkout.example.com 50
www.paypal.com 15
3ds.example.com 9
apps.rokt.com 7
cdn.cookielaw.org 6
analytics.google.com 6
Payment iframes, consent banners, analytics and upsell widgets were more than a third of it. They differ on every field that carries a session id and none of them is ever the question. So narrow to the site you are looking at:
hardiff -host checkout.example.com a.har b.har # 106 changed becomes 50
A host matches exactly, or as a parent domain on a dot boundary, so
example.com also takes checkout.example.com and 3ds.example.com but not
notexample.com. The flag is repeatable.
Nothing is ignored by default
The tempting thing is to ship a list of headers that "always differ", and it is worth resisting. Cookies, tokens and user agents differ between two captures exactly when the difference is the point. One capture being signed in and the other not is precisely the sort of thing people fail to notice for a day.
So the defaults show everything, and you drop what you have already accounted for:
hardiff -ignore cookie -ignore user-agent a.har b.har
The one exception is HTTP/2 pseudo-headers. :path and :authority are
dropped, because they repeat the method and URL that every difference is already
printed under. That is the test for whether hiding something is acceptable: a
cookie can tell you one capture was signed in, a pseudo-header can only repeat
the address.
Response headers and response bodies are off by default. Those carry a date, a request id and a cache state on almost every response, and the question is usually what was sent. -response-headers and -response-bodies turn them on.
Sharing the result
A capture is full of credentials, and a diff of two captures is mostly credentials. -redact keeps the finding and drops the value, so the output can go in an issue:
header authorization (36 bytes, redacted) -> (29 bytes, redacted)
Scripting
hardiff -json a.har b.har | jq '.changed[].fields[] | select(.section == "body")'
Exits 0 when the captures agree, 1 when they differ, and 2 on an error, the same way diff(1) does.
As a library
a, err := hardiff.Load("before.har")
b, err := hardiff.Load("after.har")
res := hardiff.Diff(a, b, hardiff.Options{})
fmt.Print(res.Render("before.har", "after.har"))
Match is exported on its own, for pairing up two captures and then asking your own questions about the pairs.
What it doesn't do
- No timing or waterfall comparison. Several tools already do that, and it is a different question from "what did I send differently".
- No HAR editing or replay. This reads and reports.
- No fuzzy matching of request bodies. Requests are matched by endpoint, not by content.
A note on test fixtures
There are no .har files in this repository, and .gitignore refuses them. A capture is somebody's session: it carries their cookies, their tokens and often their name and address. The fixtures are built in code instead.
License
MIT
Documentation
¶
Overview ¶
Package hardiff compares two HTTP Archive captures and reports what changed.
A HAR is a recording of what a browser or a client actually sent, and the usual reason to hold two of them is a question of the form "this worked yesterday and does not today", or "the browser gets through and my client does not". Answering that means finding the same request in both captures and looking at the fields, which is tedious by hand and is what this does.
The part that makes it work is the matching. Two captures of one flow never line up by position: requests arrive in a different order, cache busters and request ids differ, and one capture usually has entries the other does not. See Match.
Nothing here fetches anything or writes to a capture. It reads JSON and returns a description.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Key ¶
Key identifies a request across two captures: the method, and the URL with its query string removed.
Dropping the query is the whole reason this works. Matching on the full URL looks obviously right and is what makes a naive diff useless: a cache buster, a request id or a session token in the query means the same request appears under two different keys, so every one of them is reported as a request removed and an unrelated one added, and the actual difference is buried. The query is compared as a field instead, which is where a change in it belongs.
A URL that will not parse is used as it is. It is somebody's capture, not necessarily somebody's valid URL, and refusing to match is worse than matching on a slightly odd key.
Types ¶
type Content ¶
type Content struct {
Size int64 `json:"size"`
MimeType string `json:"mimeType"`
Text string `json:"text"`
Encoding string `json:"encoding"`
}
Content is a response body. Text may be base64 when Encoding says so, and may be absent entirely: plenty of exporters drop response bodies, and an absent body is not an empty one.
type Entry ¶
type Entry struct {
StartedDateTime string `json:"startedDateTime"`
Request Request `json:"request"`
Response Response `json:"response"`
}
Entry is one request and its response.
type Field ¶
Field is one difference between two matched requests.
InA and InB record presence, which is not the same question as the value. A header sent with an empty value and a header not sent at all are different requests, and telling them apart is most of what this is for.
type HAR ¶
type HAR struct {
Log struct {
Version string `json:"version"`
Entries []Entry `json:"entries"`
} `json:"log"`
}
HAR is the subset of the HAR 1.2 format this package reads.
Deliberately not the whole spec. Timings, cache entries, page groupings and the creator block say nothing about what was sent, and parsing fields nobody compares is a way to fail on a capture that would otherwise have worked. Unknown fields are ignored rather than rejected, because exporters disagree about the optional half of the format.
func Parse ¶
Parse reads a capture from a reader.
A file that is JSON but is not a HAR is rejected by name rather than coming back as an archive with no entries. An empty result is the same shape as a capture that recorded nothing, and those two need opposite responses from whoever is reading the output.
type NameValue ¶
NameValue is a header, a query parameter or a cookie. All three are the same shape in a HAR.
type Options ¶
type Options struct {
// IgnoreHeaders names headers to leave out, case-insensitively. Nothing is
// ignored by default.
//
// The temptation is to ship a list of headers that "always differ", and it
// is worth resisting. Cookies, tokens and user agents differ between two
// captures precisely when the difference is the answer: one capture being
// signed in and the other not is exactly the sort of thing people spend a
// day failing to notice. Hiding it by default would be the tool making that
// mistake on the reader's behalf.
IgnoreHeaders []string
// ResponseHeaders compares the response headers too. Off by default,
// because they carry a date, a request id and a cache state on almost every
// response, and a diff of two captures is usually a question about what was
// sent.
ResponseHeaders bool
// ResponseBodies compares response bodies. Off by default: they are large,
// often absent from an export, and rarely the question.
ResponseBodies bool
// Redact reports which fields differ without printing either value, so the
// result can be pasted somewhere. Captures are full of credentials and a
// diff of two of them is mostly credentials.
Redact bool
// MaxValue truncates a printed value. 0 means 120 characters. Bodies and
// tokens run to kilobytes and a diff nobody can read is not an answer.
MaxValue int
// Hosts narrows the comparison to requests made to these hosts. Empty means
// every host.
//
// A real capture is mostly other people's traffic. Measured on two 5 MB
// recordings of one checkout flow: 106 requests differed, and payment
// iframes, consent banners, analytics and upsell widgets were more than a
// third of them. They differ on every field carrying a session id and none
// of them is ever the question.
//
// A host matches exactly, or as a parent domain on a dot boundary, so
// "example.com" also takes "checkout.example.com" but not "notexample.com".
Hosts []string
}
Options controls what Diff compares and how much of a value it prints. The zero value is a sensible default.
type Pair ¶
type Pair struct {
Key string
A *Entry
B *Entry
// AIndex and BIndex are positions in the original entry lists, so a
// finding can be pointed back at a line in the capture. They are -1 on the
// side where the request is absent.
AIndex int
BIndex int
}
Pair is one request found in both captures, or in only one of them. A nil A means it appears only in B, and the other way round.
func Match ¶
Match lines up the entries of two captures.
Requests sharing a key are paired in the order they were recorded, first with first. That is a guess, and it is the right one: a page that calls the same endpoint several times does so in a stable order, and the alternative, some similarity score over bodies, would pair requests differently depending on their contents and make the output move around for reasons the reader cannot see.
The result is ordered by where the request appears in A, and requests only in B come last, so reading it top to bottom follows the first capture.
type PostData ¶
type PostData struct {
MimeType string `json:"mimeType"`
Text string `json:"text"`
Params []NameValue `json:"params"`
}
PostData is a request body.
type Request ¶
type Request struct {
Method string `json:"method"`
URL string `json:"url"`
HTTPVersion string `json:"httpVersion"`
Headers []NameValue `json:"headers"`
QueryString []NameValue `json:"queryString"`
Cookies []NameValue `json:"cookies"`
PostData *PostData `json:"postData"`
}
Request is what went out.
type Response ¶
type Response struct {
Status int `json:"status"`
StatusText string `json:"statusText"`
HTTPVersion string `json:"httpVersion"`
Headers []NameValue `json:"headers"`
Content *Content `json:"content"`
}
Response is what came back.
type Result ¶
type Result struct {
OnlyInA []Pair
OnlyInB []Pair
Changed []Changed
// Same counts the matched requests with no differences, so a run that finds
// nothing can say how much it looked at. "No differences" and "nothing was
// compared" print the same way otherwise, and they are opposite answers.
Same int
}
Result is what Diff found.
func (Result) Render ¶
Render writes the result as text, with nameA and nameB used to label the two sides.
Requests are grouped under the three questions someone actually has: what did the first capture do that the second did not, what did the second do that the first did not, and where they did the same thing, what was different about it.
type Section ¶
type Section string
Section says which part of a request or response a difference is in.
const ( SectionQuery Section = "query" SectionHeader Section = "header" SectionBody Section = "body" SectionStatus Section = "status" SectionRespHead Section = "response header" SectionRespBody Section = "response body" SectionHTTPVer Section = "http version" SectionMimeType Section = "content type" SectionBodyBytes Section = "body size" )
Source Files
Directories