elastic-trace2

README

trace2

trace2 converts the NDJSON HTTP trace logs written by Elastic Agent / Beats inputs (httplog.LoggingRoundTripper) into formats made for viewing: HAR 1.2 for HAR viewers such as mitmproxy, or OTel traces for trace viewers such as Jaeger.

It is built for local troubleshooting of recovered trace logs rather than live capture: it reads rotated .ndjson files, pairs request and response events by transaction.id (even across file rotation), groups them by producing process, and presents as much of the source data as possible — bodies included, decompressed where a plain decoding exists.

Installation

With Go installed, fetch and install the CLI in one step:

go install github.com/chrisberkhout/elastic-trace2/cmd/trace2@latest

This puts the trace2 binary in $(go env GOPATH)/bin (usually ~/go/bin); make sure that directory is on your PATH. Or build from a checkout of this repository:

go build -o ./bin/trace2 ./cmd/trace2

Quick start

Convert a directory of trace logs to HAR and open it in the mitmproxy web UI:

trace2 har ./traces -o out/        # one .har file per input policy
mitmweb -r out/o365.audit-17734b7a-26c2-4cef-8cdd-cc9cf4bf6021.har

Or send the same logs to a local trace viewer as OTel spans:

docker run --rm -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one
trace2 otel ./traces --endpoint http://localhost:4318/v1/traces
# then browse http://localhost:16686

Spans keep the original timestamps from the logs, and trace viewers search a recent time window by default — Jaeger's Lookback is "Last Hour". If Find Traces returns nothing, set Lookback to "Custom Time Range" and pick a range covering when the traces were recorded.

All usage details are documented by the tool itself — start with trace2 help, then trace2 help har and trace2 help otel. The README does not duplicate the flag reference.

Reading the HAR output in mitmproxy

Getting mitmproxy

Install it per the mitmproxy installation guide (standalone binaries, brew install --cask mitmproxy on macOS, or uv tool install mitmproxy from PyPI). Alternatively, skip installing and run the web UI from the official docker image — see below.

Web GUI (mitmweb)

mitmweb -r out/example.har

This starts the browser UI and prints a http://127.0.0.1:8081/?token=... URL (it normally opens your browser by itself; pass --no-web-open-browser to suppress that). You get a chronological flow list: click an exchange to inspect it, and switch between the Request, Response, and Connection tabs. The search box at the top takes the same filter expressions as the terminal UI, for example ~u messageTraces (URL substring), ~d login.microsoftonline.com (domain), ~c 400 (response status code), or combinations like ~c 401 & ~d graph.microsoft.com.

Without a local install, the same UI via docker:

docker run --rm -it -v "$PWD/out:/out:ro" -p 127.0.0.1:8081:8081 \
  mitmproxy/mitmproxy \
  mitmweb --web-host 0.0.0.0 --no-web-open-browser -r /out/example.har

Then open the printed ?token=... URL, replacing 0.0.0.0 with 127.0.0.1. Stop it with Ctrl+C.

Terminal UI (mitmproxy)

mitmproxy -r out/example.har gives the same flow list in the terminal; move with the arrow keys, press Enter to open an exchange, Tab to switch between the request and response panes, and q to go back. Press f to enter a filter expression (same syntax as above).

In both UIs, response bodies are decompressed and embedded by trace2, so the response pane shows API payloads directly. Response-only exchanges appear under the placeholder URL https://trace2.invalid/request-missing.

mitmproxy ignores HAR comment fields and pages, so the provenance that trace2 records (source file and line, run key, transaction id, truncation notes) is not visible there. Read it from the file instead, for example:

# provenance of every failed exchange
jq -r '.log.entries[] | select(.response.status >= 400) | .comment' out.har

# which runs (producing process lifetimes) are in the file
jq -r '.log.pages[].title' out.har

Grouping model

The tool organizes recovered exchanges on four levels, mapped consistently to both output formats:

Level	Derived from	HAR	OTel
1. Input policy	trace filename (input instance id)	separate output files with -o <dir>	resource (service.name, trace2.input_instance_id)
2. Input run	transaction.id prefix	page	synthesized trace with a root span covering the run
3. Request sequence	source trace.id (newer logs only)	entry comments	intermediate span between run root and requests
4. Request	transaction.id	entry	CLIENT span

An input run is one process lifetime of the producing input; a run can span many rotated files and one file can contain several runs, so grouping always comes from file contents, never filenames.

OTel trace and span IDs are synthesized deterministically from the policy, run key, and transaction id — repeated conversions produce identical IDs. Source trace.id/span.id values are kept as trace2.source.* attributes but deliberately never used as output IDs: they mark single poll cycles, and using them would fragment the broad one-trace-per-run view this tool is built around.

What to expect in the data

• Bodies are included by default: gzip/deflate-compressed bodies are decompressed and marked, valid UTF-8 is embedded as text, and binary content is base64-encoded. Bodies truncated by the producer are included and marked. --omit-bodies drops content but keeps sizes, MIME types, and truncation markers.
• Not invented: the producer does not log HTTP version, status text, or wire-level timing phases, so those HAR fields stay empty or -1, and the whole request-to-response duration is reported as HAR wait time.
• Incomplete pairs are kept: a request without a response gets a synthetic status-0 response; a response without a request gets a synthetic GET https://trace2.invalid/request-missing so HAR importers can parse the file. Both are marked in comments and trace2.* attributes.
• Privacy is out of scope by design: output contains everything the source logs contain, including auth headers and tokens. Keep it local or send it only to collectors you trust.

References

• HAR 1.2 spec — the authoritative HAR format description
• W3C HAR draft
• mitmproxy documentation — including filter expression syntax
• OTLP specification — the OTLP/HTTP JSON encoding the otel command emits
• OTel HTTP span conventions — the attribute names used on client spans (span names deviate deliberately: <method> <path> for readability)
• Beats httplog source — the producer of the input format

Development

make check   # go vet + go test
make build   # builds ./bin/trace2

The fixture tests use ./traces/, which is intentionally untracked and must stay local.

License

Apache-2.0