kang-go

README

kang-go

kang-go is a Go implementation of a Kang-style HTTP introspection endpoint: mount a /kang/snapshot route on any http.ServeMux and get a live JSON dump of your service's metadata, stats, and registered objects. It also includes optional runtime log-level control and per-request debug logging.

Install

go get github.com/chrisjhunter/kang-go

import "github.com/chrisjhunter/kang-go/kang"

Usage

registry := kang.NewSimpleRegistry()
stats := kang.NewSimpleStats()

kangServer := kang.NewServer(kang.Config{
	Service: kang.ServiceMetadata{
		Name:    "my-service",
		Ident:   "instance-1",
		Version: "1.0.0",
	},
	TypeLister:    registry,
	ObjectLister:  registry,
	ObjectGetter:  registry,
	StatsProvider: stats,
})

mux := http.NewServeMux()
kangServer.RegisterHandlers(mux, "/kang")

registry.Register("user", "alice", User{ID: "alice", Name: "Alice"})
stats.Increment("requests_total")

log.Fatal(http.ListenAndServe(":8080", mux))

curl http://localhost:8080/kang/snapshot | jq

See examples/basic, examples/combined_logging, and examples/full for complete, runnable programs covering increasing levels of integration (snapshot only, per-request debug logging, and both combined with live registry-backed handlers). Run any of them with go run ./examples/<name>.

Per-request debug logging

kang.DebugMiddleware lets a single request opt into debug-level logging — via the X-Debug: true header, a ?debug=true query param, or an X-Log-Level: debug|trace header — without changing the logger's global level:

debugMiddleware := kang.NewDebugMiddleware()
mux.HandleFunc("/api/users", debugMiddleware.Handler(handler).ServeHTTP)

// inside the handler:
logger := kang.GetDebugLogger(baseLogger, r)
logger.Debugf("only printed if debug is enabled for this request")

kang.LoggerManager additionally exposes runtime log-level control for registered *logrus.Loggers via POST/PUT {base}/log/level?logger=<name>, and surfaces registered loggers in the /kang/snapshot output as objects of type "logger".

Runtime stats and profiling

SimpleStats.GetStats() includes a runtime object (goroutine count, heap/stack usage, GC count and pause times) in every snapshot automatically, via kang.CollectRuntimeStats(). This is enough to compare instances at a glance — e.g. "which of these 5 containers has a goroutine leak" — without needing any extra tooling.

For root-causing why one instance looks different, mount Go's standard profiler alongside it:

kang.RegisterPprofHandlers(mux)

then drill in with the standard toolchain, pointed at that instance directly:

go tool pprof http://<host>:9080/debug/pprof/heap
go tool pprof http://<host>:9080/debug/pprof/profile?seconds=30   # CPU profile

RegisterPprofHandlers always mounts at /debug/pprof (not configurable — net/http/pprof resolves profile names by trimming that literal prefix internally, so a different mount path breaks profile lookups other than cmdline/profile/symbol/trace).

Security note

/kang/snapshot returns whatever objects, stats, and loggers you register, with no built-in authentication. The pprof endpoints above are even more sensitive — they can expose command-line arguments, memory contents, and source file paths, and CPU profiling pins CPU usage for the requested duration. Treat both as internal/operator-only — put them behind your own auth middleware, an internal-only listener, or a firewall rule before exposing them beyond localhost.

Testing

go test ./...

Multi-instance test cluster (Docker Compose)

docker-compose.yml spins up N instances of examples/basic on a private bridge network, useful for exercising service discovery and load spreading across replicas. Docker's embedded DNS resolves the kang service name to every replica's IP, so a resolver-aware client can enumerate all instances the same way it would against a real cluster.

docker compose up -d --build --scale kang=5
docker compose --profile test run --rm tester
docker compose down

The tester service (scripts/test-cluster.sh) resolves kang via dig, curls each discovered IP directly to confirm it's a distinct instance (each reports its own hostname as service.ident in the snapshot), then fires a burst of concurrent requests at the round-robin kang DNS name and reports how many of the replicas answered. Override EXPECTED_NODES, CONCURRENCY, SERVICE_NAME, or PORT via the tester service's environment: block if you change the replica count or scale differently.

scripts/test-cluster.sh must run inside a container on the compose network — it relies on Docker's embedded DNS (127.0.0.11), which your host's resolver has no route to. To run the same checks directly from your terminal instead, use scripts/test-cluster-host.sh, which discovers replica IPs via docker inspect/docker compose ps rather than DNS:

./scripts/test-cluster-host.sh

It accepts the same SERVICE_NAME, PORT, EXPECTED_NODES, and CONCURRENCY env var overrides, plus NETWORK_NAME (defaults to kang-go_kangnet, i.e. <project>_<network> — check docker network ls if you changed the compose project name).

Both scripts also accept -v/--verbose (print each instance's full snapshot JSON and per-request status/timing) and -o/--output FILE (additionally save all output to FILE):

./scripts/test-cluster-host.sh -v -o results.log

For the containerized version, docker compose run <service> [args...] replaces the service's configured command: rather than appending to it, so passing just -v would run bash -v (the entrypoint) instead of the test script. Give the full command instead:

docker compose --profile test run --rm tester /scripts/test-cluster.sh -v

Don't use -o with the containerized tester — /scripts is mounted read-only, so tee fails to open the file (silently, since it still echoes to stdout and the script exits 0 regardless). To save output from the containerized run, redirect from the host shell instead:

docker compose --profile test run --rm tester /scripts/test-cluster.sh -v > results.log 2>&1

See docs/docker-commands.md for a fuller command reference, docs/troubleshooting.md for issues encountered running this cluster locally, and docs/query-examples.md for CPU/perf comparison queries across containers plus a gnuplot-based charting script.

Testing kang instances outside this repo

scripts/test-cluster-hosts.sh runs the same identity/concurrency check against an arbitrary list of hosts, for testing real kang-instrumented services wherever they run (Kubernetes, Ansible- managed hosts, bare VMs) instead of just this repo's Docker cluster:

./scripts/test-cluster-hosts.sh --hosts 10.0.0.1,10.0.0.2,10.0.0.3
./scripts/test-cluster-hosts.sh --hostfile hosts.txt
kubectl get pods -l app=kang -o jsonpath='{.items[*].status.podIP}' | tr ' ' '\n' | ./scripts/test-cluster-hosts.sh --hostfile -

See docs/other-environments.md for the Kubernetes and Ansible discovery pipelines feeding this script.

cmd/kang: fetching and merging multiple instances into one snapshot

The scripts above check instances one at a time. cmd/kang instead fetches every host's /kang/snapshot and merges them into a single JSON document on stdout, so you can point standard tools (jq, etc.) at your whole fleet in one query instead of writing a loop yourself:

go build -o kang ./cmd/kang

./kang --hosts 10.0.0.1,10.0.0.2,10.0.0.3 | jq '.connection'
./kang --hostfile hosts.txt | jq '.service'
docker network inspect kang-go_kangnet --format '{{range .Containers}}{{.IPv4Address}}{{"\n"}}{{end}}' \
  | sed 's#/.*##' | grep -v '^$' | ./kang --hostfile - | jq '.stats'

Same --hosts/--hostfile input conventions as scripts/test-cluster-hosts.sh (--hostfile - reads from stdin). Merge semantics follow the original Kang client's approach: service and stats are keyed by a synthesized per-instance name (name.component.ident), and every other type's objects are grouped by id as a list of origin-tagged entries rather than overwritten on collision — independently-run instances commonly reuse the same object ids (every examples/basic instance seeds conn_1, conn_2, ...), so a naive last-write-wins merge would silently lose every host's data but one.

A host that fails to respond produces a warning on stderr but doesn't block the merge of the hosts that did respond; the command only exits non-zero if every host failed.

--format table: iostat-style live output

For a quick side-by-side comparison instead of a JSON document, --format table prints one aligned row per instance (its goroutines/heap/GC from kang.RuntimeStats, - if the instance doesn't use kang.SimpleStats), and --interval/--count repeat it the same way iostat/vmstat do — one header, then a growing series of timestamped blocks:

./kang --hostfile hosts.txt --format table                          # one-shot table
./kang --hostfile hosts.txt --format table --interval 2s            # like: iostat 2
./kang --hostfile hosts.txt --format table --interval 2s --count 5  # like: iostat 2 5

TIME      HOST             INSTANCE                                  GOROUTINES     HEAP_KB    GC_COUNT
20:01:04  192.168.32.3     example_service.api_server.0d95835bc5f4            4        1221           0
20:01:06  192.168.32.3     example_service.api_server.0d95835bc5f4            4        1269           0

Columns are fixed-width, not computed from each block's data — an iostat-style tool needs stable column boundaries as numbers grow between refreshes, and each --interval tick is a separate write with no memory of the previous one's column widths. With no --count, it runs until interrupted (Ctrl-C), same as iostat/top.

License

Mozilla Public License 2.0 — see LICENSE.