README
¶
epochd
A Go tool that injects a fake wall-clock time into a running Linux process without restarting it, stopping it for more than a moment, or modifying its binary. Designed for Kubernetes end-to-end tests that need to control time on specific pods (e.g. testing certificate expiry, token refresh, cron scheduling).
Two clock modes are supported:
- Advancing — the target's clock is shifted by a fixed offset from real time. The clock
keeps advancing at the normal rate;
time.Now()returnsreal_now + offset. - Frozen — the target's clock is pinned to a specific instant. Every call to
clock_gettimereturns exactly that time, regardless of how much real time passes.
After a one-time injection, updating the time requires a single process_vm_writev
syscall — no ptrace stop, no signal, no coordination with the target.
Platform: Linux x86-64 only. Everything in the codebase is guarded with
//go:build linux.
How it works
The problem with patching time at the syscall level
The naive approach — intercept every clock_gettime syscall with ptrace — requires
stopping the process for each call. Go's runtime calls clock_gettime tens of thousands of
times per second. That's unusable.
The vDSO shortcut
Modern Linux maps a small shared library called the vDSO ("virtual dynamic shared
object") into every process's address space. Glibc's wall-clock functions don't issue a
syscall at all — they call symbols in the vDSO, which reads the kernel's timekeeping data
from a shared memory page directly. This is why clock_gettime is typically ~20 ns instead
of ~200 ns.
Because the vDSO is a normal mapped region, its code is writable under PTRACE_POKETEXT
(which can write read-only-but-executable pages, exactly as debuggers write breakpoints).
Three independent functions need patching, not one. clock_gettime, gettimeofday, and
time are three separate compiled functions at three different addresses in the vDSO — they
are not aliases of each other, even though they all report the same wall clock. Patching only
clock_gettime (an earlier version of this tool did) silently misses anything that calls the
other two — notably PostgreSQL's GetCurrentTimestamp() and glibc/bash's own wall-clock reads
($EPOCHREALTIME), both of which call gettimeofday. All three are patched today, each with
its own small stub, sharing one state struct so a single SetTime/Freeze call updates all of
them consistently.
The injection sequence
┌─────────────────────────────────────────────────────────────────────────┐
│ Target process address space │
│ │
│ [vdso] 0x7fff....a40 ← clock_gettime entry point ─┐ │
│ [vdso] 0x7fff....7a0 ← gettimeofday entry point ─┼─┐ │
│ [vdso] 0x7fff....a10 ← time entry point ─┼─┼─┐ │
│ │ Before: original vDSO code │ │ │ │
│ │ After: E9 xx xx xx xx ← JMP rel32, one per entry│ │ │ │
│ └───────────────────────────────────────────────────▼─▼─▼ │
│ │
│ [anon rwx page, allocated by the target itself via mmap] │
│ ├─ clock_gettime stub (offset 0) │
│ │ intercepts clk_id ∈ {CLOCK_REALTIME, CLOCK_REALTIME_COARSE}; │
│ │ real syscall → add offsetSec/offsetNsec → normalise tv_nsec │
│ ├─ gettimeofday stub (offset 141) │
│ │ same shape, tv_usec instead of tv_nsec (offsetNsec / 1000) │
│ ├─ time stub (offset 306) │
│ │ real clock_gettime(CLOCK_REALTIME) internally (not the time() │
│ │ syscall, which discards the real fractional second before this │
│ │ stub would ever see it) → same normalised add → return tv_sec │
│ └─ shared state struct (32 bytes, at StateOffset = 436) │
│ +0 int64 offsetSec │
│ +8 int64 offsetNsec │
│ +16 uint64 enabledMask (1=advancing, 3=frozen) │
│ +24 uint32 generation (bumped on each SetTime/Freeze) │
│ +28 uint32 _pad │
└─────────────────────────────────────────────────────────────────────────┘
All three stubs live in one allocated page and share one state struct, so a single
SetTime/Freeze/Advance call updates the fake time for all three functions at once —
clock_gettime, gettimeofday, and time in a target process always agree with each other
to the second.
Step by step (see pkg/inject/inject.go):
-
vDSO discovery — parse
/proc/<pid>/mapsfor[vdso], read those bytes via/proc/<pid>/mem, parse withdebug/elf, resolve theclock_gettime,gettimeofday, andtimesymbols → three absolute addresses (pkg/vdso.VDSOInfo). -
Remote mmap — while the target is ptrace-stopped, temporarily overwrite three bytes at the
clock_gettimeentry (used as a scratch location; it isn't touched again until step 4) with0F 05 CC(syscall; int3), set registers formmap(hint, 4096, PROT_RWX, MAP_PRIVATE|MAP_ANON|MAP_FIXED_NOREPLACE, -1, 0), resume, wait for theint3SIGTRAP, readRAXfor the new page address, restore original bytes and registers. The hint is chosen by scanning/proc/<pid>/mapsfor the nearest free page within ±2 GB of the vDSO entry (required forJMP rel32reach) — all three hooked functions live in the same few-KB vDSO mapping, so one page is in reach of all three. -
Write trampoline — copy the embedded binary payload (all three stubs + shared state) into the new page with
offsetSec/offsetNsecalready set. Useprocess_vm_writev(no ptrace stop needed for writeable pages). -
Patch vDSO — for each of the three functions, write
E9 <disp32>over its first 5 bytes usingPTRACE_POKETEXT(the only way to write a read-only mapped page), targeting that function's own stub offset in the trampoline page. -
Detach — the target resumes. Every subsequent
clock_gettime,gettimeofday, ortimecall now goes through the corresponding trampoline stub. -
Update time —
SetTime/Freezewrites a new state struct (32 bytes) into the trampoline page usingprocess_vm_writev. No ptrace needed. TheenabledMaskfield controls behaviour:MaskEnabled = 1(bit 0 set) means advancing mode — each stub adds(offsetSec, offsetNsec)to the real time.MaskFrozen = 3(bits 0+1 set) means freeze mode — the offsets encode an absolute timestamp; each stub ignores the real time and returns that value directly. The stubs read the state with plain loads; a concurrent update and a clock read can race — the worst outcome is one call returning a time between the old and new values, which is acceptable for testing.
Package layout
epochd/
├── cmd/
│ ├── agent/ # Node-level gRPC daemon (runs as DaemonSet)
│ ├── controller/ # Control-plane HTTP+JSON API
│ └── faketimectl/ # Manual CLI for local testing
│
├── pkg/
│ ├── vdso/ # vDSO discovery and ELF symbol resolution
│ ├── procmem/ # ptrace wrapper + process_vm_readv/writev
│ ├── trampoline/ # Assembled payload bytes + state struct helpers
│ ├── inject/ # Injection orchestration; public API
│ ├── faketime/ # Non-Kubernetes injection (standalone module: github.com/bkaznowski/epochd/pkg/faketime)
│ ├── agentpb/ # Generated gRPC types (agent.proto)
│ ├── agentclient/ # gRPC connection pool (controller → agents)
│ ├── k8sresolve/ # Container ID → PID resolution via /proc
│ ├── api/ # Shared HTTP request/response types
│ └── sdk/ # Go client library for e2e tests
│
├── proto/
│ └── agent/v1/ # Protobuf source for the agent gRPC API
│
├── test/
│ └── targets/
│ └── clockprinter/ # Sample target: prints time.Now() every second
│
├── deploy/ # Kubernetes manifests
│ ├── rbac.yaml
│ ├── daemonset.yaml
│ └── controller-deployment.yaml
│
├── Dockerfile.agent
└── Dockerfile.controller
Prerequisites
| Tool | Purpose |
|---|---|
| Go 1.26+ | Build and test |
| Linux x86-64 | Runtime (vDSO hook is arch-specific) |
| Docker | Run tests in a Linux container when developing on macOS/Windows |
| nasm | Re-assemble the trampoline (only if editing trampoline.asm) |
The pre-assembled trampoline.bin is committed to the repo; NASM is only needed if you
edit the assembly source.
Running the tests
In Docker (recommended — works on macOS and Windows)
docker run --rm \
--cap-add SYS_PTRACE \
--security-opt seccomp=unconfined \
-v "$(pwd):/workspace" -w /workspace \
golang:1.26-alpine \
go test ./... -count=1
SYS_PTRACE and seccomp=unconfined are required because the tests use ptrace. The
test binary spawns child processes with SysProcAttr{Ptrace: true} (the child calls
PTRACE_TRACEME) to work around Docker's Yama ptrace_scope=1 restriction, which blocks
PTRACE_ATTACH even with the capability granted.
On a real Linux host
go test ./... -count=1
Requires either root or CAP_SYS_PTRACE and ptrace_scope ≤ 1:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
Run a single package verbosely
# vDSO discovery
go test ./pkg/vdso/ -v
# ptrace primitives
go test ./pkg/procmem/ -v
# trampoline encoding/decoding
go test ./pkg/trampoline/ -v
# injection mechanics + round-trip integration
go test ./pkg/inject/ -v
Key tests by name
| Test | Package | What it checks |
|---|---|---|
TestLocateSelf |
pkg/vdso |
Resolves clock_gettime, gettimeofday, and time in the test process's own vDSO |
TestTracerBasic |
pkg/procmem |
Attach, ReadMem, WriteMem, PokeText, Detach |
TestStateOffsetRegression |
pkg/trampoline |
StateOffset constant matches actual binary layout |
TestEntryOffsetRegression |
pkg/trampoline |
Each hook stub's offset matches the assembled bytes at that address |
TestEncodeDecodeRoundTrip |
pkg/trampoline |
EncodeState / DecodeState are inverses |
TestRemoteMmap |
pkg/inject |
Remote mmap allocates a new rwx page in target |
TestInjectMechanics |
pkg/inject |
State struct fields, JMP displacement, setOffset, child survives Detach |
TestInjectObserved |
pkg/inject |
End-to-end: injected process prints timestamps ~24 h ahead |
TestInjectRoundTrip |
pkg/inject |
Full cycle: inject +24 h → verify → SetTime(now) → verify reset |
Building
# Build everything (Linux target)
CGO_ENABLED=0 GOOS=linux go build ./...
# Build individual binaries
CGO_ENABLED=0 GOOS=linux go build -o agent ./cmd/agent
CGO_ENABLED=0 GOOS=linux go build -o controller ./cmd/controller
CGO_ENABLED=0 GOOS=linux go build -o faketimectl ./cmd/faketimectl
Cross-compiling from macOS or Windows: add GOARCH=amd64 if your host is not x86-64.
The build is pure Go (CGO_ENABLED=0); no CGo. The trampoline is pre-assembled bytes in a []byte.
Docker images
# Agent (must be linux/amd64)
docker build -f Dockerfile.agent --platform linux/amd64 -t epochd-agent:dev .
# Controller
docker build -f Dockerfile.controller --platform linux/amd64 -t epochd-controller:dev .
Both images use scratch as the runtime base — no shell, no libc, ~10–50 MB total.
faketimectl — manual testing CLI
faketimectl has two groups of subcommands: controller subcommands that talk to a
running epochd controller over HTTP, and local injection subcommands that directly
ptrace a process on the current machine (Linux only, requires CAP_SYS_PTRACE).
Controller subcommands
Set EPOCHD_URL or pass --url on every command.
faketimectl create --namespace=NS --selector=SEL --time=RFC3339 [--ttl=DUR] [--freeze]
faketimectl list
faketimectl get <id>
faketimectl update <id> --time=RFC3339 [--freeze]
faketimectl advance <id> --by=DURATION
faketimectl delete <id>
faketimectl status <id>
faketimectl resolve --namespace=NS --selector=SEL
| Subcommand | Description |
|---|---|
create |
Inject a fake time into all matching pods. --freeze pins the clock at --time so it never advances. --ttl auto-deletes after the given duration. |
list |
List all active timeshifts. The TIME column reflects the live effective time the processes currently see. |
get |
Print details for one timeshift, including the current effective time. |
update |
Move the clock to a new absolute time. --freeze switches to/from freeze mode. |
advance |
Shift the clock forward (or backward) by a Go duration. Preserves the current mode. |
delete |
Reset all targeted processes to the real clock and remove the timeshift. |
status |
Query each node agent for the live trampoline state (generation counter, PID, last write). |
resolve |
Preview which pods and containers would be targeted without injecting anything. |
Example — advance a frozen clock by one day
export EPOCHD_URL=http://localhost:8080
# Create a timeshift frozen at 2030-01-01.
faketimectl create --namespace=default --selector=app=web \
--time=2030-01-01T00:00:00Z --freeze
# created timeshift a1b2c3d4...
# namespace: default
# time: 2030-01-01T00:00:00Z
# frozen: yes
# applied to: web-abc/main
# An hour later in real time, the processes still see 2030-01-01T00:00:00Z.
faketimectl get a1b2c3d4
# time: 2030-01-01T00:00:00Z
# Advance by one day.
faketimectl advance a1b2c3d4 --by=24h
# time: 2030-01-02T00:00:00Z ← frozen at the new point
# Switch to advancing mode and move time to 2030-06-01.
faketimectl update a1b2c3d4 --time=2030-06-01T00:00:00Z
# time: 2030-06-01T00:00:02Z ← now advancing; seconds tick forward
# Clean up.
faketimectl delete a1b2c3d4
Local injection subcommands
These directly ptrace a process — no controller required.
faketimectl inject --pid=PID --time=RFC3339 [--freeze]
faketimectl reset --pid=PID
| Flag | Description |
|---|---|
--pid |
Target process PID (required) |
--time |
Fake wall-clock time in RFC3339, e.g. 2030-01-01T00:00:00Z |
--freeze |
Pin the clock at --time so it never advances |
--reset |
Snap the target back to the real clock |
Example — inject a fake advancing time into a running process
# Terminal 1: run the sample target
./clockprinter
# 2026-06-21T10:00:00Z
# 2026-06-21T10:00:01Z
# Terminal 2: inject +4 years, advancing
sudo faketimectl inject --pid=$(pgrep clockprinter) --time=2030-01-01T00:00:00Z
# Terminal 1 now prints:
# 2030-01-01T00:00:00Z
# 2030-01-01T00:00:01Z
# Terminal 2: reset to real time
sudo faketimectl reset --pid=$(pgrep clockprinter)
Permissions: local injection calls PTRACE_ATTACH, which requires CAP_SYS_PTRACE
(or root) and ptrace_scope ≤ 1:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
The trampoline assembly
Source: pkg/trampoline/trampoline.asm
The payload is hand-written NASM with the .asm extension (not .s) to prevent Go's
plan9 assembler from touching it. It contains three independent entry points — one per
hooked vDSO function — followed by one shared state struct. inject.go JMP-patches each
vDSO function's real entry point to the corresponding label.
The clock_gettime stub (see the full file for gettimeofday_entry and time_entry,
which follow the same shape with their own calling convention):
; entry: rdi = clk_id, rsi = struct timespec *
clock_gettime_entry:
cmp edi, 0 ; CLOCK_REALTIME == 0; edi saves a REX prefix vs rdi
je .maybe_intercept
cmp edi, 5 ; CLOCK_REALTIME_COARSE == 5 -- same wall clock, coarser resolution
jne .real_syscall
.maybe_intercept:
lea r11, [rel state] ; RIP-relative — works wherever this page lands
test byte [r11 + 16], 1 ; bit 0: interception enabled?
jz .real_syscall
test byte [r11 + 16], 2 ; bit 1: freeze mode?
jnz .freeze
push rdi
push rsi
mov eax, 228 ; SYS_clock_gettime — shorter than mov rax (no REX prefix)
syscall ; get real time first, always
pop rsi
pop rdi
lea r11, [rel state] ; reload -- syscall clobbers r11
mov r8, [r11] ; offsetSec
mov r9, [r11 + 8] ; offsetNsec
add [rsi], r8 ; tp->tv_sec += offsetSec
add [rsi + 8], r9 ; tp->tv_nsec += offsetNsec
; normalise tv_nsec to [0, 1e9) — one step is enough because offsetNsec ∈ (-1e9, 1e9)
mov rax, [rsi + 8]
cmp rax, 1000000000
jl .check_negative
sub rax, 1000000000
mov [rsi + 8], rax
inc qword [rsi]
jmp .done
.check_negative:
cmp rax, 0
jge .done
add rax, 1000000000
mov [rsi + 8], rax
dec qword [rsi]
jmp .done
.freeze:
mov r8, [r11]
mov r9, [r11 + 8]
mov [rsi], r8
mov [rsi + 8], r9
jmp .done
.real_syscall:
push rdi
push rsi
mov eax, 228
syscall
pop rsi
pop rdi
.done:
xor eax, eax ; return 0; xor eax shorter than xor rax
ret
; ... gettimeofday_entry, time_entry (same shape, own ABI/precision) ...
state:
dq 0 ; offsetSec (+0)
dq 0 ; offsetNsec (+8)
dq 1 ; enabledMask (+16) — CLOCK_REALTIME/COARSE on by default
dd 0 ; generation (+24)
dd 0 ; _pad (+28)
Key design choices:
- No call back into original vDSO code — each stub issues a raw
syscallto get the real time. This avoids the need to save, relocate, and re-execute the original instruction bytes (the fiddly part of classic inline hooks). - Position-independent —
lea r11, [rel state]is RIP-relative. The payload can be placed anywhere in the address space. clock_gettimeinterceptsCLOCK_REALTIMEandCLOCK_REALTIME_COARSE(same wall clock, the latter is just a faster/coarser read of it).CLOCK_MONOTONIC,CLOCK_BOOTTIME, etc. deliberately pass through unchanged, so internal timeout/latency logic in the target process (e.g. PostgreSQL'sstatement_timeout, lock waits) keeps seeing real elapsed time.gettimeofdayandtimeare also intercepted, not justclock_gettime— they are independent functions at independent vDSO addresses, not aliases, and PostgreSQL'sGetCurrentTimestamp()callsgettimeofday, notclock_gettime.time_entrycalls the realclock_gettime(CLOCK_REALTIME)internally, not thetime()syscall —SYS_timeonly ever returns whole seconds, discarding the real fractional second before the trampoline would ever see it, which could maketime()disagree withclock_gettime/gettimeofdayabout the current second by up to 1s wheneveroffsetNsecwas nonzero. Calling the finer-grained syscall and applying the same carry-normalised add keeps all three in agreement to the second.- REX-prefix micro-optimisations —
mov eax(3 bytes) instead ofmov rax(4 bytes),cmp ediinstead ofcmp rdi,xor eax, eaxinstead ofxor rax, rax. Saves bytes in a payload where every byte shifts later stubs and thestate:label offset.
Assembling
nasm -f bin pkg/trampoline/trampoline.asm -o pkg/trampoline/trampoline.bin
After reassembling, verify the binary and update the offset constants if the code size changed:
# Disassemble to confirm layout
objdump -D -b binary -m i386:x86-64 pkg/trampoline/trampoline.bin
# StateOffset should equal (total binary size - 32); ClockGettimeEntryOffset /
# GettimeofdayEntryOffset / TimeEntryOffset are each stub's starting offset.
# TestStateOffsetRegression and TestEntryOffsetRegression fail loudly if they diverge.
wc -c pkg/trampoline/trampoline.bin # currently 468 bytes → StateOffset = 436
The constants in pkg/trampoline/trampoline.go (StateOffset = 436,
ClockGettimeEntryOffset = 0, GettimeofdayEntryOffset = 141, TimeEntryOffset = 306)
must match. TestStateOffsetRegression asserts StateOffset == len(Payload) - StateSize;
TestEntryOffsetRegression checks the assembled bytes at each *EntryOffset against the
expected first instruction of that stub. Both catch any drift at CI time.
Embedding
//go:embed trampoline.bin
var Payload []byte
The binary is embedded at build time. No runtime file I/O; no CGo.
pkg/inject internals
findNearbyGap — why a plain mmap hint isn't enough
JMP rel32 has a range of ±2 GB. Anonymous mmap(hint, ...) asks the kernel to try
placing the region near hint, but the kernel is free to ignore it — and will, when the
address space near the vDSO is saturated (common in Docker where ASLR places the vDSO
near the top of userspace, leaving little room above it).
findNearbyGap reads /proc/<pid>/maps while the tracee is ptrace-stopped (so the map is
stable) and finds the first unmapped page-aligned gap within ±2 GB of the vDSO entry. That
address is then passed to remoteMmap with MAP_FIXED_NOREPLACE, which either lands
exactly there or fails with EEXIST (no silent fall-back to a far address).
remoteMmap — making the target allocate its own page
The target calls mmap on its own behalf so the resulting page is in its own address
space. The sequence:
- Save the tracee's registers and 8 bytes at
clock_gettime. - Overwrite those bytes with
syscall; int3(0F 05 CC). - Set
RIP = clock_gettime,RAX = SYS_mmap, and argument registers. PTRACE_CONT→ wait forSIGTRAPfrom theint3.- Read the result from
RAX. - Restore the original bytes and registers.
The clock_gettime vDSO entry is used as the scratch location for this one-time mmap call
because it is already known to be executable and reachable — gettimeofday and time
aren't touched here. The final JMP rel32 patches happen later, once per hooked function
(clock_gettime, gettimeofday, time), after the trampoline has been written.
writeState / SetTime / Freeze — live updates with no ptrace
After injection, updating the fake time requires only process_vm_writev into the state
struct. The target does not need to be stopped. A generation counter is incremented on
every write for observability (visible in TestInjectMechanics).
The public Handle API:
// Advancing mode — trampoline adds (target - now) to real clock_gettime result.
func (h *Handle) SetTime(target time.Time) error
// Frozen mode — trampoline ignores real time and always returns exactly target.
func (h *Handle) Freeze(target time.Time) error
Internally, both call writeState(sec, nsec, mask) which writes a 32-byte state struct
via process_vm_writev. The mask controls mode: MaskEnabled = 1 for advancing,
MaskFrozen = 3 for frozen.
The four top-level constructors handle both modes and both ptrace paths:
func InjectAtTime(pid int, target time.Time) (*Handle, error) // Attach, advancing
func InjectFrozen(pid int, target time.Time) (*Handle, error) // Attach, frozen
func InjectAtTimeFollowChild(pid int, target time.Time) (*Handle, error) // FollowChild, advancing
func InjectFrozenFollowChild(pid int, target time.Time) (*Handle, error) // FollowChild, frozen
Tracer — OS-thread affinity for ptrace
Linux requires that every ptrace call for a given tracee come from the same OS thread
that issued PTRACE_ATTACH. Go's scheduler moves goroutines between OS threads freely,
which would break this. Tracer solves it by owning a single goroutine that calls
runtime.LockOSThread() at startup and never releases it. All ptrace operations are sent
as closures over a channel and executed on that pinned thread.
FollowChild vs Attach
FollowChild |
Attach |
|
|---|---|---|
| Mechanism | Child calls PTRACE_TRACEME; parent waits for SIGTRAP |
Parent calls PTRACE_ATTACH; sends SIGSTOP |
| Requires | Owning the child process | CAP_SYS_PTRACE + ptrace_scope ≤ 1 |
| Used in | Tests (Docker-compatible) | faketimectl, production agent |
Freeze mode
When a timeshift is created or updated with --freeze / freeze: true, the trampoline
enters frozen mode: the enabledMask field is set to MaskFrozen = 3 (bits 0 and 1),
and the offset fields store the absolute target timestamp rather than a delta. Every
clock_gettime (CLOCK_REALTIME or CLOCK_REALTIME_COARSE), gettimeofday, and time
call in the target returns exactly that timestamp, regardless of how much real time passes.
Freeze mode is useful for:
- Reproducing time-sensitive bugs that only trigger at a specific instant.
- Tests that need a deterministic, non-advancing clock (e.g. certificate expiry checks where the exact timestamp must match).
- Pausing time while performing setup, then advancing by discrete steps.
SDK — freeze mode
client := sdk.NewClient("http://localhost:8080")
// Create a frozen timeshift.
ts, err := client.CreateFrozenTimeshift(ctx, "default", "app=web",
time.Date(2030, 1, 1, 0, 0, 0, 0, time.UTC), 0)
// Switch an existing timeshift to frozen mode.
ts, err = client.FreezeTimeshift(ctx, ts.ID, time.Date(2030, 6, 1, 0, 0, 0, 0, time.UTC))
// Test helper — runs fn with the clock frozen, restores on return.
sdk.WithFrozenTime(t, "app=web", frozenAt, func() {
// time.Now() in target pods always returns frozenAt
})
pkg/faketime — freeze mode (local processes)
// Start a child process with its clock frozen at target.
handle, err := faketime.StartFrozen(cmd, target)
// Attach to an already-running process.
handle, err = faketime.AttachFrozen(pid, target)
// Freeze/unfreeze via the handle.
handle.Freeze(newTarget)
handle.SetTime(advancingTarget) // switches back to advancing mode
// Session-level freeze.
session := faketime.NewSession(target)
session.Freeze(target)
session.Start(cmd) // new processes joined after Freeze() are also frozen
// Session with fork/exec tracking — children and exec'd processes are
// automatically injected. Close() shuts the watcher down.
session := faketime.NewSession(target, faketime.WithTracking())
defer session.Reset()
defer session.Close()
session.Start(cmd) // parent + all descendants see fake time
Advance-by-duration
Instead of providing an absolute target time, you can advance (or rewind) the current fake time by a relative amount. This works in both advancing and frozen modes and preserves the current mode.
HTTP API
PATCH /timeshifts/{id}
Content-Type: application/json
{ "duration": "24h" }
Accepted by PATCH /timeshifts/{id} alongside the existing time field (the two are
mutually exclusive). Go duration strings are supported ("24h", "-1h30m", "72h").
The time field in GET /timeshifts/{id} responses always reflects the live effective
time — the actual timestamp the targeted processes currently see. For advancing timeshifts
this grows every second; for frozen timeshifts it is constant.
CLI
faketimectl advance <id> --by=24h # advance by 24 hours
faketimectl advance <id> --by=-1h # rewind by 1 hour
SDK
// Shift forward by one day.
ts, err := client.AdvanceTimeshift(ctx, ts.ID, 24*time.Hour)
pkg/faketime — advance (local processes)
// Advance a single-process handle by one day.
err = handle.Advance(24 * time.Hour)
// Advance all processes in a session.
err = session.Advance(24 * time.Hour)
Known limitations
-
x86-64 only. The trampoline is hand-assembled for
x86_64.aarch64would need a different payload and a different JMP patch strategy (AArch64Bhas only a 26-bit offset; you'd need an indirect branch via a scratch register instead). -
Wall clock only —
CLOCK_MONOTONICandCLOCK_BOOTTIMEare not intercepted.clock_gettime,gettimeofday, andtime(all wall-clock reads) are all patched, andclock_gettimeadditionally coversCLOCK_REALTIME_COARSE. Monotonic/boottime clock IDs deliberately pass through unchanged, so internal timeout/latency logic in the target process (e.g. PostgreSQL'sstatement_timeout, lock waits) keeps seeing real elapsed time — shifting them would be surprising (seeFUTURE.mdfor a design sketch if you want this as an opt-in). -
No teardown in v1. There is no
Uninstallthat restores the original vDSO bytes. CallingSetTime(time.Now())effectively resets the clock to real time, which is sufficient for test cleanup. The trampoline page and the JMP patch remain for the life of the target process. -
One trampoline per vDSO entry. Injecting a second time allocates a new page and re-patches the JMP (the old page leaks). This is acceptable for the testing use case.
-
No synchronisation on state reads. The trampoline reads the state struct with plain word loads. A
SetTimeracing with a concurrentclock_gettimemay observe a torn state and return a time between the old and new offsets. For test scenarios this is harmless. -
process_vm_writevon the state struct requires the kernel to allow cross-process writes. On kernels with strict LSM policies beyond Yama, this may fail even withCAP_SYS_PTRACE. The production agent (phase 7) can fall back to ptrace-stop + PokeText for the update path if needed.
Project status
| Phase | Deliverable | Status |
|---|---|---|
| 0 | Project scaffolding | ✅ |
| 1 | vDSO discovery (pkg/vdso) |
✅ |
| 2 | ptrace primitives (pkg/procmem) |
✅ |
| 3 | Remote mmap (remoteMmap in pkg/inject) |
✅ |
| 4 | Trampoline assembly (pkg/trampoline) |
✅ |
| 5 | Injection orchestration (pkg/inject) |
✅ |
| 6 | Local validation harness (faketimectl, integration test) |
✅ |
| 7 | Node agent gRPC service (cmd/agent) |
✅ |
| 8 | Control-plane HTTP+JSON API (cmd/controller) |
✅ |
| 9 | Kubernetes manifests (deploy/) |
✅ |
| 10 | e2e test SDK (pkg/sdk) |
✅ |
| 11 | Optional TTL | ✅ |
| 12 | Dockerfiles | ✅ |
| 13 | GitHub open-source setup | ✅ |
| 14 | GitHub Actions CI | ✅ |
| 15 | Local cluster e2e (kind + Makefile) | ✅ |
| 16 | List timeshifts (GET /timeshifts) |
✅ |
| 17 | Health endpoint + WithTimeT SDK helper |
✅ |
| 18 | Handle recovery (pod/agent restarts) | ✅ |
| 19 | Prometheus metrics | ✅ |
| 20 | Controller restart recovery (ConfigMap persistence) | ✅ |
| 21 | Graceful agent shutdown (SIGTERM drain) | ✅ |
| 22 | Dry-run / resolve mode (GET /resolve) |
✅ |
| 23 | Agent handle status RPC (GetStatus) |
✅ |
| 25 | Local process injection (pkg/faketime, non-Kubernetes) |
✅ |
| 26 | Conflict guard (reject overlapping timeshifts, 409 Conflict) |
✅ |
| 27 | faketimectl subcommand completeness (update, status) |
✅ |
| 28 | Structured logging (log/slog, JSON output, LOG_LEVEL) |
✅ |
| 29 | TTL expiry Kubernetes Events + timeshift_expired_total counter |
✅ |
| 30 | Lease-based leader election (coordination.k8s.io/Lease) |
🔲 |
| 31 | Validating webhook admission controller | 🔲 |
| 32 | pkg/faketime Attach path (CAP_SYS_PTRACE) |
🔲 |
| 33 | Integration test harness (make test-integration, kind) |
✅ |
| 34 | Freeze mode (pin clock at fixed instant, --freeze / MaskFrozen) |
✅ |
| 35 | Advance-by-duration (PATCH duration, advance --by, AdvanceTimeshift, Handle.Advance) |
✅ |
| 36 | Offset-based timeshift storage (live time in GET responses) |
✅ |
| 37 | proto freeze field on InjectRequest / SetTimeRequest |
✅ |
| 38 | pkg/faketime: Handle.EffectiveTime(), Handle.PID(), Handle.IsAlive(), Session.Close() |
✅ |
| 39 | pkg/faketime: StartWithTracking / ChildTracker — auto-inject into forked child processes via PTRACE_O_TRACEFORK |
✅ |
| 40 | pkg/faketime: exec-survivor injection — re-inject after exec() via PTRACE_O_TRACEEXEC so processes that self-exec (e.g. PEX bootstrap) or fork+exec retain fake time |
✅ |
| 42 | gettimeofday/time vDSO interception (previously only clock_gettime was patched, silently missing e.g. PostgreSQL's GetCurrentTimestamp()); CLOCK_REALTIME_COARSE support; time() precision fix (was up to 1s off vs. the other two) |
✅ |
See plan.md for the detailed specification of all phases.
See FUTURE.md for longer-horizon improvements (auth, multi-arch, Helm, HA).
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
agent
command
Command agent is the node-level gRPC daemon that performs vDSO clock injection on behalf of the epochd controller.
|
Command agent is the node-level gRPC daemon that performs vDSO clock injection on behalf of the epochd controller. |
|
controller
command
Package main implements the epochd controller: an HTTP+JSON service that resolves pods via the Kubernetes API and orchestrates clock injection via per-node gRPC agents.
|
Package main implements the epochd controller: an HTTP+JSON service that resolves pods via the Kubernetes API and orchestrates clock injection via per-node gRPC agents. |
|
faketimectl
command
Command faketimectl manages epochd fake-time operations.
|
Command faketimectl manages epochd fake-time operations. |
|
pkg
|
|
|
agentclient
Package agentclient provides a connection pool for the epochd node agent's gRPC API.
|
Package agentclient provides a connection pool for the epochd node agent's gRPC API. |
|
api
Package api holds the HTTP+JSON request and response types for the epochd controller's REST API.
|
Package api holds the HTTP+JSON request and response types for the epochd controller's REST API. |
|
inject
Package inject ties vDSO discovery, ptrace, and the trampoline payload into the public injection API.
|
Package inject ties vDSO discovery, ptrace, and the trampoline payload into the public injection API. |
|
k8sresolve
Package k8sresolve maps a Kubernetes container ID to the host PID of the container's init process by scanning /proc/*/cgroup.
|
Package k8sresolve maps a Kubernetes container ID to the host PID of the container's init process by scanning /proc/*/cgroup. |
|
log
Package log provides a structured JSON logger backed by log/slog.
|
Package log provides a structured JSON logger backed by log/slog. |
|
procmem
Package procmem provides ptrace-based process memory access primitives.
|
Package procmem provides ptrace-based process memory access primitives. |
|
sdk
Package sdk is the Go client library for the epochd controller API.
|
Package sdk is the Go client library for the epochd controller API. |
|
trampoline
Package trampoline holds the hand-assembled vDSO hook payload and the helpers needed to encode and decode the state struct that sits immediately after it.
|
Package trampoline holds the hand-assembled vDSO hook payload and the helpers needed to encode and decode the state struct that sits immediately after it. |
|
faketime
module
|
|
|
test
|
|
|
targets/clockprinter
command
Command clockprinter prints the current time once per second.
|
Command clockprinter prints the current time once per second. |