README
¶
go-ktls
Kernel TLS offload for a Go HTTPS listener.
The package hands TLS 1.3 and TLS 1.2 (AES-GCM) record encryption to the Linux
kernel (kTLS). The handshake stays in userspace via crypto/tls; afterwards the
negotiated keys go to the kernel with setsockopt(SOL_TLS) and reads and writes
no longer pass through the userspace TLS stack. That makes sendfile(2) usable
straight through the TLS socket, and on NICs with inline TLS offload
(ConnectX-6 Dx / ConnectX-7) file-backed plaintext pages can be DMAed directly
from the page cache to the NIC, which encrypts them on the wire. This avoids
per-byte encryption on the CPU and avoids writing the resulting ciphertext back
into host DRAM.
It is a fork of Northernside/ktls (MIT) with substantial correctness and architecture changes; see doc.go for the full rationale and the design of each path.
Install
go get github.com/waipu-oss/go-ktls
Requires Go 1.25. Builds everywhere; the offload itself is Linux-only.
Usage
ln, _ := net.Listen("tcp", ":443")
srv.Serve(ktls.NewListener(ln, tlsConfig,
ktls.WithObserver(func(reason string, remoteAddr net.Addr, state tls.ConnectionState, err error) {
// reason is "offloaded" on success, otherwise a fallback bucket
}),
))
Any connection that cannot be offloaded falls back to userspace TLS (a plain
*tls.Conn) transparently, so enabling kTLS is safe even where the kernel does
not support it. The observer reports which bucket each connection landed in.
Requirements
Targets kernel 6.14 or newer, which covers every kTLS feature used, so the
offload path carries no kernel-version fallbacks. On other platforms, or without
the tls module (modprobe tls), everything falls back to userspace TLS;
Available() reports kernel support.
NIC inline offload (TLS_HW) additionally needs CONFIG_TLS_DEVICE=y, a
supported NIC and firmware, and ethtool -K <if> tls-hw-tx-offload on. Mainline
Linux gates it on TLS 1.2 AES-GCM; TLS 1.3 always runs as software kTLS
(TLS_SW), which still gives the sendfile and copy savings (see Kernel
work for the series that would lift this). Verify with
/proc/net/tls_stat (TlsTxSw vs TlsTxDevice) and ethtool -S <if> | grep -i tls.
Kernel work
kTLS series on netdev that affect this package. Status as of 2026-07-27, taken
from the netdevbpf project on patchwork.kernel.org; follow the threads for the
current state.
| Series | Why it matters here | Status |
|---|---|---|
| tls: Add TLS 1.3 hardware offload support (v15, 9 patches) | Would lift the TLS 1.2 restriction on NIC inline offload, so TLS 1.3 sessions could run as TLS_HW rather than TLS_SW, device KeyUpdate included. |
changes requested; v14 superseded |
| tls: device: push pending open record on splice EOF | Consumers relying on ReadFrom should know about this one: without it the kernel does not push a pending open record when the splice ends. Go issues sendfile(2) unbounded, so it is easy to reach. |
in mainline since v7.2-rc5, commit eaa39f9f8ac8 |
| net/mlx5e: fix NULL derefs when RX queue mapping outlives channel reconfig (2 patches) | Only with mlx5 inline offload: NULL dereferences when an RX queue mapping survives a channel reconfiguration. | awaiting upstream |
| net/mlx5e: ktls: guard RX resync against missing TLS context | Same path as above: RX resync running with no TLS context attached. | awaiting upstream |
Limitations
These connections are not broken. They are served through userspace TLS exactly
as without this package. The result label says which bucket a connection hit.
| Limitation | result |
|---|---|
| TLS 1.1 and below never offloaded. TLS 1.2 offloaded for the six AES-GCM suites only (CBC and ChaCha20 fall back). TLS 1.3 offloads all three suites; ChaCha20 as software kTLS only. | tls-version / cipher |
HTTP/2 (ALPN h2): net/http enables h2 only for concrete *tls.Conn, so those stay in userspace. |
alpn |
Client certificates (ClientAuth != NoClientCert) are not offloaded. |
client-auth |
GetConfigForClient returning a non-nil config bypasses the key capture. |
secrets |
TLS 1.3 without a session ticket: the kernel TX sequence is derived from the NewSessionTicket record, so a connection that gets none is not offloaded. Applies to SessionTicketsDisabled and to clients that do not offer psk_dhe_ke (browsers and curl do; a Go client only with a ClientSessionCache). |
framing |
| Peer-initiated TLS 1.3 key updates are rejected (CPU-amplification guard; the rekey machinery exists behind a flag). | conn closed |
| TLS 1.2 renegotiation and 0-RTT / early data are not supported. | conn closed / n/a |
| Half-offload failure (TX up, later step fails): the socket cannot return to userspace and is closed. | conn-unusable |
The offloaded connection is not a *tls.Conn. Use ConnectionState() (net/http
populates Request.TLS through it) and NetConn() (unwrap to *net.TCPConn).
Performance
Steady-state data transfer adds no userspace overhead: offloaded connections
read and write the raw *net.TCPConn, and ReadFrom delegates to it for
sendfile. The per-connection offload setup runs once at handshake time.
In production
Live video edge (DASH, HLS) on AMD EPYC Zen2, ConnectX-6 Dx 2x100G, kernel 7.2 netdev tree with the TLS 1.3 device offload series applied, TX on the device path (RX was enabled during v14 testing):
- 130 Gbit/s egress, ~99.5% of eligible TX traffic HW-offloaded
- ~19k concurrent streams, 85% TLS 1.3, 15% TLS 1.2
- DRAM bytes per egress byte: 5.4x without kTLS, 2.2x with
On this workload the main win is not the AES offload but avoiding the ciphertext write back into DRAM. Mainline currently only exposes device offload for the TLS 1.2 AES-GCM path; TLS 1.3 needs this series.
What it does not fix
Removing the encrypt-and-copy path reduces memory traffic and CPU/cache pressure. It does not shrink the per-connection kernel state, so the ceiling set by L3 capacity versus connection count is unchanged. Two separate effects: per-byte memory traffic, which kTLS moves, and per-connection state locality, which kTLS does not.
Microbenchmarks
Apple M2 Max, loopback, go test -bench .. Handshake and record framer only;
the offload is Linux-only and is not exercised here:
BenchmarkHandshake/ktls/full 667µs/op (vs 707µs baseline crypto/tls; ECDHE dominates)
BenchmarkHandshake/ktls/resumed 570µs/op (vs 474µs; ~0.1ms per-handshake goroutine hop)
BenchmarkFramerRead 54-57 GB/s, 0 allocs
BenchmarkBuildCryptoInfo 0.8µs (x2 per conn: TX and RX)
Documentation
¶
Overview ¶
Package ktls offloads TLS 1.3 and TLS 1.2 (AES-GCM) record encryption and decryption to the Linux kernel (kTLS) while keeping the handshake in userspace via crypto/tls.
TLS 1.2 support exists primarily because mainline Linux gates TLS *device* offload (TLS_HW, NIC inline crypto) on TLS 1.2 AES-GCM: TLS 1.3 sessions always run as software kTLS. The TLS 1.2 path is simpler than 1.3: on a full handshake the key material comes from the CLIENT_RANDOM key log line plus the server random captured from our own outgoing ServerHello (RFC 5246 key expansion). A resumed handshake logs no CLIENT_RANDOM, so there the master secret is recovered from our own session ticket: the listener owns WrapSession/UnwrapSession and stashes the master in the ticket's public Extra field at issue time, reading it back on resumption. The client random is read from the inbound ClientHello instead. Both record sequences are deterministically 1 after the handshake (each side's Finished is the only record sent under the new keys; session tickets go out in plaintext before the CipherSpec change). Before the irreversible offload, the derived server-write key is trial-decrypted against the captured server Finished record (mirroring the TLS 1.3 trial decryption in computeTXSeq), so any derivation error falls back to userspace cleanly instead of shipping a corrupt first record.
Fork of github.com/Northernside/ktls (MIT License, Copyright (c) 2026 Northernside) with substantial changes:
- Correct TX record sequence number: crypto/tls sends a NewSessionTicket encrypted under the application traffic key during Handshake(), which the upstream library did not account for (breaking every connection with clients that offer psk_dhe_ke, i.e. all browsers). We trial-decrypt the last record written during the handshake with the application traffic key to determine how many records already consumed TX sequence numbers (see computeTXSeq) and seed the kernel accordingly.
- Resumed TLS 1.2 sessions are offloaded, not only full handshakes: a resumed handshake logs no CLIENT_RANDOM, so the listener owns session- ticket issuance/redemption (WrapSession/UnwrapSession over a rotating key ring), stashing the master secret in the ticket's public Extra field at issue time and reading it back on resumption, with the client random taken from the wire (see the TLS 1.2 note above). Gated so a caller's own ticket scheme is never overridden.
- Record-boundary framing during the handshake: the wrapped connection never reads past a TLS record boundary from the socket, so half-RTT application data sent by the client stays in the kernel receive queue and is picked up by kTLS RX. This removes the upstream drain logic and its race (partially buffered ciphertext lost in tls.Conn.rawInput) and makes the RX sequence number 0 by construction.
- RX offload is mandatory: with TX-only kTLS the receive path returns raw ciphertext records, which cannot work for request/response protocols. Offload is all-or-nothing; if it cannot be enabled cleanly the connection falls back to userspace TLS.
- Each handshake runs in its own goroutine instead of serially inside Accept, mirroring net/http's connection-per-goroutine model, so one slow client cannot stall the accept loop.
- Connections that negotiate "h2" via ALPN (or TLS < 1.3) are returned as real *tls.Conn so net/http's HTTP/2 path keeps working (net/http only enables HTTP/2 for concrete *tls.Conn values).
- The offloaded connection implements io.ReaderFrom, delegating to the underlying *net.TCPConn, so net/http can use sendfile(2) for file responses (the kernel encrypts in place; this is the main perf win).
- close_notify alerts are sent through the kernel TLS control-message interface on Close/CloseWrite.
- Reads use recvmsg with a TLS_GET_RECORD_TYPE cmsg buffer: the kernel fails non-application-data records with EIO on a plain read(2), which would turn a peer's clean close_notify into an I/O error. A peer close_notify maps to io.EOF; other alerts surface as errors.
- Key updates (TLS 1.3): peer-initiated KeyUpdates are currently not supported (keyUpdateSupported = false): each one costs HKDF derivations plus setsockopt/sendmsg syscalls, which a peer interleaving KeyUpdates with small application records could exploit as a CPU amplification vector. Such connections are terminated cleanly; clients in the wild virtually never send KeyUpdate mid-session and recover with a fresh handshake. The full rekey flow (RFC 8446 section 4.6.3, kernel >= 6.14) is implemented behind the flag.
Requirements: Linux with the tls module loaded (modprobe tls). This package targets kernel >= 6.14, which covers every feature it uses (TLS 1.2/1.3 TX+RX, MSG_SPLICE_PAGES sendfile, TLS_TX_ZEROCOPY_RO, TLS_RX_EXPECT_NO_PAD, and key updates), so there is no version-gated fallback within the offload path. On other platforms (and if the tls module is absent) every connection transparently falls back to userspace TLS.
On kernels older than 6.14 (e.g. 6.8) everything works except key updates, which need the rekey support added in 6.14: a second setsockopt(TLS_TX/RX) fails with EBUSY there and the kernel cannot switch the RX key, so the next peer record fails to decrypt (EBADMSG) and aborts the session. (With keyUpdateSupported = false a peer KeyUpdate closes the connection on every kernel regardless; see the KeyUpdate note above.)
Known limitations:
- Configs returned from GetConfigForClient bypass the key log capture, so those connections fall back to userspace TLS.
- Client-certificate configurations are not offloaded.
- TLS 1.3 sessions that carry no NewSessionTicket are not offloaded: the kernel TX sequence is derived from that record (see computeTXSeq), so SessionTicketsDisabled, and clients that do not offer psk_dhe_ke, fall back to userspace TLS.
Index ¶
Examples ¶
Constants ¶
const ( ReasonHandshakeError = "handshake-error" ReasonPanic = "panic" // recovered panic in the handshake goroutine ReasonTLSVersion = "tls-version" // < TLS 1.2, stays in userspace ReasonALPN = "alpn" // h2 negotiated; net/http needs a *tls.Conn for HTTP/2 ReasonClientAuth = "client-auth" // client certificates configured, not offloaded ReasonKernel = "kernel" // kTLS unavailable (module missing or non-Linux) ReasonFeatureDisabled = "feature-disabled" // offload turned off at runtime via the gate (feature flag) ReasonCipher = "cipher" // negotiated cipher suite not offloadable (e.g. TLS 1.2 CBC) ReasonFraming = "framing" // unexpected record layout during handshake ReasonSecrets = "secrets" // key material not captured (e.g. GetConfigForClient) ReasonSetsockopt = "setsockopt" // kernel rejected the offload, clean fallback ReasonConnUnusable = "conn-unusable" // partial offload, connection closed )
Fallback / event reasons reported to the observer (see WithObserver).
Variables ¶
This section is empty.
Functions ¶
func Available ¶
func Available() bool
Available reports whether the kernel supports kTLS (tls module loaded), probed once and cached. When false, every connection falls back to userspace TLS (ReasonKernel).
func OffloadErrno ¶
OffloadErrno returns the symbolic errno (e.g. "EBUSY") from a failed offload error, or "" when it carries no syscall errno, used to label why the kernel rejected a kTLS install (ReasonSetsockopt / ReasonConnUnusable).
Types ¶
type Conn ¶
type Conn interface {
net.Conn
syscall.Conn
ConnectionState() tls.ConnectionState
NetConn() net.Conn
}
Conn is implemented by connections returned from Accept when kTLS offload is active. Fallback connections are plain *tls.Conn instead. net/http populates Request.TLS through the ConnectionState method; NetConn mirrors (*tls.Conn).NetConn for callers unwrapping to the raw TCP connection.
type Listener ¶
type Listener struct {
// contains filtered or unexported fields
}
Listener accepts TCP connections, performs the TLS handshake in userspace and hands the established TLS 1.2 or 1.3 session keys to the kernel. Each handshake runs in its own goroutine (mirroring net/http's connection-per-goroutine model, since the offload requires the handshake to complete before the conn reaches net/http); Accept returns fully established connections (either kTLS-offloaded or *tls.Conn fallbacks).
func NewListener ¶
NewListener wraps inner. cfg is used for the userspace handshake and must be non-nil with a certificate source configured.
Example ¶
ExampleNewListener serves HTTPS with kernel TLS offload. The listener is a drop-in net.Listener; connections that cannot be offloaded fall back to a plain *tls.Conn transparently, so enabling it is always safe.
package main
import (
"crypto/tls"
"log"
"net"
"net/http"
ktls "github.com/waipu-oss/go-ktls"
)
func main() {
cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
if err != nil {
log.Fatal(err)
}
inner, err := net.Listen("tcp", ":443")
if err != nil {
log.Fatal(err)
}
ln := ktls.NewListener(inner, &tls.Config{
Certificates: []tls.Certificate{cert},
// h2 stays in userspace, so offer only http/1.1 to drive the kTLS path.
NextProtos: []string{"http/1.1"},
}, ktls.WithObserver(func(reason string, remoteAddr net.Addr, state tls.ConnectionState, err error) {
// reason is "offloaded" on success, otherwise a ktls.Reason* bucket.
if reason != "offloaded" {
log.Printf("ktls fallback %q from %s: %v", reason, remoteAddr, err)
}
}))
defer ln.Close()
srv := &http.Server{
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Served via sendfile(2) through the kernel TLS socket.
http.ServeFile(w, r, "/var/www/html"+r.URL.Path)
}),
}
log.Fatal(srv.Serve(ln))
}
Output:
func (*Listener) Accept ¶
Accept returns the next established connection: a kTLS-offloaded Conn, or a *tls.Conn on fallback. The handshake has already completed (the offload requires it); its failures reach the observer, not the caller.
type Option ¶
type Option func(*Listener)
Option configures a Listener.
func WithHandshakeTimeout ¶
WithHandshakeTimeout bounds the duration of the userspace handshake (default 10s). Connections exceeding it are closed.
func WithObserver ¶
func WithObserver(f func(reason string, remoteAddr net.Addr, state tls.ConnectionState, err error)) Option
WithObserver registers a callback invoked once per connection with the offload outcome: reason "offloaded" with a nil error on success, otherwise one of the Reason* constants (err may be nil, e.g. for ALPN fallbacks). The connection state carries the negotiated TLS version and cipher suite for labelling (zero value on a handshake error, before any negotiation), and remoteAddr identifies the peer (e.g. for handshake-error logs matching net/http's "TLS handshake error from <addr>" format). Useful for metrics. Must be safe for concurrent use.
func WithOffloadGate ¶
WithOffloadGate sets a predicate consulted per connection: when it returns false the connection falls back to userspace TLS (reported as ReasonFeatureDisabled), so an external runtime switch (e.g. a feature flag) can disable kTLS without a restart. Already-offloaded connections are unaffected. nil/unset means always offload. fn must be safe for concurrent use.
Source Files