proxy

Documentation

Overview

Command proxy rewrites HTML response bodies in a reverse proxy, and skips the ones it must not touch.

$ proxy -listen :8080 -upstream http://localhost:3000
rewriting text/html; skipping everything else

The rewrite itself is four lines. What takes the rest of this program is deciding which responses to apply it to, and every one of those decisions is a way to break a site.

Content-Type

Only text/html. A JSON body survives a rewrite unchanged - it is valid UTF-8 and contains no markup - so the mistake looks harmless until a body arrives that is not valid UTF-8. Then it does not survive: see below.

The charset parameter is the encoding the bytes are in, and it is the authority - a meta in the document is ordinary markup to the rewriter. text/html;charset=windows-1252 becomes WithEncoding("windows-1252").

A label the library cannot use is a reason to pass the body through rather than to guess, and the way to find out is to build the rewriter: NewWriter returns an lolhtml.EncodingError for a label it does not know and for one that is not ASCII-compatible. The second is the one that matters in a proxy - "utf-16le" is a real encoding and a real Content-Type, and a rewriter cannot work in it at all:

windows-1252   fine
iso-8859-1     fine
utf-16le       EncodingError: Expected ASCII-compatible encoding
utf-7          EncodingError: Unknown character encoding
bogus          EncodingError: Unknown character encoding

So this program tries to build and falls back to passing the body through, which needs no list of supported labels in the proxy and cannot drift from the library's.

Content-Encoding

This is the one that destroys responses. A compressed body is not text, and what happens to it depends on something a reader would not expect - whether a text handler is registered:

body                  with an element handler   with a text handler
gzip, 36 bytes        36 bytes, identical       64 bytes, not gzip any more
a PNG header          identical                 35 bytes of 33
256 random bytes      identical                 482 bytes
valid UTF-8           identical                 identical

A text handler decodes and re-encodes the document, so every byte that is not valid in the declared encoding becomes U+FFFD - three bytes where there was one. With only element handlers nothing decodes, so a compressed body passes through untouched and the rewrite silently does nothing, which is the other classic proxy bug. Neither case reports an error.

So: either ask upstream not to compress (Accept-Encoding: identity on the way out) or decompress before the rewriter and recompress after it. This program does the first, because it is one header and no buffering.

Content-Length

Delete it. The rewrite changes the length, and a Content-Length that disagrees with the body is a protocol error - the client either truncates the page or waits for bytes that never come. Go's httputil.ReverseProxy will not fix this for you.

Streaming

The rewriter writes as it goes, so the client can start rendering before the upstream has finished - which is most of the reason to rewrite in a proxy rather than in a template. That only holds if nothing between the rewriter and the socket buffers, so the response writer is flushed as output arrives, and ReverseProxy is given a FlushInterval.

What a failure costs

A rewrite that fails partway has already sent a prefix of the page: the status line and headers went out with the first byte, so there is no way to turn a broken rewrite into a 502. The prefix is well-formed HTML as far as it goes - see the package documentation on stopping early - which is worse than an error page in one way and better in another: the client sees a short page rather than an error, and nothing in it is malformed. Log it; do not pretend the response can be retracted.