README
¶
go-yenc-simd
A fast yEnc decoder for Go, for articles from Usenet (NNTP). Pure Go, no cgo.
With GOEXPERIMENT=simd it uses AVX, AVX2 or AVX-512 on amd64 and NEON on
arm64; everywhere else it is plain Go.
go get github.com/go-while/go-yenc-simd
Decode an article
You have the article body as the server sent it (lines ending in \r\n,
ending with a line holding a single .):
import yencsimd "github.com/go-while/go-yenc-simd"
data, meta, err := yencsimd.Decode(article)
if err != nil {
// yencsimd.ErrCRC, ErrSize, ErrTruncated or ErrNoYbegin
}
fmt.Println(meta.Name, len(data))
Decode finds the =ybegin line (headers before it are skipped, so an
ARTICLE response works too), decodes the data, and checks the size and the
CRC32 against the =yend line.
Stream from an NNTP connection
Send BODY <message-id>, read the 222 status line, then hand the
connection's buffered reader to NewReader:
tp := textproto.NewConn(conn)
id, _ := tp.Cmd("BODY %s", msgid)
tp.StartResponse(id)
if _, _, err := tp.ReadCodeLine(222); err != nil {
return err
}
d := yencsimd.NewReader(tp.R) // tp.R, NOT tp.DotReader()
_, err := io.Copy(file, d) // io.EOF -> nil: size and CRC were checked
tp.EndResponse(id)
meta := d.Meta() // Name, Size, Part, Total, Begin, End, CRC32
NewReader reads through the closing . line, so the connection is ready
for the next command.
Each step decodes what the bufio.Reader has buffered (never reading past
the article), straight into the caller's buffer when that can take it or
holds 8 KiB or more (io.Copy's 32 KiB buffer does). So the buffer size sets how much
work a step does and how often the connection is read: textproto.NewConn
uses the 4 KiB default; textproto.NewReader(bufio.NewReaderSize(conn, 64<<10)) makes bigger steps and up to 16 times fewer reads.
Don't use tp.DotReader(). It removes NNTP's dot-stuffing (.. at a
line start → .), and the decoder does that too, so a line that really
starts with a dot would lose it.
Multi-part files
Each article is one part. Meta says where it goes:
data, meta, err := yencsimd.Decode(article)
// meta.Part of meta.Total, bytes meta.Begin..meta.End (1-based, inclusive)
// of a file of meta.Size bytes
f.WriteAt(data, meta.Begin-1)
For a part, the CRC checked is the pcrc32= of that part.
Build with SIMD
GOEXPERIMENT=simd go build ./...
amd64 with GOEXPERIMENT=simd |
AVX-512 VBMI2, AVX2 or AVX, chosen at run time |
arm64 with GOEXPERIMENT=simd |
NEON, 64 bytes per step |
| anything else | plain Go, 16 bytes per step (about 1–3.5 GB/s); one byte per step on big-endian CPUs and those slow at unaligned loads (riscv64, mips, 32-bit arm) |
GOEXPERIMENT=simd needs Go 1.27 and is an experiment, outside the Go 1
compatibility promise. Without it the package builds and works the same,
only slower. yencsimd.Kernel() tells you which path runs.
Low level: DecodeIncremental
Reader is built on DecodeIncremental, a drop-in for rapidyenc's
rapidyenc_decode_incremental. Use it if you parse the =y lines yourself
or feed data in your own chunks:
st := yencsimd.StateCRLF // the data starts at a line start
dst := make([]byte, len(chunk)) // output is never longer than input
n, used, end, err := yencsimd.DecodeIncremental(dst, chunk, &st)
// dst[:n]: decoded data; chunk[:used] was consumed
// end == EndControl: "=y" reached; chunk[used-2:] starts the =yend line
// end == EndArticle: "\r\n.\r\n" reached (the article ended without =yend)
// end == EndNone: all of chunk decoded; pass the rest with the same st
It expects the data between the =ybegin/=ypart lines and =yend, still
dot-stuffed. It does not check the CRC.
Speed
750 KB article, median of 3 pinned runs of go test -bench ., in MB/s of
input. rapidyenc is the C reference (animetosho/rapidyenc, gcc 14, same input,
same core).
| Intel i7-10710U (AVX2), fixed 2 GHz | AMD EPYC 9B45 (Zen 5, AVX-512) | |
|---|---|---|
DecodeIncremental, random data |
2.9 GB/s | 24.7 GB/s (AVX-512), 15.1 GB/s (AVX2) |
| same, data that needs no escapes | 4.8 GB/s | 26.0 GB/s, 22.7 GB/s |
| same, an escape in every second byte | 3.0 GB/s | 21.9 GB/s, 11.3 GB/s |
same, plain Go (no GOEXPERIMENT=simd) |
0.95 GB/s | 3.4 GB/s |
Decode (whole article, size and CRC checked) |
1.8 GB/s | 10.4 GB/s |
Reader on a 64 KiB bufio.Reader |
2.0 GB/s | 12.6 GB/s |
| rapidyenc, random data | 3.8 GB/s | 17.3 GB/s (VBMI2), 18.8 GB/s (AVX2) |
Decode and Reader include the CRC32 (hash/crc32: 10 GB/s on the i7,
33 GB/s on the EPYC), which is now a large share of their time.
On arm64 (NEON) the numbers are from GitHub's runners: yencbench -time 500ms in five CI runs. They are shared VMs, not pinned, so the Apple M1
numbers vary a lot between runs:
Apple M1 (macos-latest) |
Neoverse N2 (ubuntu-24.04-arm) |
|
|---|---|---|
DecodeIncremental, random data |
6.0–8.0 GB/s | 5.4 GB/s |
| same, data that needs no escapes | 7.0–11.8 GB/s | 7.7 GB/s |
| same, an escape in every second byte | 6.2–9.2 GB/s | 5.8 GB/s |
same, plain Go (no GOEXPERIMENT=simd) |
1.9–2.7 GB/s | 2.6–2.7 GB/s |
Decode (whole article, size and CRC checked) |
2.1–3.2 GB/s | 3.4 GB/s |
One run of macos-latest got an Apple M2 Pro instead: 8.9 GB/s on random
data, plain Go 2.8 GB/s.
Benchmarks
The quickest way, no clone needed (Go 1.27):
GOEXPERIMENT=simd go run github.com/go-while/go-yenc-simd/cmd/yencbench@latest
It prints MB/s for every kernel your CPU can run (AVX-512 VBMI2, AVX2, AVX or
NEON, and scalar) on three 750 KB articles (random data, data that needs no
escapes, an escape in every second byte), and for Decode of a whole article
with the CRC check. Without GOEXPERIMENT=simd it measures the plain Go
build. Flags: -time 1s per measurement, -size 750000 data bytes per
article, -kernels avx2,scalar to pick kernels. Pin it to one core for steady
numbers (taskset -c 2 ... on Linux).
From a program, for example behind a -yencbench flag:
import "github.com/go-while/go-yenc-simd/yencbench"
if *bench {
if err := yencbench.Run(os.Stdout, yencbench.Options{}); err != nil {
log.Fatal(err)
}
return
}
yencbench.Measure returns the numbers instead of printing them; the kernel
in use is restored afterwards. yencsimd.Kernels() lists the kernels and
yencsimd.SetKernel picks one (only while nothing is decoding).
The full set, including Reader with 4 KiB and 64 KiB buffers, is in the
package's tests:
git clone https://github.com/go-while/go-yenc-simd && cd go-yenc-simd
GOEXPERIMENT=simd go test -run '^$' -bench . -benchtime 1s
How it works, and tests
- A 7-state machine that matches rapidyenc's scalar reference byte for byte.
- Vector loops (16, 32 or 64 bytes) for the common bytes;
=escapes, line breaks and stuffed dots are handled in the vector code too. Rare cases (==, escaped line breaks, the end markers) go through the state machine. - Tests: 504 golden outputs from rapidyenc's C decoder on every vector path,
each vector path against the state machine (random inputs; every string of
up to 4 of
=,\r,\n,.,yand a plain byte, at every offset from 0 to 140, so across every vector boundary; fuzzing), encode/decode round trips, andReaderon whole and byte-by-byte input. On amd64 a test reads the machine code and checks that each kernel uses only instructions its CPU check guarantees (an AVX2 instruction in the AVX kernel would crash Sandy Bridge).
GOEXPERIMENT=simd go test ./...
Credits
The decoding rules follow rapidyenc by animetosho (public domain / CC0).
License
MIT, see LICENSE.
Documentation
¶
Overview ¶
Package yencsimd decodes yEnc articles from Usenet (NNTP), in pure Go, with SIMD on amd64 and arm64 when built with GOEXPERIMENT=simd.
Most programs need one of two functions:
- Decode(article) for an article in memory: it returns the data and its Meta (name, size, part) and checks size and CRC32.
- NewReader(r) to stream an article from an NNTP connection: pass the connection's *bufio.Reader after the "222" status line (with net/textproto: tp.R, not tp.DotReader()).
Both expect the article as the server sends it: CRLF lines, dot-stuffed, ending with a "." line.
DecodeIncremental is the low-level step underneath, a drop-in for rapidyenc's rapidyenc_decode_incremental: it decodes the yEnc data in chunks of any size, undoes the dot-stuffing and stops at "=y" (the =yend line) or at the end of the article. Parsing the =y lines and the CRC are then up to the caller.
The byte-wise decoder is a small state machine that matches rapidyenc's scalar reference (decoder.cc, do_decode_end_scalar<isRaw=true>) state for state. With GOEXPERIMENT=simd, vector loops decode 16, 32 or 64 bytes per step: AVX, AVX2 or AVX-512 VBMI2 on amd64, NEON on arm64; elsewhere it is plain Go, 16 bytes per step (two 64-bit words, SWAR), or one on CPUs that are big-endian or slow at unaligned loads.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNoYbegin: the article ended before a "=ybegin" line. ErrNoYbegin = errors.New("yencsimd: no =ybegin line") // ErrTruncated: the article ended inside the yEnc data, before "=yend". ErrTruncated = errors.New("yencsimd: article ends before =yend") // ErrSize: the decoded size differs from the =ybegin/=ypart/=yend sizes. ErrSize = errors.New("yencsimd: size mismatch") // ErrCRC: the CRC32 of the decoded data differs from the =yend line. ErrCRC = errors.New("yencsimd: crc32 mismatch") )
Errors returned by Reader and Decode.
var ErrDstTooSmall = errors.New("yencsimd: destination must be at least the length of source")
ErrDstTooSmall is returned when dst is shorter than src.
Functions ¶
Types ¶
type End ¶
type End int
End reports whether, and how, decoding stopped early.
func DecodeIncremental ¶
DecodeIncremental decodes src into dst, starting in *state, and sets *state to the state after the last byte it read. It stops after an end sequence (see End). dst must be at least len(src): the output is never longer than the input. dst and src must not overlap: the decoder writes scratch bytes past the output position, so in-place decoding is not supported.
Example ¶
The low-level step: decode yEnc data (the lines between =ybegin and =yend) in chunks of any size. Reader is built on this.
package main
import (
"fmt"
yencsimd "github.com/go-while/go-yenc-simd"
)
func main() {
body := []byte("r\x8f\x96\x96\x99VJ\xa3o\x98\x8dK\r\n=yend size=12\r\n")
st := yencsimd.StateCRLF // the data starts at a line start
dst := make([]byte, len(body))
n, used, end, _ := yencsimd.DecodeIncremental(dst, body, &st)
fmt.Printf("%q, stopped at %q (end=%v)\n", dst[:n], body[used-2:used+3], end == yencsimd.EndControl)
}
Output: "Hello, yEnc!", stopped at "=yend" (end=true)
type Meta ¶
type Meta struct {
Name string // file name (=ybegin name=)
Size int64 // size of the whole file (=ybegin size=)
Part int // part number (=ybegin part=), 0 for a single-part post
Total int // number of parts (=ybegin total=), 0 if not given
Begin int64 // first byte of this part in the file, 1-based (=ypart begin=); 1 for single-part
End int64 // last byte of this part, inclusive (=ypart end=); Size for single-part
CRC32 uint32 // CRC32 the trailer gives for this part (pcrc32=, or crc32= for single-part)
HasCRC bool // whether the trailer had one; without it the CRC isn't checked
}
Meta is what the yEnc header and trailer lines say about the data. Name, Size, Part, Total, Begin and End are set once Read returned the first data (or an error); CRC32 and HasCRC once Read returned io.EOF.
func Decode ¶
Decode decodes one yEnc article held in memory (as Reader expects it: CRLF lines, dot-stuffed, optionally with headers before =ybegin) and checks its size and CRC32. It returns the same errors as Reader, with the data decoded up to the error.
The data is decoded in one step into a buffer as long as the rest of the article after the header lines (the output is never longer), so the returned slice has some spare capacity: about 3% for a typical post.
Example ¶
Decode a whole article you already have in memory.
package main
import (
"fmt"
yencsimd "github.com/go-while/go-yenc-simd"
)
// The article as a server sends it after "222 ... body follows":
// "Hello, yEnc!" encoded, dot-stuffed, ending in a "." line.
const article = "=ybegin line=128 size=12 name=hello.txt\r\n" +
"r\x8f\x96\x96\x99VJ\xa3o\x98\x8dK\r\n" +
"=yend size=12 crc32=0337ab3d\r\n" +
".\r\n"
func main() {
data, meta, err := yencsimd.Decode([]byte(article))
if err != nil {
fmt.Println("error:", err) // ErrCRC, ErrSize, ErrTruncated or ErrNoYbegin
return
}
fmt.Printf("%s: %q (%d bytes)\n", meta.Name, data, meta.Size)
}
Output: hello.txt: "Hello, yEnc!" (12 bytes)
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader decodes one yEnc article from an NNTP connection or a buffer.
It expects the article the way it comes off the wire after a "222" (BODY) or "220" (ARTICLE) status line: lines end in CRLF, lines starting with "." are dot-stuffed, and the article ends with a line holding a single ".". Header lines and anything else before "=ybegin" are skipped. Do not put a net/textproto DotReader in front of it: that already removes the dot-stuffing, and Reader would remove it a second time.
Read returns the decoded data. At the end it checks the sizes and the CRC32 and returns io.EOF, or ErrSize / ErrCRC / ErrTruncated / ErrNoYbegin. By then it has read through the closing "." line (or to the end of the input), so an NNTP connection is ready for its next command. A malformed header (a part without an =ypart line, or another =y line in the data) gives a plain error; after it the input is still inside the article, so drop the connection or read to the "." line yourself.
func NewReader ¶
NewReader returns a Reader that decodes one article from r. If r is a *bufio.Reader it is used directly (net/textproto's Reader.R is one), so no bytes of the next response are read ahead and lost.
Each step decodes what the bufio.Reader has buffered, so its buffer size sets how much work a step does and how often the connection is read: for an NNTP connection give textproto a bufio.NewReaderSize(conn, 64<<10) rather than the 4 KiB default; Reader never reads past the article.
Example ¶
Stream an article straight from an NNTP connection (net/textproto). Pass the connection's bufio.Reader (tp.R), not tp.DotReader(): Reader handles the dot-stuffing and the closing "." line itself, and leaves the connection at the next response.
package main
import (
"bufio"
"fmt"
"io"
"net/textproto"
"strings"
yencsimd "github.com/go-while/go-yenc-simd"
)
// The article as a server sends it after "222 ... body follows":
// "Hello, yEnc!" encoded, dot-stuffed, ending in a "." line.
const article = "=ybegin line=128 size=12 name=hello.txt\r\n" +
"r\x8f\x96\x96\x99VJ\xa3o\x98\x8dK\r\n" +
"=yend size=12 crc32=0337ab3d\r\n" +
".\r\n"
func main() {
// a stand-in for the server's reply to "BODY <id>"
tp := textproto.NewReader(bufio.NewReader(strings.NewReader("222 0 <id@example.test> body\r\n" + article)))
if _, _, err := tp.ReadCodeLine(222); err != nil {
fmt.Println("error:", err)
return
}
d := yencsimd.NewReader(tp.R)
var out strings.Builder
if _, err := io.Copy(&out, d); err != nil {
fmt.Println("error:", err)
return
}
fmt.Printf("%s: %q\n", d.Meta().Name, out.String())
}
Output: hello.txt: "Hello, yEnc!"
func (*Reader) Read ¶
Read implements io.Reader. When p can take all that is buffered, or holds 8 KiB or more, it decodes straight into p, so io.Copy (32 KiB pieces) and io.ReadAll (once its buffer has grown) get the data without a copy. Read may use p[n:] as scratch space, as io.Reader allows: bytes of p past n are not preserved.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
yencbench
command
Command yencbench prints how fast go-yenc-simd decodes on this machine, for every kernel the CPU can run:
|
Command yencbench prints how fast go-yenc-simd decodes on this machine, for every kernel the CPU can run: |
|
Package yencbench measures how fast yencsimd decodes on this machine, with the same inputs as the package's own benchmarks.
|
Package yencbench measures how fast yencsimd decodes on this machine, with the same inputs as the package's own benchmarks. |