README
¶
rapidyenc
rapidyenc is a high-performance Go library for decoding yEnc. It provides fast, memory-efficient decoding with robust error handling, supporting multiple platforms and architectures.
The decoder expects an NNTP stream of data, it will perform dot unstuffing and search for the end of responses ".\r\n" this behaviour is not currently configurable.
The module exposes the highly efficient encoding and decoding implementations provided by the C compatible library animetosho/rapidyenc taking advantage CPU features.
Features
- Fast yEnc encoding/decoding using native C implementation via CGO.
- Streaming interface for efficient handling of large files.
- Cross-platform: Supports Linux, Windows, macOS on
amd64andarm64 - Header parsing: Extracts yEnc
Meta(filename, size, CRC32, etc). - Error detection: CRC mismatch, data corruption, and missing headers.
Experimental usage without CGO
Experimental support using simd/archsimd is available without the need for CGO, allowing safer more portable usage.
A port of the AVX2 implementations are available, unsupported platforms will use a generic and slow scalar implementation.
CGO_ENABLED=0 GOEXPERIMENT=simd
Hopefully simd/archsimd will add arm64/neon support in the future and if promoted from an experiment I expect CGO usage/support will be removed entirely.
Usage Examples
Encoding
// An io.Reader of raw data, here random data, but could be a file, bufio.Reader, etc.
raw := make([]byte, 768_000)
_, err := rand.Read(raw)
input := bytes.NewReader(raw)
// yEnc headers
meta := Meta{
FileName: "filename",
FileSize: int64(len(raw)),
PartSize: int64(len(raw)),
PartNumber: 1,
TotalParts: 1,
}
// io.Writer for output
encoded := bytes.NewBuffer(nil)
// Pass input through the Encoder
enc, err := NewEncoder(encoded, meta)
_, err = io.Copy(enc, input)
// Must close to write the =yend footer
err = enc.Close()
Decoding
// An io.Reader of encoded data
input := bytes.NewReader(raw)
output := bytes.NewBuffer(nil)
// Will read from input until io.EOF or ".\r\n"
dec := NewDecoder(input)
meta, err := dec.Next(output) // Writes decoded data to output
// if err == nil then meta contains yEnc headers
Building from Source
It may not be desirable to use the included binary blobs, I could not find a way of avoiding it as there didn't appear to be a way to pass per-file CFLAGS when using CGO. If things have changed or there is a better way please let me know.
See Makefile and build.yml for how the blobs are compiled.
Adding support for other platforms involves creating a toolchain-*.cmake file, adjust Makefile, compile and update cgo.go
Contributing
Pull requests and issues are welcome! Please open an issue for bug reports, questions, or feature requests.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
Functions ¶
func DecodeKernel ¶
func DecodeKernel() string
DecodeKernel returns the name of the implementation being used for decode operations
func EncodeKernel ¶
func EncodeKernel() string
EncodeKernel returns the name of the implementation being used for encode operations
Types ¶
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
func NewDecoder ¶
func NewDecoder(r io.Reader, opts ...DecoderOption) *Decoder
type DecoderOption ¶
type DecoderOption func(d *Decoder)
func WithBufferSize ¶
func WithBufferSize(size int) DecoderOption
WithBufferSize allows the caller to customise the size of the internal buffer used by the decoder
func WithDataFunc ¶
func WithDataFunc(dataFunc func() []byte) DecoderOption
WithDataFunc allows a function to be called when responses need a []byte, for example from a sync.Pool to reduce GC pressure
func WithStatusLineAlreadyRead ¶
func WithStatusLineAlreadyRead() DecoderOption
WithStatusLineAlreadyRead the decoder assumes the stream is positioned at the start of a multiline response body and the caller has already consumed the first line of the response
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
func NewEncoder ¶
NewEncoder returns a new Encoder. Writes to the returned writer are yEnc encoded and written to w.
It is the caller's responsibility to call Close on the Encoder when done.
func (*Encoder) Close ¶
Close flushes any pending output from the encoder and writes the trailing header. It is an error to call Write after calling Close.
func (*Encoder) Reset ¶
Reset discards the Encoder e's state and makes it equivalent to the result of its original state from NewEncoder, but writing to w instead. This permits reusing a Encoder rather than allocating a new one.
type EncoderOption ¶
type EncoderOption func(*Encoder)
func WithLineLength ¶
func WithLineLength(lineLength int) EncoderOption
WithLineLength configures the encoded line length
type End ¶
type End int
End is the State for incremental decoding, whether the end of the yEnc data was reached
type Meta ¶
type Meta struct {
FileName string
FileSize int64 // Total size of the file
PartNumber int64
TotalParts int64
Offset int64 // Offset of the part within the file relative to the start, like io.Seeker or io.WriterAt
PartSize int64 // Size of the unencoded data
}
Meta is the result of parsing the yEnc headers (ybegin, ypart, yend)
type Response ¶
type Response struct {
Metadata ResponseMeta
Data []byte
// contains filtered or unexported fields
}
Source Files