README
¶
ToyTLV
TLV or Type-Length-Value is the default way to implement binary protocols. You may ask, why not Protobuf? The answer is, Protobuf is a TLV protocol, as almost every other binary protocol out there. ToyTLV is a bare-bones TLV, nothing else.
This lib implements a really simple TLV where every record type is a letter (A-Z), while the length is either 8- or 32-bit. The value of a record has arbitrary structure, ToyTLV mandates nothing in this regard. Hint: nesting TLV records is trivial.
More formally, a ToyTLV record can go in 3 forms:
- long: the type letter is uppercase [A-Z], the length is a little-endian uint32,
- short: the type letter is lowercase [a-z], the length is uint8. That is merely an optimization to handle lots of small records (like short strings).
- tiny: the type is known in advance, the length is ASCII [0-9]. This is for tiny records (e.g. small ints), where one byte of overhead can make a difference (that happens).
The lib implements basic ToyTLV file and network I/O. That is:
- reading/writing TLV files,
- basic TLV over TCP fun: connecting, listening, reconnecting with exponential backoff, and otherwise managing the connections (see TCPDepot).
That is all it does.
Documentation
¶
Index ¶
- Constants
- Variables
- func Append(into []byte, lit byte, body ...[]byte) (res []byte)
- func AppendHeader(into []byte, lit byte, bodylen int) (ret []byte)
- func AppendRead(buf []byte, rdr io.Reader, lenHint int) ([]byte, error)
- func AppendTiny(into []byte, lit byte, body []byte) (res []byte)
- func CloseHeader(buf []byte, bookmark int)
- func Concat(msg ...[]byte) []byte
- func Incomplete(data []byte) int
- func Join(records ...[]byte) (ret toyqueue.Records)
- func Lit(rec []byte) byte
- func OpenHeader(buf []byte, lit byte) (bookmark int, res []byte)
- func ProbeHeader(data []byte) (lit byte, hdrlen, bodylen int)
- func ProbeHeaders(lits string, data []byte) int
- func ReadBuf(buf []byte, rdr io.Reader) ([]byte, error)
- func Record(lit byte, body ...[]byte) []byte
- func Records(lit byte, bodies ...[]byte) (recs toyqueue.Records)
- func RoundPage(l int) int
- func Split(data []byte) (recs toyqueue.Records, rest []byte, err error)
- func Take(lit byte, data []byte) (body, rest []byte)
- func TakeAny(data []byte) (lit byte, body, rest []byte)
- func TakeAnyRecord(data []byte) (lit byte, rec, rest []byte)
- func TakeAnyWary(data []byte) (lit byte, body, rest []byte, err error)
- func TakeRecord(lit byte, data []byte) (rec, rest []byte)
- func TakeWary(lit byte, data []byte) (body, rest []byte, err error)
- func TinyRecord(lit byte, body []byte) (tiny []byte)
- func TotalLen(inputs [][]byte) (sum int)
- type Jack
- type ReadCloser2FeedCloser
- type ReadSeekCloser2FeedSeekCloser
- type ReadSeeker2FeedSeeker
- type Reader2Feeder
- type TCPConn
- type TCPDepot
- func (de *TCPDepot) Close()
- func (de *TCPDepot) Connect(addr string) (err error)
- func (de *TCPDepot) Disconnect(addr string) (err error)
- func (de *TCPDepot) DrainTo(recs toyqueue.Records, addr string) error
- func (de *TCPDepot) KeepListening(addr string)
- func (de *TCPDepot) Listen(addr string) (err error)
- func (de *TCPDepot) Open(jack Jack)
- func (de *TCPDepot) StopListening(addr string) error
- type WritCloser2DrainCloser
- type Writer2Drainer
Constants ¶
const CaseBit uint8 = 'a' - 'A'
const DefaultPreBufLength = 4096
const MAX_RETRY_PERIOD = time.Minute
const MIN_RETRY_PERIOD = time.Second / 2
const MaxOutQueueLen = 1 << 20 // 16MB of pointers is a lot
const MinRecommendedRead = 512
const MinRecommendedWrite = 400
const TYPICAL_MTU = 1500
Variables ¶
var ErrAddressUnknown = errors.New("address unknown")
var ErrBadRecord = errors.New("bad TLV record format")
var ErrDisconnected = errors.New("disconnected by user")
var ErrIncomplete = errors.New("incomplete data")
Functions ¶
func Append ¶
Append appends a record to the buffer; note that uppercase type is always explicit, lowercase can be defaulted.
func AppendHeader ¶
Feeds the header into the buffer. Subtle: lower-case lit allows for defaulting, uppercase must be explicit.
func AppendRead ¶ added in v0.1.8
AppendRead reads data from io.Reader into the *spare space* of the provided buffer, i.e. those cap(buf)-len(buf) vacant bytes. If the spare space is smaller than lenHint, allocates (as reading less bytes might be unwise).
func CloseHeader ¶
CloseHeader closes a streamed TLV record
func Incomplete ¶ added in v0.1.8
Incomplete returns the number of supposedly yet-unread bytes. 0 for complete, -1 for bad format, >0 for least-necessary read to complete either header or record.
func OpenHeader ¶
OpenHeader opens a streamed TLV record; use append() to create the record body, then call CloseHeader(&buf, bookmark)
func ProbeHeader ¶
ProbeHeader probes a TLV record header. Return values:
- 0 0 0 incomplete header
- '-' 0 0 bad format
- 'A' 2 123 success
func ProbeHeaders ¶
func Take ¶
Take is used to read safe TLV inputs (e.g. from own storage) with record types known in advance.
func TakeAnyRecord ¶
func TakeAnyWary ¶
TakeWary reads TLV records of arbitrary type from unsafe input.
func TakeRecord ¶
func TinyRecord ¶ added in v0.2.0
Types ¶
type ReadCloser2FeedCloser ¶
type ReadCloser2FeedCloser struct {
Reader io.ReadCloser
// contains filtered or unexported fields
}
func (*ReadCloser2FeedCloser) Close ¶
func (fs *ReadCloser2FeedCloser) Close() error
type ReadSeekCloser2FeedSeekCloser ¶
type ReadSeekCloser2FeedSeekCloser struct {
Reader io.ReadSeekCloser
// contains filtered or unexported fields
}
func (*ReadSeekCloser2FeedSeekCloser) Close ¶
func (fs *ReadSeekCloser2FeedSeekCloser) Close() error
type ReadSeeker2FeedSeeker ¶
type ReadSeeker2FeedSeeker struct {
Reader io.ReadSeeker
// contains filtered or unexported fields
}
type Reader2Feeder ¶
Feeder reads TLV records from an io.Reader stream. Note that Feeder is buffered, i.e. it reads ahead. When doing Seek() on a file, recreate Feeder, that is cheap.
type TCPConn ¶
func (*TCPConn) KeepTalking ¶
func (tcp *TCPConn) KeepTalking()
type TCPDepot ¶
type TCPDepot struct {
// contains filtered or unexported fields
}
A TCP server/client for the use case of real-time async communication. Differently from the case of request-response (like HTTP), we do not wait for a request, then dedicating a thread to processing, then sending back the resulting response. Instead, we constantly fan sendQueue tons of tiny messages. That dictates different work patterns than your typical HTTP/RPC server as, for example, we cannot let one slow receiver delay event transmission to all the other receivers.
func (*TCPDepot) Disconnect ¶
func (*TCPDepot) KeepListening ¶
func (*TCPDepot) StopListening ¶
type WritCloser2DrainCloser ¶
type WritCloser2DrainCloser struct {
Writer io.WriteCloser
}
func (*WritCloser2DrainCloser) Close ¶
func (dc *WritCloser2DrainCloser) Close() error
Source Files