README
¶
sijsop - SImple JSOn Protocol
sijsop provides a JSON-based wire or file protocol that provides extremely good bang-for-the-buck, as well as being very easy to implement in other languages. See the godoc for more information.
Code Signing
I will be signing this repository with the "jerf" keybase account. If you are viewing this repository through GitHub, you should see the commits as showing as "verified" in the commit view.
(Bear in mind that due to the nature of how git commit signing works, there may be runs of unverified commits; what matters is that the top one is signed.)
Changlog
- 1.0.2
- Add the ability to set the indent on the JSON marshaling process, for debugging purposes. (You can set the indent up to make the messages on the wire more readable for debugging.)
- 1.0.1
- Finish removing the restrictions around a 255-char limit on types. I mean, in my opinion you shouldn't have types that long, but there's no reason to limit you.
- Remove the error return from Register for documentation simplicity.
- 1.0.0
- Per ESR, this changes the protocol to pure text. This is API compatible with 0.9.0, but protocol incompatible.
- 0.9.0
- This was pulled from some production code and then modified to be suitable for public release. The modifications haven't been tested. I used the predecessor in production code, and the test cases are passing, so I'm reasonably confident this is useful code, but I'm going to let it bake in a bit (and take some time later to switch my internal code to using this) to be sure before I declare it 1.0.
Documentation
¶
Overview ¶
Package sijsop provides a SImple JSOn Protocol that is easily parsed by numerous languages, guided by experience in multiple languages.
This protocol has one of the best bang-for-the-buck ratios you can find for any protocol. It is so easy to implement that you can easily implement this for another language, yet as powerful as the JSON processing in your language. There's better protocols on every other dimension, for compactness, efficiency, safety, etc., but when you just want to bash out a line protocol quickly and those other concerns aren't all that pressing, this is a pretty decent choice. It also has the advantage of not committing you very hard to a particular implementation, because any network engineer can bang out an implementation of this in another language in a couple of hours.
Usage In Go ¶
This protocol is defined in terms of messages, which must implement sijsop.Message. This involves writing two static class methods, one which declares a string which uniquely identifies this type for a given Definition, and one which simply returns a new instance of the given type. These must be defined on the pointer receiver for the type, because we must be able to modify the values for this code to work.
Once you have a Definition, you can use it to wrap a Reader, which will permit you to receive messages, a Writer, which will permit you to send them, or a ReadWriter, which allows for bidirectional communication.
sijsop is careful to never read more than the next message, so it is quite legal using this protocol to send some message that indicates some concrete length, then use the underlying Reader/Writer/ReadWriter directly, then smoothly resume using the sijsop protocol. See the example file shown below.
The Wire Protocol ¶
The protocol is a message-based protocol which works as follows:
- An ASCII number ("1", "342", etc.), followed by a newline (byte 10).
- That many bytes of arbitrary string indicating the type of the next message, the "type tag", followed by a newline. (The newline is NOT included in the first number.)
- An ASCII number ("1", "342", etc.), followed by a newline (byte 10).
- That much JSON, followed by a newline. (Again, the newline is NOT included in the length count, but it must be present.)
Commentary: I've often used variants of this protocol that used just the JSON portion, but I found myself repeatedly having to parse the JSON once for the type, and then again with the right type. Giving statically-typed languages a chance to pick the type out in advance helps with efficiency. Static languages can just ignore than field if they like.
Everything else is left up to the next layer. Protocols that are rigid about message order are easy. Protocols that need to match requests to responses are responsible for their own next-higher-level mapping.
Example ¶
This example demonstrates transferring a binary file using the sijsop protocol, with the file transferred out-of-band. Left is going to read a file and send it to Right, who will acknowledge it.
package main
import (
"bytes"
"fmt"
"io"
"os"
"sync"
)
type FileTransfer struct {
Name string `json:"name"`
Size int64 `json:"size"`
}
func (ft *FileTransfer) SijsopType() string {
return "file_transfer"
}
func (ft *FileTransfer) New() Message {
return &FileTransfer{}
}
type FileTransferAck struct {
Name string `json:"name"`
}
func (fta *FileTransferAck) SijsopType() string {
return "file_transfer_ack"
}
func (fta *FileTransferAck) New() Message {
return &FileTransferAck{}
}
type RW struct {
io.ReadCloser
io.WriteCloser
}
// This example demonstrates transferring a binary file using the sijsop
// protocol, with the file transferred out-of-band. Left is going to read a
// file and send it to Right, who will acknowledge it.
func main() {
toRightRead, fromLeftWrite := io.Pipe()
toLeftRead, fromRightWrite := io.Pipe()
protocol := &Definition{}
protocol.Register(&FileTransfer{}, &FileTransferAck{})
wg := sync.WaitGroup{}
wg.Add(2)
// The left side, who will be reading the file and sending it
go func() {
sp := protocol.Wrap(RW{toLeftRead, fromLeftWrite})
f, err := os.Open("sijsop_example_test.go")
if err != nil {
panic("Couldn't read this file")
}
fi, err := f.Stat()
if err != nil {
panic("couldn't stat this file")
}
transfer := &FileTransfer{
Name: "sijsop_example_test.go",
Size: fi.Size(),
}
err = sp.Send(transfer)
if err != nil {
panic("Couldn't send transfer request")
}
// now we copy the file over the stream directly
_, _ = io.Copy(fromLeftWrite, f)
ack := &FileTransferAck{}
err = sp.Unmarshal(ack)
if err != nil {
panic("didn't get the expected ack")
}
// deliberately done here, to ensure we received the ack
wg.Done()
}()
// The right side, who will be receiving it and ack'ing it.
go func() {
sp := protocol.Wrap(RW{toRightRead, fromRightWrite})
// show just receiving next message
msg, err := sp.ReceiveNext()
if err != nil {
panic("couldn't get the file transfer request")
}
size := msg.(*FileTransfer).Size
// Now directly extract the file from the bytestream
r := &io.LimitedReader{toRightRead, size}
buf := &bytes.Buffer{}
_, _ = io.Copy(buf, r)
// and send the ack, transititioning back to JSON
err = sp.Send(&FileTransferAck{msg.(*FileTransfer).Name})
if err != nil {
panic("couldn't send acknowledgement")
}
fileContents := buf.String()
// print out the package declaration to show the file was truly
// transferred.
fmt.Println(fileContents[:15])
wg.Done()
}()
wg.Wait()
}
Output: package sijsop
Index ¶
Examples ¶
Constants ¶
const (
DefaultSizeLimit = 2 << 30
)
Variables ¶
var ErrClosed = errors.New("can't send/receive on closed stream")
ErrClosed is returned when the Handler is currently closed.
var ErrNoUnmarshalTarget = errors.New("can't unmarshal a nil message")
ErrNoUnmarshalTarget is returned when you attempt to Unmarshal(nil).
var ErrTypeAlreadyRegistered = errors.New("sijsop type already registered")
ErrTypeAlreadyRegistered is returned if you attempt to register a type when the SijsopType() has already been registered.
var ErrTypeTooLong = errors.New("sijsop type name too long")
ErrTypeTooLong is returned if you attempt to register a type that claims a name that is more than 255 characters long.
Functions ¶
This section is empty.
Types ¶
type Definition ¶
type Definition struct {
// contains filtered or unexported fields
}
A Definition defines a specific protocol, with specific messages.
Types should be registered with the definition before use; once a Reader, Writer or Handler has been created from the Definition it is no longer safe to register more types.
func (*Definition) Reader ¶
func (d *Definition) Reader(r io.Reader) *Reader
Reader creates a new Reader around the given io.Reader that implements the protocol given by the Definition.
func (*Definition) Register ¶
func (d *Definition) Register(types ...Message)
Register registers the given messages with given Definition.
The last registration for a given .SijsopType() will be the one that is used.
func (*Definition) Wrap ¶
func (d *Definition) Wrap(rw io.ReadWriter) *Handler
Wrap creates a new Handler around the given io.ReadWriter that implements the protocol given by the Definition.
type ErrJSONTooLarge ¶
ErrJSONTooLarge is returned when the encoded JSON of a message is above the legal threshold.
func (ErrJSONTooLarge) Error ¶
func (ejtl ErrJSONTooLarge) Error() string
type ErrUnknownType ¶
type ErrUnknownType struct {
ReceivedType string
}
ErrUnknownType is returned when the message couldn't be returned by ReceiveNext because it's an unregistered type.
func (ErrUnknownType) Error ¶
func (eut ErrUnknownType) Error() string
Error implements the Error interface.
type ErrWrongType ¶
ErrWrongType is returned when using Unmarshal but the wrong type is passed in. The ExpectedType and ReceivedType will be the JSONType of the types.
func (ErrWrongType) Error ¶
func (ewt ErrWrongType) Error() string
Error implements the Error interface.
type Message ¶
type Message interface {
SijsopType() string
// Returns a new zero instance of the struct in question.
New() Message
}
Message describes messages that can be registered with a Definition, and subsequently sent or recieved.
SijsopType should be a unique string for a given Definition. It MUST be a constant string, or sijsop does not guarantee correct functioning.
New should return an empty instance of the same struct, for use in the unmarshaling. It MUST be the same as what is called, or sijsop does not guarantee correct functioning.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
A Reader implements the protocol from the Definition used to create it.
If the io.Reader implements io.Closer, it will be closed if you call .Close on this object.
func (*Reader) Close ¶
Close closes this reader, which means it can no longer be used to receive messages. If the underlying io.Reader implements io.Closer, the io.Reader will have Close called on it as well.
func (*Reader) ReceiveNext ¶
ReceiveNext will receive the next message from the Handler.
If an error is received, the stream is in an unstable condition.
func (*Reader) Unmarshal ¶
Unmarshal takes the given object and attempts to unmarshal the next JSON message into that object. If the types do not match, an ErrWrongType will be returned.
This allows you to receive a concrete type directly when you know what the type will be.
If an error is returned, the stream is now in an unstable condition.
type Writer ¶
type Writer struct {
SizeLimit int
Prefix string
Indent string
// contains filtered or unexported fields
}
A Writer implements the protocol from the Definition used to create it.
If the io.Writer implements io.Closer, it will be closed when this object is closed.
SizeLimit can be set after construction to set a limit on how big a message may be on the way out. If -1, there is no limit. If 0, the default of DefaultSizeLimit is used, of a gigabyte. If a number, anything that size or greater will instead error out.
Prefix and Indent will be passed to SetIndent on the encoder if either is non-empty. This can be convenient during debugging to make the messages more readable.
func (*Writer) Close ¶
Close closes this writer, which means it can no longer be used to send message. If the underlying io.Writer implements io.Closer, the io.Writer will have Close called on it as well.
func (*Writer) Send ¶
Send sends the given JSON message.
This method uses a buffer to generate the message. If you are sending multiple messages, it is somewhat more efficient to send them all as parameters to one Send call, so the same buffer can be used for all of them.
If an error is returned, the stream is now in an unknown condition and can not be trusted for further use.
Source Files