Documentation

Overview

    Package serialize provides methods for reading and writing concatenable, bytewise-sortable forms of the datatypes defined in the datastore package.

    Index

    Constants

    View Source
    const MaxIndexColumns = 64

      MaxIndexColumns is the maximum number of sort columns (e.g. sort orders) that ReadIndexDefinition is willing to deserialize. 64 was chosen as a likely-astronomical number.

      View Source
      const ReadKeyNumToksReasonableLimit = 50

        ReadKeyNumToksReasonableLimit is the maximum number of Key tokens that ReadKey is willing to read for a single key.

        View Source
        const ReadPropertyMapReasonableLimit uint64 = 30000

          ReadPropertyMapReasonableLimit sets a limit on the number of rows and number of properties per row which can be read by ReadPropertyMap. The total number of Property objects readable by this method is this number squared (e.g. Limit rows * Limit properties)

          Variables

          View Source
          var WritePropertyMapDeterministic = false

            WritePropertyMapDeterministic allows tests to make WritePropertyMap deterministic.

            Functions

            func Increment

            func Increment(bstr []byte) ([]byte, bool)

              Increment attempts to increment a copy of bstr as if adding 1 to an integer.

              If it overflows, the returned []byte will be all 0's, and the overflow bool will be true.

              func Invert

              func Invert(bs []byte) []byte

                Invert simply inverts all the bytes in bs.

                func Join

                func Join(itms ...[]byte) []byte

                  Join is a convenience invocation of bytes.Join(itms, nil)

                  func ReadGeoPoint

                  func ReadGeoPoint(buf ReadBuffer) (gp ds.GeoPoint, err error)

                    ReadGeoPoint reads a GeoPoint from the buffer.

                    func ReadIndexColumn

                    func ReadIndexColumn(buf ReadBuffer) (c ds.IndexColumn, err error)

                      ReadIndexColumn reads an IndexColumn from the buffer.

                      func ReadIndexDefinition

                      func ReadIndexDefinition(buf ReadBuffer) (i ds.IndexDefinition, err error)

                        ReadIndexDefinition reads an IndexDefinition from the buffer.

                        func ReadKey

                        func ReadKey(buf ReadBuffer, context KeyContext, inKC ds.KeyContext) (ret *ds.Key, err error)

                          ReadKey deserializes a key from the buffer. The value of context must match the value of context that was passed to WriteKey when the key was encoded. If context == WithoutContext, then the appid and namespace parameters are used in the decoded Key. Otherwise they're ignored.

                          func ReadKeyTok

                          func ReadKeyTok(buf ReadBuffer) (ret ds.KeyTok, err error)

                            ReadKeyTok reads a KeyTok from the buffer. You usually want ReadKey instead of this.

                            func ReadProperty

                            func ReadProperty(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (p ds.Property, err error)

                              ReadProperty reads a Property from the buffer. `context` and `kc` behave the same way they do for ReadKey, but only have an effect if the decoded property has a Key value.

                              func ReadPropertyMap

                              func ReadPropertyMap(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (pm ds.PropertyMap, err error)

                                ReadPropertyMap reads a PropertyMap from the buffer. `context` and friends behave the same way that they do for ReadKey.

                                func ReadTime

                                func ReadTime(buf ReadBuffer) (time.Time, error)

                                  ReadTime reads a time.Time from the buffer.

                                  func ToBytes

                                  func ToBytes(i interface{}) []byte

                                    ToBytes serializes i to a byte slice, if it's one of the type supported by this library. If an error is encountered (e.g. `i` is not a supported type), this method panics.

                                    Key types will be serialized using the 'WithoutContext' option (e.g. their encoded forms will not contain AppID or Namespace).

                                    func ToBytesErr

                                    func ToBytesErr(i interface{}) ([]byte, error)

                                      ToBytesErr serializes i to a byte slice, if it's one of the type supported by this library, otherwise it returns an error.

                                      Key types will be serialized using the 'WithoutContext' option (e.g. their encoded forms will not contain AppID or Namespace).

                                      func ToBytesWithContext

                                      func ToBytesWithContext(i interface{}) []byte

                                        ToBytesWithContext serializes i to a byte slice, if it's one of the type supported by this library. If an error is encountered (e.g. `i` is not a supported type), this method panics.

                                        Key types will be serialized using the 'WithContext' option (e.g. their encoded forms will not contain AppID or Namespace).

                                        func ToBytesWithContextErr

                                        func ToBytesWithContextErr(i interface{}) ([]byte, error)

                                          ToBytesWithContextErr serializes i to a byte slice, if it's one of the type supported by this library, otherwise it returns an error.

                                          Key types will be serialized using the 'WithContext' option (e.g. their encoded forms will contain AppID and Namespace).

                                          func WriteGeoPoint

                                          func WriteGeoPoint(buf WriteBuffer, gp ds.GeoPoint) (err error)

                                            WriteGeoPoint writes a GeoPoint to the buffer.

                                            func WriteIndexColumn

                                            func WriteIndexColumn(buf WriteBuffer, c ds.IndexColumn) (err error)

                                              WriteIndexColumn writes an IndexColumn to the buffer.

                                              func WriteIndexDefinition

                                              func WriteIndexDefinition(buf WriteBuffer, i ds.IndexDefinition) (err error)

                                                WriteIndexDefinition writes an IndexDefinition to the buffer

                                                func WriteIndexProperty

                                                func WriteIndexProperty(buf WriteBuffer, context KeyContext, p ds.Property) error

                                                  WriteIndexProperty writes a Property to the buffer as its native index type. `context` behaves the same way that it does for WriteKey, but only has an effect if `p` contains a Key as its IndexValue.

                                                  func WriteKey

                                                  func WriteKey(buf WriteBuffer, context KeyContext, k *ds.Key) (err error)

                                                    WriteKey encodes a key to the buffer. If context is WithContext, then this encoded value will include the appid and namespace of the key.

                                                    func WriteKeyTok

                                                    func WriteKeyTok(buf WriteBuffer, tok ds.KeyTok) (err error)

                                                      WriteKeyTok writes a KeyTok to the buffer. You usually want WriteKey instead of this.

                                                      func WriteProperty

                                                      func WriteProperty(buf WriteBuffer, context KeyContext, p ds.Property) error

                                                        WriteProperty writes a Property to the buffer. `context` behaves the same way that it does for WriteKey, but only has an effect if `p` contains a Key as its IndexValue.

                                                        func WritePropertyMap

                                                        func WritePropertyMap(buf WriteBuffer, context KeyContext, pm ds.PropertyMap) (err error)

                                                          WritePropertyMap writes an entire PropertyMap to the buffer. `context` behaves the same way that it does for WriteKey.

                                                          If WritePropertyMapDeterministic is true, then the rows will be sorted by property name before they're serialized to buf (mostly useful for testing, but also potentially useful if you need to make a hash of the property data).

                                                          Write skips metadata keys.

                                                          func WriteTime

                                                          func WriteTime(buf WriteBuffer, t time.Time) error

                                                            WriteTime writes a time.Time to the buffer.

                                                            The supplied time is rounded via datastore.RoundTime and written as a microseconds-since-epoch integer to comform to datastore storage standards.

                                                            Types

                                                            type InvertibleBuffer

                                                            type InvertibleBuffer interface {
                                                            	WriteBuffer
                                                            	SetInvert(inverted bool)
                                                            }

                                                              InvertibleBuffer is just like Buffer, except that it also has a stateful Invert() method, which will cause all reads and writes to/from it to be inverted (e.g. every byte XOR 0xFF).

                                                              Implementing queries requires manipulating the index entries (e.g. synthesizing them, parsing them, etc.). In particular, when you have a reverse-sorted field (e.g. high to low instead of low to high), it's achieved by having all the bits inverted.

                                                              All the serialization formats include delimiter information, which the parsers only know to parse non-inverted. If we don't have this buffer, we'd basically have to invert every byte in the []byte array when we're trying to decode a reverse-ordered field (including the bytes of all fields after the one we intend to parse) so that the parser can consume as many bytes as it needs (and it only knows the number of bytes it needs as it decodes them). This InvertibleBuffer lets that happen on the fly without having to flip the whole []byte.

                                                              If you know you need it, you'll know it's the right thing. If you're not sure then you definitely don't need it!

                                                              func Invertible

                                                              func Invertible(b WriteBuffer) InvertibleBuffer

                                                                Invertible returns an InvertibleBuffer based on the Buffer.

                                                                type KeyContext

                                                                type KeyContext bool

                                                                  KeyContext controls whether the various Write and Read serializtion routines should encode the context of Keys (read: the appid and namespace). Frequently the appid and namespace of keys are known in advance and so there's no reason to redundantly encode them.

                                                                  const (
                                                                  	WithContext    KeyContext = true
                                                                  	WithoutContext            = false
                                                                  )

                                                                    With- and WithoutContext indicate if the serialization method should include context for Keys. See KeyContext for more information.

                                                                    type ReadBuffer

                                                                    type ReadBuffer interface {
                                                                    	Len() int
                                                                    
                                                                    	Read([]byte) (int, error)
                                                                    	ReadByte() (byte, error)
                                                                    }

                                                                      ReadBuffer is the interface which corresponds to the subset of *bytes.Reader that this package requires.

                                                                      type SerializedPmap

                                                                      type SerializedPmap map[string]SerializedPslice

                                                                        SerializedPmap maps from

                                                                        prop name -> [<serialized DSProperty>, ...]
                                                                        

                                                                        includes special values '__key__' and '__ancestor__' which contains all of the ancestor entries for this key.

                                                                        func PropertyMapPartially

                                                                        func PropertyMapPartially(k *ds.Key, pm ds.PropertyMap) (ret SerializedPmap)

                                                                          PropertyMapPartially turns a regular PropertyMap into a SerializedPmap. Essentially all the []Property's become SerializedPslice, using cmpbin and datastore/serialize's encodings.

                                                                          type SerializedPslice

                                                                          type SerializedPslice [][]byte

                                                                            SerializedPslice is all of the serialized DSProperty values in ASC order.

                                                                            func PropertySlice

                                                                            func PropertySlice(vals ds.PropertySlice) SerializedPslice

                                                                              PropertySlice serializes a single row of a DSProperty map.

                                                                              It does not differentiate between single- and multi- properties.

                                                                              func (SerializedPslice) Len

                                                                              func (s SerializedPslice) Len() int

                                                                              func (SerializedPslice) Less

                                                                              func (s SerializedPslice) Less(i, j int) bool

                                                                              func (SerializedPslice) Swap

                                                                              func (s SerializedPslice) Swap(i, j int)

                                                                              type WriteBuffer

                                                                              type WriteBuffer interface {
                                                                              	ReadBuffer
                                                                              
                                                                              	String() string
                                                                              	Bytes() []byte
                                                                              
                                                                              	Grow(int)
                                                                              
                                                                              	Write([]byte) (int, error)
                                                                              	WriteByte(c byte) error
                                                                              	WriteString(s string) (int, error)
                                                                              }

                                                                                WriteBuffer is the interface which corresponds to the subset of *bytes.Buffer that this package requires.