stalecucumber

README

stalecucumber

This package reads and writes pickled data. The format is the same as the Python "pickle" module.

Protocols 0,1,2 are implemented. These are the versions written by the Python 2.x series. Python 3 defines newer protocol versions, but can write the older protocol versions so they are readable by this package.

Full documentation is available here.

TLDR

Read a pickled string or unicode object

pickle.dumps("foobar")
---
var somePickledData io.Reader
mystring, err := stalecucumber.String(stalecucumber.Unpickle(somePickledData))

Read a pickled integer

pickle.dumps(42)
---
var somePickledData io.Reader
myint64, err := stalecucumber.Int(stalecucumber.Unpickle(somePickledData))

Read a pickled list of numbers into a structure

pickle.dumps([8,8,2005])
---
var somePickledData io.Reader
numbers := make([]int64,0)

err := stalecucumber.UnpackInto(&numbers).From(stalecucumber.Unpickle(somePickledData))

Read a pickled dictionary into a structure

pickle.dumps({"apple":1,"banana":2,"cat":"hello","Dog":42.0})
---
var somePickledData io.Reader
mystruct := struct{
	Apple int
	Banana uint
	Cat string
	Dog float32}{}

err := stalecucumber.UnpackInto(&mystruct).From(stalecucumber.Unpickle(somePickledData))

Pickle a struct

buf := new(bytes.Buffer)
mystruct := struct{
		Apple int
		Banana uint
		Cat string
		Dog float32}{}

err := stalecucumber.NewPickler(buf).Pickle(mystruct)

Pickle a python class using the dict reconstructor

buf := new(bytes.Buffer)
obj := stalecucumber.Class{
    			"my.package.path", "CustomClass",
 			stalecucumber.ClassArgs{"foo": "bar", "hello": "world"},
		}
err := stalecucumber.NewPickler(buf).Pickle(obj)

Documentation

Overview

This package reads and writes pickled data. The format is the same as the Python "pickle" module.

Protocols 0,1,2 are implemented. These are the versions written by the Python 2.x series. Python 3 defines newer protocol versions, but can write the older protocol versions so they are readable by this package.

To read data, see stalecucumber.Unpickle.

To write data, see stalecucumber.NewPickler.

TLDR

Read a pickled string or unicode object

pickle.dumps("foobar")
---
var somePickledData io.Reader
mystring, err := stalecucumber.String(stalecucumber.Unpickle(somePickledData))

Read a pickled integer

pickle.dumps(42)
---
var somePickledData io.Reader
myint64, err := stalecucumber.Int(stalecucumber.Unpickle(somePickledData))

Read a pickled list of numbers into a structure

pickle.dumps([8,8,2005])
---
var somePickledData io.Reader
numbers := make([]int64,0)

err := stalecucumber.UnpackInto(&numbers).From(stalecucumber.Unpickle(somePickledData))

Read a pickled dictionary into a structure

pickle.dumps({"apple":1,"banana":2,"cat":"hello","Dog":42.0})
---
var somePickledData io.Reader
mystruct := struct{
	Apple int
	Banana uint
	Cat string
	Dog float32}{}

err := stalecucumber.UnpackInto(&mystruct).From(stalecucumber.Unpickle(somePickledData))

Pickle a struct

buf := new(bytes.Buffer)
mystruct := struct{
		Apple int
		Banana uint
		Cat string
		Dog float32}{}

err := stalecucumber.NewPickler(buf).Pickle(mystruct)

Recursive objects

You can pickle recursive objects like so

a = {}
a["self"] = a
pickle.dumps(a)

Python's pickler is intelligent enough not to emit an infinite data structure when a recursive object is pickled.

I recommend against pickling recursive objects in the first place, but this library handles unpickling them without a problem. The result of unpickling the above is map[interface{}]interface{} with a key "a" that contains a reference to itself.

Attempting to unpack the result of the above python code into a structure with UnpackInto would either fail or recurse forever.

Unpickling Python objects

The Python Pickle module can pickle most Python objects. By default, some Python objects such as the set type and bytearray type are automatically supported by this library.

To support unpickling custom Python objects, you need to implement a resolver. A resolver meets the PythonResolver interface, which is just this function

Resolve(module string, name string, args []interface{})

The module and name are the class name. So if you have a class called "Foo" in the module "bar" the first argument would be "bar" and the second would be "Foo". You can pass in your custom resolver by calling

UnpickleWithResolver(buf, yourCustomResolver)

The third argument of the Resolve function is originally a Python tuple, so it is slice of anything. For most user defined objects this is just a Python dictionary. However, if a Python object implements the __reduce__ method it could be anything.

If your resolver can't identify the type named by module & string, just return stalecucumber.ErrUnresolvablePythonGlobal. Otherwise convert the args into whatever you want and return that as the value from the function with nil for the error.

To avoid reimplementing the same logic over and over, you can chain resolvers together. You can use your resolver in addition to the default resolver by doing the following

UnpickleWithResolver(reader, MakePythonResolverChain(customResolver, PythonBuiltinResolver{}))

Protocol Performance

If the version of Python you are using supports protocol version 1 or 2, you should always specify that protocol version. By default the "pickle" and "cPickle" modules in Python write using protocol 0. Protocol 0 requires much more space to represent the same values and is much slower to parse.

Unsupported Opcodes

The pickle format is incredibly flexible and as a result has some features that are impractical or unimportant when implementing a reader in another language.

Each set of opcodes is listed below by protocol version with the impact.

Protocol 0

PERSID

This opcode is used to reference concrete definitions of objects between a pickler and an unpickler by an ID number. The pickle protocol doesn't define what a persistent ID means.

This opcode is unlikely to ever be supported by this package.

Protocol 1

OBJ

This opcodes is used in recreating pickled python objects. That is currently not supported by this package.

This opcode will supported in a future revision to this package that allows the unpickling of instances of Python classes.

BINPERSID

This opcode is equivalent to PERSID in protocol 0 and won't be supported for the same reason.

Protocol 2

NEWOBJ

This opcodes is used in recreating pickled python objects. That is currently not supported by this package.

This opcode will supported in a future revision to this package that allows the unpickling of instances of Python classes.

EXT1
EXT2
EXT4

These opcodes allow using a registry of popular objects that are pickled by name, typically classes. It is envisioned that through a global negotiation and registration process, third parties can set up a mapping between ints and object names.

These opcodes are unlikely to ever be supported by this package.

Index

• Constants
• Variables
• func Big(v interface{}, err error) (*big.Int, error)
• func Bool(v interface{}, err error) (bool, error)
• func Dict(v interface{}, err error) (map[interface{}]interface{}, error)
• func DictString(v interface{}, err error) (map[string]interface{}, error)
• func Float(v interface{}, err error) (float64, error)
• func Int(v interface{}, err error) (int64, error)
• func ListOrTuple(v interface{}, err error) ([]interface{}, error)
• func Set(v interface{}, err error) (map[interface{}]bool, error)
• func String(v interface{}, err error) (string, error)
• func UnpackInto(dest interface{}) unpacker
• func Unpickle(reader io.Reader) (interface{}, error)
• func UnpickleWithResolver(reader io.Reader, resolver PythonResolver) (interface{}, error)
• type Class
• type ClassArgs
• type PickleMachine
  • func (pm *PickleMachine) Opcode_Invalid() error
• type PickleMachineError
  • func (pme PickleMachineError) Error() string
• type PickleMark
  • func (_ PickleMark) String() string
• type PickleNone
  • func (_ PickleNone) String() string
• type PickleTuple
  • func NewTuple(v ...interface{}) PickleTuple
• type Pickler
  • func NewPickler(writer io.Writer) *Pickler
  • func (p *Pickler) Pickle(v interface{}) (int, error)
• type PicklingError
  • func (pe PicklingError) Error() string
• type PythonBuiltinResolver
  • func (this PythonBuiltinResolver) Resolve(module string, name string, args []interface{}) (interface{}, error)
• type PythonResolver
• type PythonResolverChain
  • func MakePythonResolverChain(args ...PythonResolver) PythonResolverChain
  • func (this PythonResolverChain) Resolve(module string, name string, args []interface{}) (interface{}, error)
• type UnbuildableValueError
  • func (this UnbuildableValueError) Error() string
• type UnpackingError
  • func (ue UnpackingError) Error() string
• type UnparseablePythonGlobalError
  • func (this UnparseablePythonGlobalError) Error() string
• type UnreducibleValueError
  • func (this UnreducibleValueError) Error() string
• type WrongTypeError
  • func (wte WrongTypeError) Error() string

Constants

const BININT_MAX = (1 << 31) - 1

const BININT_MIN = 0 - BININT_MAX

const OPCODE_APPEND = 0x61

const OPCODE_APPENDS = 0x65

const OPCODE_BINFLOAT = 0x47

const OPCODE_BINGET = 0x68

const OPCODE_BININT = 0x4a

const OPCODE_BININT1 = 0x4b

const OPCODE_BININT2 = 0x4d

const OPCODE_BINPERSID = 0x51

const OPCODE_BINPUT = 0x71

const OPCODE_BINSTRING = 0x54

const OPCODE_BINUNICODE = 0x58

const OPCODE_BUILD = 0x62

const OPCODE_DICT = 0x64

const OPCODE_DUP = 0x32

const OPCODE_EMPTY_DICT = 0x7d

const OPCODE_EMPTY_LIST = 0x5d

const OPCODE_EMPTY_TUPLE = 0x29

const OPCODE_EXT1 = 0x82

const OPCODE_EXT2 = 0x83

const OPCODE_EXT4 = 0x84

const OPCODE_FLOAT = 0x46

const OPCODE_GET = 0x67

const OPCODE_GLOBAL = 0x63

const OPCODE_INST = 0x69

const OPCODE_INT = 0x49

const OPCODE_LIST = 0x6c

const OPCODE_LONG = 0x4c

const OPCODE_LONG1 = 0x8a

const OPCODE_LONG4 = 0x8b

const OPCODE_LONG_BINGET = 0x6a

const OPCODE_LONG_BINPUT = 0x72

const OPCODE_MARK = 0x28

const OPCODE_NEWFALSE = 0x89

const OPCODE_NEWOBJ = 0x81

const OPCODE_NEWTRUE = 0x88

const OPCODE_NONE = 0x4e

const OPCODE_OBJ = 0x6f

const OPCODE_PERSID = 0x50

const OPCODE_POP = 0x30

const OPCODE_POP_MARK = 0x31

const OPCODE_PROTO = 0x80

const OPCODE_PUT = 0x70

const OPCODE_REDUCE = 0x52

const OPCODE_SETITEM = 0x73

const OPCODE_SETITEMS = 0x75

const OPCODE_SHORT_BINSTRING = 0x55

const OPCODE_STOP = 0x2e

const OPCODE_STRING = 0x53

const OPCODE_TUPLE = 0x74

const OPCODE_TUPLE1 = 0x85

const OPCODE_TUPLE2 = 0x86

const OPCODE_TUPLE3 = 0x87

const OPCODE_UNICODE = 0x56

const PICKLE_TAG = "pickle"

Variables

var ErrEmptyInterfaceNotPickleable = errors.New("The empty interface is not pickleable")

var ErrInputTruncated = errors.New("Input to the pickle machine was truncated")

var ErrMarkNotFound = errors.New("Mark could not be found on the stack")

var ErrNilPointer = errors.New("Destination cannot be a nil pointer")

var ErrNoResult = errors.New("Input did not place a value onto the stack")

var ErrNotPointer = errors.New("Destination must be a pointer type")

var ErrOpcodeInvalid = errors.New("Opcode is invalid")

var ErrOpcodeNotImplemented = errors.New("Input encountered opcode that is not implemented")

var ErrOpcodeStopped = errors.New("STOP opcode found")

var ErrStackTooSmall = errors.New("Stack is too small to perform requested operation")

var ErrTargetTypeMismatch = errors.New("Target type does not match source type")

var ErrTargetTypeNotPointer = errors.New("Target type must be a pointer to unpack this value")

var ErrTargetTypeOverflow = errors.New("Value overflows target type")

var ErrTypeNotPickleable = errors.New("Can't pickle this type")

var ErrUnresolvablePythonGlobal = errors.New("Unresolvable Python global value")

Functions

func Big

func Big(v interface{}, err error) (*big.Int, error)

This helper attempts to convert the return value of Unpickle into a *big.Int.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Bool

func Bool(v interface{}, err error) (bool, error)

This helper attempts to convert the return value of Unpickle into a bool.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Dict

func Dict(v interface{}, err error) (map[interface{}]interface{}, error)

This helper attempts to convert the return value of Unpickle into a map[interface{}]interface{}.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func DictString

func DictString(v interface{}, err error) (map[string]interface{}, error)

This helper attempts to convert the return value of Unpickle into a map[string]interface{}.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Float

func Float(v interface{}, err error) (float64, error)

This helper attempts to convert the return value of Unpickle into a float64.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Int

func Int(v interface{}, err error) (int64, error)

This helper attempts to convert the return value of Unpickle into a int64.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func ListOrTuple

func ListOrTuple(v interface{}, err error) ([]interface{}, error)

This helper attempts to convert the return value of Unpickle into a []interface{}.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Set

func Set(v interface{}, err error) (map[interface{}]bool, error)

This helper attempts to convert the return value of Unpickle into a map[interface{}]bool.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func String

func String(v interface{}, err error) (string, error)

This helper attempts to convert the return value of Unpickle into a string.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func UnpackInto

func UnpackInto(dest interface{}) unpacker

func Unpickle

func Unpickle(reader io.Reader) (interface{}, error)

Unpickle a value from a reader. This function takes a reader and attempts to read a complete pickle program from it. This is normally the output of the function "pickle.dump" from Python.

The returned type is interface{} because unpickling can generate any type. Use a helper function to convert to another type without an additional type check.

This function returns an error if the reader fails, the pickled data is invalid, or if the pickled data contains an unsupported opcode. See unsupported opcodes in the documentation of this package for more information.

Type Conversions

Types conversion Python types to Go types is performed as followed

int -> int64
string -> string
unicode -> string
float -> float64
long -> big.Int from the "math/big" package
lists -> []interface{}
tuples -> []interface{}
dict -> map[interface{}]interface{}

The following values are converted from Python to the Go types

True & False -> bool
None -> stalecucumber.PickleNone, sets pointers to nil

Helper Functions

The following helper functions were inspired by the github.com/garyburd/redigo package. Each function takes the result of Unpickle as its arguments. If unpickle fails it does nothing and returns that error. Otherwise it attempts to convert to the appropriate type. If type conversion fails it returns an error

String - string from Python string or unicode
Int - int64 from Python int or long
Bool - bool from Python True or False
Big - *big.Int from Python long
ListOrTuple - []interface{} from Python Tuple or List
Float - float64 from Python float
Dict - map[interface{}]interface{} from Python dictionary
DictString -
	map[string]interface{} from Python dictionary.
	Keys must all be of type unicode or string.

Unpacking into structures

If the pickled object is a python dictionary that has only unicode and string objects for keys, that object can be unpickled into a struct in Go by using the "UnpackInto" function. The "From" receiver on the return value accepts the result of "Unpickle" as its actual parameters.

The keys of the python dictionary are assigned to fields in a structure. Structures may specify the tag "pickle" on fields. The value of this tag is taken as the key name of the Python dictionary value to place in this field. If no field has a matching "pickle" tag the fields are looked up by name. If the first character of the key is not uppercase, it is uppercased. If a field matching that name is found, the value in the python dictionary is unpacked into the value of the field within the structure.

A list of python dictionaries can be unpickled into a slice of structures in Go.

A homogeneous list of python values can be unpickled into a slice in Go with the appropriate element type.

A nested python dictionary is unpickled into nested structures in Go. If a field is of type map[interface{}]interface{} it is of course unpacked into that as well.

By default UnpackInto skips any missing fields and fails if a field's type is not compatible with the object's type.

This behavior can be changed by setting "AllowMissingFields" and "AllowMismatchedFields" on the return value of UnpackInto before calling From.

func UnpickleWithResolver

func UnpickleWithResolver(reader io.Reader, resolver PythonResolver) (interface{}, error)

Types

type Class

type Class struct {
	Package string
	Name    string
	Args    ClassArgs
}

type ClassArgs

type ClassArgs map[string]interface{}

type PickleMachine

type PickleMachine struct {
	Stack  []interface{}
	Memo   []interface{}
	Reader io.Reader
	// contains filtered or unexported fields
}

This struct is current exposed but not useful. It is likely to be hidden in the near future.

func (*PickleMachine) Opcode_Invalid

func (pm *PickleMachine) Opcode_Invalid() error

type PickleMachineError

type PickleMachineError struct {
	Err       error
	StackSize int
	MemoSize  int
	Opcode    uint8
}

This type is returned whenever Unpickle encounters an error in pickled data.

func (PickleMachineError) Error

func (pme PickleMachineError) Error() string

type PickleMark

type PickleMark struct{}

This type is used internally to represent a concept known as a mark on the Pickle Machine's stack. Oddly formed pickled data could return this value as the result of Unpickle. In normal usage this type is needed only internally.

func (PickleMark) String

func (_ PickleMark) String() string

type PickleNone

type PickleNone struct{}

This type is used to represent the Python object "None"

func (PickleNone) String

func (_ PickleNone) String() string

type PickleTuple

type PickleTuple []interface{}

func NewTuple

func NewTuple(v ...interface{}) PickleTuple

type Pickler

type Pickler struct {
	W io.Writer
	// contains filtered or unexported fields
}

func NewPickler

func NewPickler(writer io.Writer) *Pickler

This type is used to pickle data.Picklers are created by calling NewPickler. Each call to Pickle writes a complete pickle program to the underlying io.Writer object.

Its safe to assign W to other values in between calls to Pickle.

Failures return the underlying error or an instance of PicklingError.

Data is always written using Pickle Protocol 2. This format is compatible with Python 2.3 and all newer version.

Type Conversions

Type conversion from Go types to Python types is as follows

uint8,uint16,int8,int16,int32 -> Python int
int,int64,uint,uint64 -> Python int if it fits, otherwise Python Long
string -> Python unicode
slices, arrays -> Python list
maps -> Python dict
bool -> Python True and False
big.Int -> Python Long
struct -> Python dict

Structs are pickled using their field names unless a tag is present on the field specifying the name. For example

type MyType struct {
	FirstField int
	SecondField int `pickle:"meow"`
}

This struct would be pickled into a dictionary with two keys: "FirstField" and "meow".

Embedded structs are marshalled as a nested dictionary. Exported types are never pickled.

Pickling Tuples

There is no equivalent type to Python's tuples in Go. You may not need to use tuples at all. For example, consider the following Python code

a, b, c = pickle.load(data_in)

This code tries to set to the variables "a", "b", and "c" from the result of unpickling. In this case it does not matter if the source type is a Python list or a Python tuple.

If you really need to write tuples, call NewTuple and pass the data in as the arguments. This special type exists to inform stalecucumber.Pickle that a tuple should be pickled.

func (*Pickler) Pickle

func (p *Pickler) Pickle(v interface{}) (int, error)

type PicklingError

type PicklingError struct {
	V   interface{}
	Err error
}

func (PicklingError) Error

func (pe PicklingError) Error() string

type PythonBuiltinResolver

type PythonBuiltinResolver struct{}

func (PythonBuiltinResolver) Resolve

func (this PythonBuiltinResolver) Resolve(module string, name string, args []interface{}) (interface{}, error)

type PythonResolver

type PythonResolver interface {
	Resolve(module string, name string, args []interface{}) (interface{}, error)
}

A type to convert to a GLOBAL opcode to something meaningful in golang

type PythonResolverChain

type PythonResolverChain []PythonResolver

func MakePythonResolverChain

func MakePythonResolverChain(args ...PythonResolver) PythonResolverChain

func (PythonResolverChain) Resolve

func (this PythonResolverChain) Resolve(module string, name string, args []interface{}) (interface{}, error)

type UnbuildableValueError

type UnbuildableValueError struct {
	Value interface{}
}

func (UnbuildableValueError) Error

func (this UnbuildableValueError) Error() string

type UnpackingError

type UnpackingError struct {
	Source      interface{}
	Destination reflect.Value
	Err         error
}

func (UnpackingError) Error

func (ue UnpackingError) Error() string

This type is returned when a call to From() fails. Setting "AllowMissingFields" and "AllowMismatchedFields" on the result of "UnpackInto" controls if this error is returned or not.

type UnparseablePythonGlobalError

type UnparseablePythonGlobalError struct {
	Args    interface{}
	Message string
}

func (UnparseablePythonGlobalError) Error

func (this UnparseablePythonGlobalError) Error() string

type UnreducibleValueError

type UnreducibleValueError struct {
	Value interface{}
}

func (UnreducibleValueError) Error

func (this UnreducibleValueError) Error() string

type WrongTypeError

type WrongTypeError struct {
	Result  interface{}
	Request string
}

This type is returned whenever a helper cannot convert the result of Unpickle into the desired type.

func (WrongTypeError) Error

func (wte WrongTypeError) Error() string