sophie

package module
Version: v0.0.0-...-731618e Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 13, 2016 License: BSD-2-Clause Imports: 9 Imported by: 58

README

sophie GoSearch

A sequencial data processing library.

Documentation

Overview

Package sophie provides an raw mechanism for serializing data.

It aims at more efficiency than other serialization methods because of the following reasons:

* Maximum of reusing objects, allocation and GC are avoided * No reflections

Since the serialization is flexible, one can also make some trade-offs between efficiency and convinience. E.g., if the data structure may be changed in the future, in the ReadFrom/WriteTo, God codec can be used to provide future compatibility.

Sub packages:

mr  MapReduce library
kv  A file format storing key-value pairs.

Index

Constants

View Source
const UNKNOWN_LEN = -1

The constants for unknown length. @see SophieReader.ReadFrom

Variables

View Source
var (
	// Returned if some Sophie file is found to have bad format.
	ErrBadFormat = errors.New("Bad Sophie format")
)
View Source
var NullCollectCloser = nullCollectCloser{}

A helper variable with a CollectCloser ignoring every thing collecting to it

Functions

func ReadStringSlice

func ReadStringSlice(r Reader, sl *[]string) (err error)

A helper function that reads a slice String from a Reader. The data was serialized by WriteStringSlice function.

func WriteStringSlice

func WriteStringSlice(w Writer, sl []string) error

A helper function that writes a slice of Strings to a Writer. Serialized data can be read by ReadStringSlice function.

Types

type BufferedFileReader

type BufferedFileReader struct {
	*bufio.Reader
	// contains filtered or unexported fields
}

BufferedFileReader is a sophie.ReadCloser with buffer

func (BufferedFileReader) Close

func (b BufferedFileReader) Close() error

sophie.ReadCloser interface

func (BufferedFileReader) Skip

func (b BufferedFileReader) Skip(n int64) (int64, error)

sophie.Reader interface

type BufferedFileWriter

type BufferedFileWriter struct {
	*bufio.Writer
	// contains filtered or unexported fields
}

BufferedFileWriter is a sophie.WriteCloser with buffer.

func (BufferedFileWriter) Close

func (b BufferedFileWriter) Close() error

sophie.WriteCloser interface

type ByteSlice

type ByteSlice []byte

*ByteSlice implements Sophier interface.

func (*ByteSlice) ReadFrom

func (ba *ByteSlice) ReadFrom(r Reader, l int) error

SophieReader interface

func (ByteSlice) WriteTo

func (ba ByteSlice) WriteTo(w Writer) error

SophieWriter interface

type CollectCloser

type CollectCloser interface {
	Collector
	io.Closer
}

CollectCloser is Collector + io.Closer

type CollectCloserStruct

type CollectCloserStruct struct {
	CollectF func(SophieWriter, SophieWriter) error
	CloseF   func() error
}

CollectCloserStruct is a struct whose pointer implements CollectCloser interface

func (*CollectCloserStruct) Close

func (c *CollectCloserStruct) Close() error

func (*CollectCloserStruct) Collect

func (c *CollectCloserStruct) Collect(k, v SophieWriter) error

type Collector

type Collector interface {
	Collect(key, val SophieWriter) error
}

Collector is an interface for collecting Sophie key-value pairs.

type CollectorF

type CollectorF func(key, val SophieWriter) error

A func type implementing Collector interface. A nil func is a no-op Collector.

func (CollectorF) Collect

func (c CollectorF) Collect(key, val SophieWriter) error

Collector interface.

type EmptyClose

type EmptyClose struct{}

EmptyClose is a helper type defining an empty Close method

func (EmptyClose) Close

func (EmptyClose) Close() error

io.Closer interface

type FileSystem

type FileSystem interface {
	// Create creates a file of a specified name.
	Create(fn string) (WriteCloser, error)
	// Mkdir makes the directory and its parents if necessary of specifiy path
	// and perm.
	Mkdir(path string, perm os.FileMode) error
	// Open opens a ReadCloser for reading the file of a specified name.
	Open(fn string) (ReadCloser, error)
	// ReadDir reads the FileInfos of the files under a specified directory.
	ReadDir(dir string) ([]os.FileInfo, error)
	// Stat returns the FileInfo of a file/directory of a specified name.
	Stat(fn string) (os.FileInfo, error)
	// Remove deletes a file or a directory(and all its file/directories in it)
	Remove(fn string) error
}

An interface defining some actions an file-system should have.

var (
	// LocalFS is a FileSystem representing the local file-system.
	LocalFS FileSystem = localFileSystem{}
)

type FsPath

type FsPath struct {
	// The FileSystem
	Fs FileSystem
	// The path
	Path string
}

FsPath is a pair of FileSystem and a path

func LocalFsPath

func LocalFsPath(path string) FsPath

LocalFsPath returns an FsPath of the LocalFS and the specified path

func TempDirPath

func TempDirPath() FsPath

TempDirPath returns the OS temporary dir as the fs path.

func (FsPath) Create

func (fp FsPath) Create() (WriteCloser, error)

Calls FileSystem.Create with the path

func (FsPath) Join

func (fp FsPath) Join(sub string) FsPath

Join returns a new FsPath with the same FileSystem and the path joined with sub

func (FsPath) Mkdir

func (fp FsPath) Mkdir(perm os.FileMode) error

Calls FileSystem.Mkdir with the path

func (FsPath) Open

func (fp FsPath) Open() (ReadCloser, error)

Calls FileSystem.Open with the path

func (FsPath) ReadDir

func (fp FsPath) ReadDir() ([]os.FileInfo, error)

Calls FileSystem.ReadDir with the path

func (FsPath) Remove

func (fp FsPath) Remove() error

Calls FileSystem.Remove with the path

func (FsPath) Stat

func (fp FsPath) Stat() (os.FileInfo, error)

Calls FileSystem.Stat with the path

type Int32

type Int32 int32

*Int32 implements Sophie interface

func (*Int32) ReadFrom

func (i *Int32) ReadFrom(r Reader, l int) error

SophieReader interface

func (*Int32) Val

func (i *Int32) Val() int32

func (Int32) WriteTo

func (i Int32) WriteTo(w Writer) error

SophieWriter interface

type IterateCloser

type IterateCloser interface {
	Iterator
	io.Closer
}

sohpie.IterateCloser is Iterator + io.Closer

type IterateCloserStruct

type IterateCloserStruct struct {
	NextF   func(key, val SophieReader) error
	CloserF func() error
}

A struct implementing IterateCloser by funcs.

func (*IterateCloserStruct) Close

func (ics *IterateCloserStruct) Close() error

io.Closer struct.

func (*IterateCloserStruct) Next

func (ics *IterateCloserStruct) Next(key, val SophieReader) error

Iterator interface

type Iterator

type Iterator interface {
	Next(key, val SophieReader) error
}

Iterator is an interface for iterating Sophier kv pairs.

type Null

type Null struct{}

Null is an empty data structure implementing Sophie interface.

var NULL Null = Null{}

NULL is a variable of type Null.

func (Null) ReadFrom

func (Null) ReadFrom(r Reader, l int) error

SophieReader interface

func (Null) WriteTo

func (Null) WriteTo(w Writer) error

SophieWriter interface

type RawByteSlice

type RawByteSlice []byte

*RawByteSlice implements Sophier interface. It encodes byte-slice assuming the length of buffer will be known when decoding.

func (*RawByteSlice) ReadFrom

func (ba *RawByteSlice) ReadFrom(r Reader, sz int) error

SophieReader interface

func (RawByteSlice) WriteTo

func (ba RawByteSlice) WriteTo(w Writer) error

SophieWriter interface

type RawString

type RawString string

*RawString implements Sophie interface. It assumes the length to be known.

func (*RawString) ReadFrom

func (s *RawString) ReadFrom(r Reader, l int) error

SophieReader interface

func (*RawString) String

func (s *RawString) String() string

func (*RawString) Val

func (s *RawString) Val() string

func (RawString) WriteTo

func (s RawString) WriteTo(w Writer) error

SophieWriter interface.

type RawVInt

type RawVInt int

*RawVInt implements Sophie interface and serializing as a vint. It assumes the length to be known.

func (*RawVInt) ReadFrom

func (i *RawVInt) ReadFrom(r Reader, l int) error

SophieReader interface

func (*RawVInt) String

func (i *RawVInt) String() string

func (*RawVInt) Val

func (i *RawVInt) Val() int

func (RawVInt) WriteTo

func (i RawVInt) WriteTo(w Writer) error

SophieWriter interface

type ReadCloser

type ReadCloser interface {
	Reader
	io.Closer
}

sophie.ReadCloser is sohpie.Reader + io.Closer

type Reader

type Reader interface {
	io.Reader
	io.ByteReader
	// Skip skips n bytes, returns the number of actually skipped
	Skip(n int64) (int64, error)
}

sophie.Reader is an interface extended from io.Reader + io.ByteReader

type SophieReader

type SophieReader interface {
	// ReadFrom reads fields from a Reader. If l is not UNKNOWN_LEN, it can be
	// used to determine the border of the serialized data.
	// @param l  the number of bytes to read. UNKNOWN_LEN(-1) means unknown
	//           length
	ReadFrom(r Reader, l int) error
}

SophieReader is the interface for some data structure that can reads fields from a Reader. The data in the Reader could have known length. For all predefined Sophies with prefix Raw, the length must be specified.

type SophieWriter

type SophieWriter interface {
	// WriteTo writes fields to the Writer
	WriteTo(w Writer) error
}

SophieWriter is the interface for some data structure that can write fields to a Writer.

type Sophier

type Sophier interface {
	SophieReader
	SophieWriter
}

Sophier is a basic data structure for serialization. It is SophieReader + SophieWriter

func NewByteSlice

func NewByteSlice() Sophier

Returns a new instance of *ByteSlice as a Sophier

func NewInt32

func NewInt32() Sophier

Returns a new instance of *Int32 as a Sophier

func NewRawByteSlice

func NewRawByteSlice() Sophier

Returns a new instance of *RawByteSlice as a Sophier

func NewRawString

func NewRawString() Sophier

Returns a new instance of *RawByteSlice as a Sophier

func NewRawVInt

func NewRawVInt() Sophier

Returns a new instance of *RawVInt as a Sophier

func NewString

func NewString() Sophier

Returns a new instance of *String as a Sophier

func NewTime

func NewTime() Sophier

Returns a new instance of *Time as a Sophier

func NewVInt

func NewVInt() Sophier

Returns a new instace of *VInt as a Sophier

func ReturnNULL

func ReturnNULL() Sophier

type String

type String string

*String implements Sophie interface

func ReadString

func ReadString(r Reader) (String, error)

A helper function that reads a String from a Reader.

func (*String) ReadFrom

func (s *String) ReadFrom(r Reader, l int) error

SophieReader interface

func (*String) String

func (s *String) String() string

func (*String) Val

func (s *String) Val() string

func (String) WriteTo

func (s String) WriteTo(w Writer) error

SophieWriter interface

type Time

type Time time.Time

Time is a time.Time, and *Time implements Sophie interface.

func (*Time) ReadFrom

func (t *Time) ReadFrom(r Reader, l int) error

SophieReader interface

func (Time) WriteTo

func (t Time) WriteTo(w Writer) error

SophieWriter interface

type VInt

type VInt int

*VInt implements Sophie interface and serializing as a vint

func (*VInt) ReadFrom

func (i *VInt) ReadFrom(r Reader, l int) error

SophieReader interface

func (*VInt) String

func (i *VInt) String() string

func (*VInt) Val

func (i *VInt) Val() int

func (VInt) WriteTo

func (i VInt) WriteTo(w Writer) error

SophieWriter interface

type WriteCloser

type WriteCloser interface {
	Writer
	io.Closer
}

sophie.WriteCloser is sophie.Writer + io.Closer

type Writer

type Writer interface {
	io.Writer
	io.ByteWriter
}

sophie.Writer is io.Writer + io.ByteWriter

Source Files

Directories

Path Synopsis
Package kv supporting read and write of a simple file formating for Sophie, which stores key-value pairs.
Package kv supporting read and write of a simple file formating for Sophie, which stores key-value pairs.
Package mr provides a local concurrent computing model(MapReduce) using Sophie serialization.
Package mr provides a local concurrent computing model(MapReduce) using Sophie serialization.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL