rdbtools

README

rdbtools

This is a parser for Redis RDB snapshot files. It is loosely based on redis-rdb-tools.

It is compatible with Redis RDB files generated since Redis 1.0 and up to the last version. It comes with an extensive test suite.

Documentation

The reference is available here. Right now it's not really useful, but it will be improved.

License

This project is licensed under the MIT license. See the LICENSE file for details.

Documentation

Overview

Package rdbtools is a Redis RDB snapshot file parser.

Parsing a file

Example of how to parse a RDB file.

ctx := rdbtools.ParserContext{
	ListMetadataCh: make(chan rdbtools.ListMetadata),
	ListDataCh: make(chan interface{}),
}
p := rdbtools.NewParser(ctx)

go func() {
	stop := false
	for !stop {
		select {
		case md, ok := <-ctx.ListMetadataCh:
			if !ok {
				ctx.ListMetadataCh = nil
				break
			}

			// do something with the metadata
		case d, ok := <-ctx.ListDataCh:
			if !ok {
				ctx.ListDataCh = nil
				break
			}

			str := rdbtools.DataToString(d)
			// do something with the string
		}

		if ctx.Invalid() {
			break
		}
	}
}()

f, _ := os.Open("/var/lib/redis/dump.rdb")
if err := p.Parse(f); err != nil {
	log.Fatalln(err)
}

The context holds the channels you will use to receive data from the parser. You only need to provide a channel if you care about it. In the example above, we only care about the lists in the RDB file, so we don't provide all the other channels.

The parser only has one method Parse(ParserContext) which takes a context. After a call to Parse, the parser can't be reused. We plan to change that though.

Why interfaces everywhere

interface{} is used everywhere in rdbtools. The reason is simple: in RDB files, keys and values can be encoded as strings or integers or even binary data. You might call a key "1" but Redis will happily encode that as an integer.

The majority of the time you will have strings in your keys, and a lot of the times your values will be strings as well. For this reason, we provided the function DataToString(interface{}) which takes care of casting or converting to a string.

Index

• Constants
• Variables
• func DataToString(i interface{}) string
• type HashEntry
  • func (e HashEntry) String() string
• type HashMetadata
  • func (m HashMetadata) String() string
• type KeyObject
  • func NewKeyObject(key interface{}, expiryTime int64) KeyObject
  • func (k KeyObject) Expired() bool
  • func (k KeyObject) String() string
• type ListMetadata
  • func (m ListMetadata) String() string
• type Parser
  • func NewParser(ctx ParserContext) Parser
• type ParserContext
  • func (c *ParserContext) Invalid() bool
• type SetMetadata
  • func (m SetMetadata) String() string
• type SortedSetEntry
  • func (e SortedSetEntry) String() string
• type SortedSetMetadata
  • func (m SortedSetMetadata) String() string
• type StringObject
  • func (s StringObject) String() string

Constants

const (
	// The last version of RDB files
	RedisRdbVersion = 6
)

Variables

var (
	ErrInvalidMagicString            = errors.New("invalid magic string")
	ErrInvalidRDBVersionNumber       = errors.New("invalid RDB version number")
	ErrInvalidChecksum               = errors.New("invalid checksum")
	ErrUnexpectedEncodedLength       = errors.New("unexpected encoded length")
	ErrUnknownValueType              = errors.New("unknown value type")
	ErrUnknownLengthEncoding         = errors.New("unknown length encoding")
	ErrUnexpectedPrevLengthEntryByte = errors.New("unexpected prev length entry byte")
)

Functions

func DataToString

func DataToString(i interface{}) string

Cast or convert an interface{} object to a string If the type is not string, fmt.Stringer, []byte or an integer, it panics

Types

type HashEntry

type HashEntry struct {
	Key   interface{}
	Value interface{}
}

Represents an entry in a hash

func (HashEntry) String

func (e HashEntry) String() string

Returns a string visualization of the entry

type HashMetadata

type HashMetadata struct {
	Key KeyObject
	Len int64
}

Represents the metadata of a hash, which is the key and the hash length

func (HashMetadata) String

func (m HashMetadata) String() string

Returns a visualization of the hash metadata

type KeyObject

type KeyObject struct {
	ExpiryTime time.Time   // The expiry time of the key. If none, this object IsZero() method will return true
	Key        interface{} // The key value
}

Represents a Redis key.

func NewKeyObject

func NewKeyObject(key interface{}, expiryTime int64) KeyObject

Create a new key. If expiryTime >= 0 it will be used.

func (KeyObject) Expired

func (k KeyObject) Expired() bool

Returns true if the key is expired (meaning the key's expiry time is before now), false otherwise.

func (KeyObject) String

func (k KeyObject) String() string

Return a visualization of the key.

type ListMetadata

type ListMetadata struct {
	Key KeyObject
	Len int64
}

Represents the metadata of a list, which is the key and the list length

func (ListMetadata) String

func (m ListMetadata) String() string

Returns a visualization of the list metadata

type Parser

type Parser interface {
	Parse(r io.Reader) (err error)
}

func NewParser

func NewParser(ctx ParserContext) Parser

Create a new parser using the provided context

type ParserContext

type ParserContext struct {
	DbCh                chan int
	StringObjectCh      chan StringObject
	ListMetadataCh      chan ListMetadata
	ListDataCh          chan interface{}
	SetMetadataCh       chan SetMetadata
	SetDataCh           chan interface{}
	HashMetadataCh      chan HashMetadata
	HashDataCh          chan HashEntry
	SortedSetMetadataCh chan SortedSetMetadata
	SortedSetEntriesCh  chan SortedSetEntry
	// contains filtered or unexported fields
}

A ParserContext holds the channels used to receive data from the parser

func (*ParserContext) Invalid

func (c *ParserContext) Invalid() bool

Invalid returns true if the context is invalid (all channels are nil), false otherwise. This is needed to actually terminate parsing if you use a for-select loop

type SetMetadata

type SetMetadata struct {
	Key KeyObject
	Len int64
}

Represents the metadata of a set, which is the key and the set length

func (SetMetadata) String

func (m SetMetadata) String() string

Returns a visualization of the set metadata

type SortedSetEntry

type SortedSetEntry struct {
	Value interface{}
	Score float64
}

Represents an entry in a sorted set.

func (SortedSetEntry) String

func (e SortedSetEntry) String() string

Returns a visualization of a sorted set entry

type SortedSetMetadata

type SortedSetMetadata struct {
	Key KeyObject
	Len int64
}

Represents the metadata of a sorted set, which is the key and the sorted set length

func (SortedSetMetadata) String

func (m SortedSetMetadata) String() string

Returns a visualization of the sorted set metadata

type StringObject

type StringObject struct {
	Key   KeyObject
	Value interface{}
}

Represents a Redis string (which you get/set with SET, GET, MSET, MGET, etc).

func (StringObject) String

func (s StringObject) String() string

Returns a visualization of the string.