rdb

package module
v0.0.0-...-12db9b2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 10, 2017 License: MIT Imports: 19 Imported by: 1

README

rdb Build Status GoDoc

WARNING: This project is not production ready.

Package rdb is a Redis RDB File parser written in Go.

Feature

  • Concurrent parsing.
  • Built in memory report.
  • Simple filter support.
  • Layered skipping strategies.

Install

go get -u github.com/matthewjhe/rdb

Usage

Basic Example

The basic use case is very simple, just implements Filter interface and pass it to Parse:

package main

import (
    "github.com/matthewjhe/rdb"
)

func (f filter) Key(k rdb.Key) bool { return false }
func (f filter) Type(t rdb.Type) bool   { return false }
func (f filter) Database(db rdb.DB) bool { return false }
func (f filter) Set(v *rdb.Set) { }
func (f filter) List(v *rdb.List) { }
func (f filter) Hash(v *rdb.Hash) { }
func (f filter) String(v *rdb.String) { }
func (f filter) SortedSet(v *rdb.SortedSet) { }

func main() {
    const file = "/tmp/dump.rdb"
    reader, err := rdb.NewBufferReader(file, 0)
    if err != nil {
        panic(err)
    }

    if err := rdb.Parse(reader, rdb.WithFilter(filter{})); err != nil {
        panic(err)
    }
}
Skipping

NOTE: RDB file is read sequentially, when we say skips a database, we also need to parse this database's every single key, we just simply skips actions like decompress, unzip ziplist, etc. It is impossible to skips a key without reading its metadata.

  • Global Skip Strategy

Global skip strategy applies to the whole parsing lifetime, it can be overwritten by Filter's Database method, it will be restored when parsing a database completely.

Global skip strategy is set by a ParseOption when rdb.Parse is called:

    // reads memory report only
    strategy := rdb.WithStrategy(rdb.SkipExpiry | rdb.SkipMeta | rdb.SkipValue)
    err := rdb.Parse(reader, strategy, rdb.WithFilter(filter{}))
    // ...
  • Database Skip Strategy

Database skip strategy applies to a database lifetime, it is set by Filter's Database method, and it can be overwritten by Filter's Key or Type method. Database skip strategy is set to global skip strategy by default.

func (f filter) Database(db rdb.DB) bool {
    // skips this database
    db.Skip(rdb.SkipAll)

    // skips database's metadata and key's expiry
    // db.Skip(rdb.SkipMeta | rdb.SkipExpiry)
    return false
}
  • Type and Key Skip Strategy

Type and key skip strategy overwrite database skip strategy. It only applies the key and value which will read next.

func (f filter) Type(t rdb.Type) bool {
    // skips this type
    t.Skip(rdb.SkipAll)
    return false
}

func (f filter) Key(key rdb.Key) bool {
    // skips this key
    // key.Skip(rdb.SkipAll)

    // skips value
    key.Skip(rdb.SkipValue)
    return false
}
Syncing

By default, rdb uses multiple goroutines to parse keys which would cause keys disorder. If the order of keys is important, use EnableSync ParseOption.

    rdb.Parse(reader, rdb.WithFilter(filter{}), rdb.EnableSync())
Aborting

Sometimes we want to abort the parsing process, this can be done by return true when one of the following methods is called: Note that the abort action is taking effect immediately when the function returns.

func (f filter) Key(k rdb.Key) bool {
    return true
}
func (f filter) Type(t rdb.Type) bool   {
    return true
}
func (f filter) Database(db rdb.DB) bool {
    return true
}

References

License

MIT

Documentation

Overview

Package rdb implements a Redis RDB File parser.

Basic Example

The basic use case is very simple, just implements Filter interface and pass it to Parse:

package main

import (
    "github.com/matthewjhe/rdb"
)

func (f filter) Key(k rdb.Key) bool { return false }
func (f filter) Type(t rdb.Type) bool   { return false }
func (f filter) Database(db rdb.DB) bool { return false }
func (f filter) Set(v *rdb.Set) { }
func (f filter) List(v *rdb.List) { }
func (f filter) Hash(v *rdb.Hash) { }
func (f filter) String(v *rdb.String) { }
func (f filter) SortedSet(v *rdb.SortedSet) { }

func main() {
    const file = "/tmp/dump.rdb"
    reader, err := rdb.NewBufferReader(file, 0)
    if err != nil {
        panic(err)
    }

    if err := rdb.Parse(reader, rdb.WithFilter(filter{})); err != nil {
        panic(err)
    }
}

Skipping

NOTE: RDB file is read sequentially, when we say skips a database, we also need to parse this database's every single key, we just simply skips actions like decompress, unzip ziplist, etc. It is impossible to skips a key without reading its metadata.

Global skip strategy applies to the whole parsing lifetime, it can be overwritten by Filter's Database method, it will be restored when parsing a database completely.

Global skip strategy is set by a ParseOption when rdb.Parse is called:

// reads memory report only
strategy := rdb.WithStrategy(rdb.SkipExpiry | rdb.SkipMeta | rdb.SkipValue)
err := rdb.Parse(reader, strategy, rdb.WithFilter(filter{}))

Database skip strategy applies to a database lifetime, it is set by Filter's Database method, and it can be overwritten by Filter's Key or Type method. Database skip strategy is set to global skip strategy by default.

func (f filter) Database(db rdb.DB) bool {
    // skips this database
    db.Skip(rdb.SkipAll)

    // skips database's metadata and key's expiry
    // db.Skip(rdb.SkipMeta | rdb.SkipExpiry)
    return false
}

Type and key skip strategy overwrite database skip strategy. It only applies the key and value which will read next.

func (f filter) Type(t rdb.Type) bool {
    // skips this type
    t.Skip(rdb.SkipAll)
    return false
}

func (f filter) Key(key rdb.Key) bool {
    // skips this key
    // key.Skip(rdb.SkipAll)

    // skips value
    key.Skip(rdb.SkipValue)
    return false
}

Syncing

By default, rdb uses multiple goroutines to parse keys which would cause keys disorder. If the order of keys is important, use EnableSync ParseOption.

rdb.Parse(reader, rdb.WithFilter(filter{}), rdb.EnableSync())

Aborting

Sometimes we want to abort the parsing process, this can be done by return true when one of the following methods is called: Note that the abort action is taking effect immediately when the function returns.

func (f filter) Key(k rdb.Key) bool {
    return true
}
func (f filter) Type(t rdb.Type) bool   {
    return true
}
func (f filter) Database(db rdb.DB) bool {
    return true
}

Index

Constants

View Source
const (
	SkipMeta = 1 << iota
	SkipExpiry
	SkipValue
	SkipAll
)

Parse strategies

View Source
const (
	EncodingString       = 0
	EncodingList         = 1
	EncodingSet          = 2
	EncodingSortedSet    = 3
	EncodingHash         = 4
	EncodingSortedSet2   = 5
	EncodingZipmap       = 9
	EncodingZiplist      = 10
	EncodingIntset       = 11
	EncodingSortedSetZip = 12
	EncodingHashZip      = 13
	EncodingQuicklist    = 14
)

Redis value encodings.

View Source
const (
	TypeSet       = "set"
	TypeList      = "list"
	TypeHash      = "hash"
	TypeString    = "string"
	TypeSortedSet = "sortedset"
)

Redis types.

Variables

View Source
var (
	ErrInvalidRDB            = stderr.New("Invalid RDB file")
	ErrUnsupportedRDB        = stderr.New("Unsupported RDB version")
	ErrInvalidZipmapEntry    = stderr.New("Invalid zipmap entry")
	ErrInvalidLengthEncoding = stderr.New("Invalid length encoding")
	ErrInvalidCompressedData = stderr.New("Invalid compressed data")
)

Parse errors

Functions

func Encoding2String

func Encoding2String(encoding byte) string

Encoding2String returns the string format of encoding.

func Encoding2Type

func Encoding2Type(encoding byte) string

Encoding2Type returns the actual redis type of encoding.

func Parse

func Parse(r Reader, opts ...ParseOption) error

Parse parses a Redis RDB file.

Types

type BufferReader

type BufferReader struct {
	*bufio.Reader
	// contains filtered or unexported fields
}

BufferReader is a Reader that reads from a *bufio.Reader.

func (*BufferReader) Close

func (r *BufferReader) Close() error

Close closes the file.

func (*BufferReader) Discard

func (r *BufferReader) Discard(n int)

Discard skips the next n bytes.

func (*BufferReader) ReadBytes

func (r *BufferReader) ReadBytes(n int) ([]byte, error)

ReadBytes reads and returns exactly n bytes. If ReadBytes reads fewer than n bytes, it also returns an error.

type DB

type DB struct {
	Num int
	// contains filtered or unexported fields
}

DB represents a redis database.

func (DB) Skip

func (db DB) Skip(strategy int)

Skip sets next database's skip strategy.

type Filter

type Filter interface {
	Key(key Key) bool
	Type(typ Type) bool
	Database(db DB) bool

	Set(s *Set)
	List(l *List)
	Hash(h *Hash)
	String(s *String)
	SortedSet(s *SortedSet)
}

A Filter controls the parser's behaviors.

NOTE: Filter is not safe for concurrent use.

type Hash

type Hash struct {
	Key    Key
	Values map[string]string
	// contains filtered or unexported fields
}

Hash represents redis hash.

func (Hash) Memory

func (h Hash) Memory() uint64

Memory reports memory used by l.

type Key

type Key struct {
	Encoding byte
	DB       int
	Expiry   int
	Key      string
	// contains filtered or unexported fields
}

Key represents a redis key.

func (Key) Skip

func (k Key) Skip(strategy int)

Skip sets next item's skipping strategy.

type List

type List struct {
	Key    Key
	Values []string
	// contains filtered or unexported fields
}

List represents redis list.

func (List) Memory

func (l List) Memory() uint64

Memory reports memory used by l.

type MemReader

type MemReader struct {
	// contains filtered or unexported fields
}

MemReader is a Reader that reads rdb data from memory.

func (*MemReader) Discard

func (r *MemReader) Discard(n int)

Discard skips the next n bytes.

func (*MemReader) ReadByte

func (r *MemReader) ReadByte() (byte, error)

ReadByte reads and returns a single byte. If no byte is available, returns an error.

func (*MemReader) ReadBytes

func (r *MemReader) ReadBytes(n int) ([]byte, error)

ReadBytes reads and returns exactly n bytes. If fewer than n bytes is available, returns an error.

NOTE: It's not safe to modify the returned slice.

type ParseOption

type ParseOption func(*Parser)

ParseOption configures the behaviors when parsing a rdb file.

func EnableSync

func EnableSync() ParseOption

EnableSync returns a ParseOption which disable async filtering.

func WithFilter

func WithFilter(filter Filter) ParseOption

WithFilter returns a ParseOption which sets a parse filter.

func WithStrategy

func WithStrategy(strategy int) ParseOption

WithStrategy returns a ParseOption which sets default parse strategy.

type Parser

type Parser struct {
	sync.WaitGroup
	Reader
	// contains filtered or unexported fields
}

Parser represents a Redis RDB parser.

func (*Parser) Parse

func (p *Parser) Parse() error

Parse parses a Redis RDB file.

type Reader

type Reader interface {
	Discard(n int)
	ReadByte() (byte, error)
	ReadBytes(n int) ([]byte, error)
	// contains filtered or unexported methods
}

Reader is the interface that wraps the operations against rdb data.

Discard skips the next n bytes. ReadByte reads and returns a single byte. ReadBytes reads and returns exactly n bytes.

func NewBufferReader

func NewBufferReader(file string, size int) (Reader, error)

NewBufferReader returns a new BufferReader reading from file. It's buffer has at least the specified size. If size == 0, use default size.

func NewMemReader

func NewMemReader(file string) (Reader, error)

NewMemReader memory-maps the named file and returns a MemReader that reads from it.

type Set

type Set struct {
	Key    Key
	Values map[interface{}]struct{}
	// contains filtered or unexported fields
}

Set represents redis set.

func (Set) Memory

func (s Set) Memory() uint64

Memory reports memory used by s.

type SortedSet

type SortedSet struct {
	Key    Key
	Values map[string]float64
	// contains filtered or unexported fields
}

SortedSet represents redis sortedset.

func (SortedSet) Memory

func (ss SortedSet) Memory() uint64

Memory reports memory used by ss.

type String

type String struct {
	Key   Key
	Value string
	// contains filtered or unexported fields
}

String represents redis sds.

func (String) Memory

func (s String) Memory() uint64

Memory reports memory used by s.

type Type

type Type struct {
	Encoding byte
	// contains filtered or unexported fields
}

Type represents a redis type.

func (Type) Skip

func (t Type) Skip(strategy int)

Skip sets next item's skipping strategy.

Directories

Path Synopsis
cmd
rmr command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL