rdr

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 16, 2018 License: Apache-2.0 Imports: 17 Imported by: 1

README

RDR: redis data reveal

RDR(redis data reveal) is a tool to parse redis rdbfile. Comparing to redis-rdb-tools, RDR is implemented by golang, much faster (5GB rdbfile takes about 2mins on my PC).

Usage

NAME:
   rdr - a tool to parse redis rdbfile

USAGE:
   rdr [global options] command [command options] [arguments...]

VERSION:
   v0.0.1

COMMANDS:
     show     show statistical information of rdbfile by webpage
     keys     get all keys from rdbfile
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version
NAME:
   rdr show - show statistical information of rdbfile by webpage

USAGE:
   rdr show [command options] FILE1 [FILE2] [FILE3]...

OPTIONS:
   --port value, -p value  Port for rdr to listen (default: 8080)
NAME:
   rdr keys - get all keys from rdbfile

USAGE:
   rdr keys FILE1 [FILE2] [FILE3]...

Linux amd64 Download

OSX Download

After downloading meybe need add permisson to execute.

$ chmod a+x ./rdr

Exapmle

$ ./rdr show -p 8080 *.rdb

Note that the memory usage is approximate. show example

$ ./rdr keys example.rdb
portfolio:stock_follower_count:ZH314136
portfolio:stock_follower_count:ZH654106
portfolio:stock_follower:ZH617824
portfolio:stock_follower_count:ZH001019
portfolio:stock_follower_count:ZH346349
portfolio:stock_follower_count:ZH951803
portfolio:stock_follower:ZH924804
portfolio:stock_follower_count:INS104806

License

This project is under Apache v2 License. See the LICENSE file for the full license text.

Documentation

Overview

This code is mostly copied from Beego BeeMap

Index

Constants

This section is empty.

Variables

View Source
var Counters = NewSafeMap()
View Source
var (
	TplCommonData = map[string]interface{}{}
)

Functions

func Index

func InitHTMLTmpl

func InitHTMLTmpl()

func RdbReveal

func RdbReveal(w http.ResponseWriter, r *http.Request, p httprouter.Params)

func ServeHTML

func ServeHTML(w http.ResponseWriter, layout string, content string, data map[string]interface{})

ServeHTML generate and write html to client

Types

type Counter

type Counter struct {
	// contains filtered or unexported fields
}

Counter for redis memory useage

func NewCounter

func NewCounter() *Counter

NewCounter return a pointer of Counter

func (*Counter) Count

func (c *Counter) Count(in <-chan *Entry)

Count by various dimensions

func (*Counter) GetLargestEntries

func (c *Counter) GetLargestEntries(num int) []*Entry

GetLargestEntries from heap, num max is 500

func (*Counter) GetLargestKeyPrefixes

func (c *Counter) GetLargestKeyPrefixes() []*PrefixEntry

GetLargestKeyPrefixes from heap

func (*Counter) GetLenLevelCount

func (c *Counter) GetLenLevelCount() []*PrefixEntry

GetLenLevelCount from map

type Decoder

type Decoder struct {
	Entries chan *Entry

	nopdecoder.NopDecoder
	// contains filtered or unexported fields
}

Decoder decode rdb file

func NewDecoder

func NewDecoder() *Decoder

NewDecoder new a rdb decoder

func (*Decoder) EndHash

func (d *Decoder) EndHash(key []byte)

EndHash is called when there are no more fields in a hash.

func (*Decoder) EndList

func (d *Decoder) EndList(key []byte)

EndList is called when there are no more values in a list.

func (*Decoder) EndRDB

func (d *Decoder) EndRDB()

EndRDB is called when parsing of the RDB file is complete.

func (*Decoder) EndSet

func (d *Decoder) EndSet(key []byte)

EndSet is called when there are no more fields in a set. Same as EndHash

func (*Decoder) EndZSet

func (d *Decoder) EndZSet(key []byte)

EndZSet is called when there are no more members in a sorted set.

func (*Decoder) Hset

func (d *Decoder) Hset(key, field, value []byte)

Hset is called once for each field=value pair in a hash.

func (*Decoder) Rpush

func (d *Decoder) Rpush(key, value []byte)

Rpush is called once for each value in a list.

func (*Decoder) Sadd

func (d *Decoder) Sadd(key, member []byte)

Sadd is called once for each member of a set.

func (*Decoder) Set

func (d *Decoder) Set(key, value []byte, expiry int64)

Set is called once for each string key.

func (*Decoder) StartHash

func (d *Decoder) StartHash(key []byte, length, expiry int64)

StartHash is called at the beginning of a hash. Hset will be called exactly length times before EndHash.

func (*Decoder) StartList

func (d *Decoder) StartList(key []byte, length, expiry int64)

StartList is called at the beginning of a list. Rpush will be called exactly length times before EndList. If length of the list is not known, then length is -1

func (*Decoder) StartSet

func (d *Decoder) StartSet(key []byte, cardinality, expiry int64)

StartSet is called at the beginning of a set. Sadd will be called exactly cardinality times before EndSet.

func (*Decoder) StartZSet

func (d *Decoder) StartZSet(key []byte, cardinality, expiry int64)

StartZSet is called at the beginning of a sorted set. Zadd will be called exactly cardinality times before EndZSet.

func (*Decoder) Zadd

func (d *Decoder) Zadd(key []byte, score float64, member []byte)

Zadd is called once for each member of a sorted set.

type Entry

type Entry struct {
	Key                string
	Bytes              uint64
	Type               string
	NumOfElem          uint64
	LenOfLargestElem   uint64
	FieldOfLargestElem string
}

Entry is info of a redis recored

type MemProfiler

type MemProfiler struct{}

MemProfiler get memory use for all kinds of data stuct

func (*MemProfiler) ElemLen

func (m *MemProfiler) ElemLen(element []byte) uint64

ElemLen get length of a element

func (*MemProfiler) HashtableEntryOverhead

func (m *MemProfiler) HashtableEntryOverhead() uint64

HashtableEntryOverhead get memory use of hashtable entry See https://github.com/antirez/redis/blob/unstable/src/dict.h Each dictEntry has 2 pointers + int64

func (*MemProfiler) HashtableOverhead

func (m *MemProfiler) HashtableOverhead(size uint64) uint64

HashtableOverhead get memory use of a hashtable See https://github.com/antirez/redis/blob/unstable/src/dict.h See the structures dict and dictht 2 * (3 unsigned longs + 1 pointer) + int + long + 2 pointers

Additionally, see **table in dictht The length of the table is the next power of 2 When the hashtable is rehashing, another instance of **table is created Due to the possibility of rehashing during loading, we calculate the worse case in which both tables are allocated, and so multiply the size of **table by 1.5

func (*MemProfiler) KeyExpiryOverhead

func (m *MemProfiler) KeyExpiryOverhead(expiry int64) uint64

KeyExpiryOverhead get memory useage of a key expiry Key expiry is stored in a hashtable, so we have to pay for the cost of a hashtable entry The timestamp itself is stored as an int64, which is a 8 bytes

func (*MemProfiler) LinkedListEntryOverhead

func (m *MemProfiler) LinkedListEntryOverhead() uint64

LinkedListEntryOverhead get memory use of a linked list entry See https://github.com/antirez/redis/blob/unstable/src/adlist.h A node has 3 pointers

func (*MemProfiler) LinkedlistOverhead

func (m *MemProfiler) LinkedlistOverhead() uint64

LinkedlistOverhead get memory use of a linked list See https://github.com/antirez/redis/blob/unstable/src/adlist.h A list has 5 pointers + an unsigned long

func (*MemProfiler) RobjOverhead

func (m *MemProfiler) RobjOverhead() uint64

RobjOverhead get memory useage of a robj

func (*MemProfiler) SizeofString

func (m *MemProfiler) SizeofString(bytes []byte) uint64

SizeofString get memory use of a string https://github.com/antirez/redis/blob/unstable/src/sds.h

func (*MemProfiler) SkiplistEntryOverhead

func (m *MemProfiler) SkiplistEntryOverhead() uint64

SkiplistEntryOverhead get memory use of a skiplist entry

func (*MemProfiler) SkiplistOverhead

func (m *MemProfiler) SkiplistOverhead(size uint64) uint64

SkiplistOverhead get memory use of a skiplist

func (*MemProfiler) TopLevelObjOverhead

func (m *MemProfiler) TopLevelObjOverhead() uint64

TopLevelObjOverhead get memory use of a top level object Each top level object is an entry in a dictionary, and so we have to include the overhead of a dictionary entry

type PrefixEntry

type PrefixEntry struct {
	Bytes uint64
	Num   uint64
	// contains filtered or unexported fields
}

PrefixEntry record value by prefix

type SafeMap

type SafeMap struct {
	// contains filtered or unexported fields
}

SafeMap is a map which concurrency safe

func NewSafeMap

func NewSafeMap() *SafeMap

NewSafeMap return new safemap

func (*SafeMap) Check

func (m *SafeMap) Check(k interface{}) bool

Check seturns true if k is exist in the map.

func (*SafeMap) Delete

func (m *SafeMap) Delete(k interface{})

Delete the given key and value.

func (*SafeMap) Get

func (m *SafeMap) Get(k interface{}) interface{}

Get from maps return the k's value

func (*SafeMap) Items

func (m *SafeMap) Items() map[interface{}]interface{}

Items returns all items in safemap.

func (*SafeMap) Set

func (m *SafeMap) Set(k interface{}, v interface{}) bool

Set the given key and value. Returns false if the key is already in the map and changes nothing.

func (*SafeMap) String

func (m *SafeMap) String() (s string)

String transfer map into json

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL