fstack

README

About

This is extended stack implementation on Go-lang with the file as storage

For API see http://godoc.org/github.com/reddec/file-stack

Used in file-stack-db

Installation

By Go build

go get -u github.com/reddec/file-stack/cmd/...

For Debian/Centos by packager.io

Introduction

Use-case

When a lot of messages (events from IoT for example) with simple structure have to be saved on disk for later usage with instant (or near that) access to last message (I mean something like get last state). Processing of historical events may be non-realtime operation.

Requirements

Expected load:

• Rate: about 40'000+ messages/seconds
• Message size:
• 90% - less then 1KB (means status updates or notifications)
• 10% - about 5-50MB (means camera images or batch updates)
• Message contains:
• Small (can easily fits into one memory block - about 4KB) header
• Big body

Expected environment:

• Fast, big and reliable file storage - like SSD, enterprise HDD or NAS
• Low-cost multi-cores computing unit - like virtual machine in Digital Ocean, Supermicro or Raspberry Pi 2

Expected usage:

• State query (aka get last message): about 5'000+ requests/seconds
• Full historical dump: about 1 times/week
• Full historical headers scan: about 0.5 time/second
• Almost messages have not to be lost

Formal minimal operations descriptions:

• Put full message
• Get last message
• Iterate over headers

Possible decisions

There are several implementations: SQL databases, no-SQL, KV databases, in-memory storage and others.

Tested:

PostgreSQL

• Tested version: 9.4

Pros

• Stability
• Clustering
• Low resource usage

Cons

• Not enough speed (7000-8000 inserts/second on target hardware)

Cassandra DB

Pros

• Fastest solution in theory
• Stability

Cons

• Awful memory usage - about 10 GB in idle state

LevelDB

Pros

• Stable
• Fast

Cons

• Not enough fast (about 20000 inserts/second)
• Too simple structure: can't iterate over keys without loading full item

Current implementation

12-cores I7 5-gen, SSD 840 PRO

• Push: 200'000 messages/second
• Pop: 200'000 messages/second
• Last message: 1'000'000 requests/second

Documentation

Index

• type Stack
  • func CreateStack(filename string) (*Stack, error)
  • func NewStack(file *os.File) (*Stack, error)
  • func OpenStack(filename string) (*Stack, error)
  • func (s *Stack) Close() error
  • func (s *Stack) Depth() int
  • func (s *Stack) IterateBackward(handler func(depth int, header io.Reader, body io.Reader) bool) error
  • func (s *Stack) IterateForward(handler func(depth int, header io.Reader, body io.Reader) bool) error
  • func (s *Stack) LastAccess() time.Time
  • func (s *Stack) Peak() (header, data []byte, err error)
  • func (s *Stack) PeakHeader() (header []byte, err error)
  • func (s *Stack) Pop() (header, data []byte, err error)
  • func (s *Stack) Push(header, data []byte) (depth int, err error)
  • func (s *Stack) Repare() error

Types

type Stack

type Stack struct {
	io.Closer
	// contains filtered or unexported fields
}

Stack in file

func CreateStack

func CreateStack(filename string) (*Stack, error)

CreateStack - create or truncate stack

func NewStack

func NewStack(file *os.File) (*Stack, error)

NewStack - create new stack based on file

func OpenStack

func OpenStack(filename string) (*Stack, error)

OpenStack - open or create stack

func (*Stack) Close

func (s *Stack) Close() error

Close backend stack file. If access is requried, file will automatically reopened

func (*Stack) Depth

func (s *Stack) Depth() int

Depth of stack - count of segments

func (*Stack) IterateBackward

func (s *Stack) IterateBackward(handler func(depth int, header io.Reader, body io.Reader) bool) error

IterateBackward - iterate over hole stack segment-by-segment from end to begining

func (*Stack) IterateForward

func (s *Stack) IterateForward(handler func(depth int, header io.Reader, body io.Reader) bool) error

IterateForward - iterate over hole stack segment-by-segment from begining to end. If all segments iterated stack may be repaired

func (*Stack) LastAccess

func (s *Stack) LastAccess() time.Time

LastAccess - time point of last access to stack

func (*Stack) Peak

func (s *Stack) Peak() (header, data []byte, err error)

Peak of stack - get one segment from stack but not remove

func (*Stack) PeakHeader

func (s *Stack) PeakHeader() (header []byte, err error)

PeakHeader get only header part from tail segment from stack without remove

func (*Stack) Pop

func (s *Stack) Pop() (header, data []byte, err error)

Pop one segment from tail of stack. Returns nil,nil,nil if depth is 0

func (*Stack) Push

func (s *Stack) Push(header, data []byte) (depth int, err error)

Push header and body to stack. Returns new value of stack depth

func (*Stack) Repare

func (s *Stack) Repare() error

Repare stack segements