laozi

README

laozi

archiver of events.

stores events to s3 partitioned however you want. imagine AWS firehose service but with a configurable partition method.

usage

package main

import (
	"fmt"
	"time"

	laozi "github.com/seedboxtech/laozi"
)

func main() {
	l := laozi.NewLaozi(&laozi.Config{
		LoggerFactory: laozi.S3LoggerFactory{
			Bucket: "laozi-test",
			Region: "us-east-1",
			Prefix: "events/", // optional
			FlushInterval: time.Second * 30, // optional
			Compression: "gzip", // optional
			IsDupeFunc: func(event []byte, line []byte) bool {
				// implement some method of checking for duplicates
				return string(event) == string(line)
			}
		},
		EventChannelSize: 10000000,
		LoggerTimeout:    time.Minute,
		PartitionKeyFunc: func(e []byte) (string, error) {
			return "event-file.csv.gz", nil
		},
	})

	quit := time.After(2 * time.Minute)
	i := 0
loop:
	for {
		select {
		case <-quit:
			break loop
		default:
			i++
			l.Log([]byte(fmt.Sprintf("%d\n", i)))
			time.Sleep(time.Millisecond)

		}
	}

	fmt.Println("Done logging!")
	fmt.Println(" - logged:", i)

	// give some time for the final flush to happenx
	<-time.After(2 * time.Minute)

}

testing

currently this package uses s3 directly in tests. this does mean tests will cost a very small amount of money to run and requires a connection to the internet. tests can be run...

go test ./...

Documentation

Overview

Package laozi is an archiver of events. stores events to s3 partitioned however you want. imagine AWS firehose service but with a configurable partition method.

Index

• type Config
• type Laozi
  • func New(c *Config) Laozi
• type Logger
• type LoggerFactory
• type MockLaozi
  • func (d MockLaozi) Close()
  • func (d MockLaozi) Log(b []byte)
• type S3LoggerFactory
  • func (lf *S3LoggerFactory) New(key string) Logger
  • func (lf *S3LoggerFactory) Release(logger Logger)

Types

type Config

type Config struct {
	LoggerFactory    LoggerFactory
	LoggerTimeout    time.Duration
	PartitionKeyFunc func([]byte) (string, error)
	EventChannelSize int
	RoutingWorkers   int
}

Config is a struct used to configure Laozi to your implementation.

type Laozi

type Laozi interface {
	Log([]byte)
	Close()
}

Laozi is an archiver responsible for receiving events and archiving them to a safe, reliable storage. Currently it achieves this by implementing s3 storage. It is named after one of the most famous archivists in the world, https://en.wikipedia.org/wiki/Laozi

func New

func New(c *Config) Laozi

New creates a new laozi to start receiving events and start the logger monitoring

type Logger

type Logger interface {
	// send event data
	Log([]byte)
	// Close method called before it removed from internal map
	Close() error
	// LastActive is used to get the time a logger last logged. Used for deleting stale
	// loggers from internal map.
	LastActive() time.Time
}

Logger defines the behaviour of all loggers.

type LoggerFactory

type LoggerFactory interface {
	New(key string) Logger
	Release(Logger)
}

LoggerFactory is an interface that defines how to make a new logger. This Logger will be responsible for logging all events to it that match the same partition key.

type MockLaozi

type MockLaozi struct{}

func (MockLaozi) Close

func (d MockLaozi) Close()

func (MockLaozi) Log

func (d MockLaozi) Log(b []byte)

type S3LoggerFactory

type S3LoggerFactory struct {
	Prefix         string
	Bucket         string
	Region         string
	FlushInterval  time.Duration
	Compression    string
	IsDupeFunc     func(event []byte, line []byte) bool
	LogChannelSize int
	// contains filtered or unexported fields
}

S3LoggerFactory is a logger factory for creating loggers that log received events to S3.

func (*S3LoggerFactory) New

func (lf *S3LoggerFactory) New(key string) Logger

NewLogger return a new instance of an S3 Logger for a corresponding partition key.

func (*S3LoggerFactory) Release

func (lf *S3LoggerFactory) Release(logger Logger)

NewLogger return a new instance of an S3 Logger for a corresponding partition key.