tissa

README

Tissa

Time Series Storage Archive

Package tissa provides a simple embedded time series storage engine.

A Tissa TimeSeries can track a number of key-value pairs over time. When declaring a timeseries, you can specify one or more archives, each with its own granularity and retention. Rollups are done automatically upon insertion of new data. Unlike RRDs, keys need not be static over the lifecycle of a timeseries. This makes Tissa work well for tracking entities that come and go.

Tissa supports persistence, but only when Write() is called. There's no WAL or immediate consistency. Tissa tracks data in "chunks", of (default) 2000 intervals, and all reads and writes are of an entire chunk. So, while you can call Write() as often as you like, do realize that each write call writes a full chunk. Retentions are also exercised whenever Write() is called. Chunks that are fully beyond the retention time are deleted.

Getting Started

go get github.com/fred-lewis/tissa

Example:

tsc := TimeSeriesConfig{
	Archives: []ArchiveConfig{
		{SECOND, HOUR}, // retain 1-second data for 1 hour
		{MINUTE, DAY}, // retain 1-minute rollups for 1 day
	},
	DefaultValue: 0,
}

os.RemoveAll("/tmp/my_timeseries")
ts, _ := NewTimeSeries("/tmp/my_timeseries", tsc)

t := int64(1560632038)
ts.AddValue("thing1", 100.0, t)
ts.AddValue("thing2", 200.0, t)

t++
ts.AddValue("thing1", 200.0, t)
ts.AddValue("thing2", 300.0, t)

t += 5
ts.AddValue("thing1", 300.0, t)
ts.AddValue("thing2", 400.0, t)

The last timestamp increment crosses a minute boundary and incurs rollups for the minute ending 1560632040.

averages, timestamps, _ := ts.Averages(1560632038, t+1, SECOND)
fmt.Printf("%+v\n%+v\n", timestamps, averages)

Output:

[1560632038 1560632039 1560632040 1560632041 1560632042 1560632043 1560632044]
map[thing1:[100 200 0 0 0 0 300] thing2:[200 300 0 0 0 0 400]]

averages, timestamps, _ = ts.Averages(1560632038, t+1, MINUTE)
fmt.Printf("%+v\n%+v\n", timestamps, averages)

Output:

[1560632040]
map[thing1:[150] thing2:[250]]

GoDoc

https://godoc.org/github.com/fred-lewis/tissa

Things to know:

• Tissa normalizes datapoints as they are added. Timestamps are aligned to interval boundaries, and any missing intervals are filled in with the specified defaul value
• The latest chunk of each archive is cached in-memory

Documentation

Overview

Package tissa provides a simple embedded time series storage engine.

A Tissa TimeSeries can track a number of key-value pairs over time. When declaring a timeseries, you can specify one or more archives, each with its own granularity and retention. Rollups are done automatically upon insertion of new data. Unlike RRDs, keys need not be static over the lifecycle of a timeseries. This makes Tissa work well for tracking entities that come and go.

Tissa supports persistence, but only when Write() is called. There's no WAL or immediate consistency. Tissa tracks data in "chunks", of (default) 2000 intervals, and all reads and writes are of an entire chunk. So, while you can call Write() as often as you like, do realize that each write call writes a full chunk. Retentions are also exercised whenever Write() is called. Chunks that are fully beyond the retention time are deleted.

Example:

tsc := TimeSeriesConfig{
	Archives: []ArchiveConfig{
		{SECOND, HOUR}, // retain 1-second data for 1 hour
		{MINUTE, DAY}, // retain 1-minute rollups for 1 day
	},
	DefaultValue: 0,
}

os.RemoveAll("/tmp/my_timeseries")
ts, _ := NewTimeSeries("/tmp/my_timeseries", tsc)

t := int64(1560632038)
ts.AddValue("thing1", 100.0, t)
ts.AddValue("thing2", 200.0, t)

t++
ts.AddValue("thing1", 200.0, t)
ts.AddValue("thing2", 300.0, t)

t += 5
ts.AddValue("thing1", 300.0, t)
ts.AddValue("thing2", 400.0, t)

The last timestamp increment crosses a minute boundary and incurs rollups for the minute ending 1560632040.

averages, timestamps, _ := ts.Averages(1560632038, t+1, SECOND)
fmt.Printf("%+v\n%+v\n", timestamps, averages)

[1560632038 1560632039 1560632040 1560632041 1560632042 1560632043 1560632044]

map[thing1:[100 200 0 0 0 0 300] thing2:[200 300 0 0 0 0 400]]

averages, timestamps, _ = ts.Averages(1560632038, t+1, MINUTE)
fmt.Printf("%+v\n%+v\n", timestamps, averages)

[1560632040]

map[thing1:[150] thing2:[250]]

Index

• Constants
• type ArchiveConfig
• type Rollup
• type TimeSeries
  • func NewTimeSeries(dir string, config TimeSeriesConfig) (*TimeSeries, error)
  • func OpenTimeSeries(dir string) (*TimeSeries, error)
  • func (t *TimeSeries) AddValue(key string, val float64, timestamp int64) error
  • func (t *TimeSeries) AddValues(vals map[string]float64, timestamp int64) error
  • func (t *TimeSeries) Averages(startTime, endTime, resolution int64) (map[string][]float64, []int64, error)
  • func (t *TimeSeries) Latest() (val map[string]float64, timestamp int64)
  • func (t *TimeSeries) Maximums(startTime, endTime, resolution int64) (map[string][]float64, []int64, error)
  • func (t *TimeSeries) Minimums(startTime, endTime, resolution int64) (map[string][]float64, []int64, error)
  • func (t *TimeSeries) Rollups(startTime, endTime, resolution int64) (map[string][]Rollup, []int64, error)
  • func (t *TimeSeries) Write() error
• type TimeSeriesConfig

Constants

const (
	SECOND        int64 = 1
	TEN_SECOND          = 10
	THIRTY_SECOND       = 30
	MINUTE              = 60
	FIVE_MINUTE         = 5 * MINUTE
	TEN_MINUTE          = 10 * MINUTE
	THIRTY_MINUTE       = 30 * MINUTE
	HOUR                = 60 * MINUTE
	SIX_HOUR            = 6 * HOUR
	TWELVE_HOUR         = 12 * HOUR
	DAY                 = 24 * HOUR
)

Each divisible by all priors

Types

type ArchiveConfig

type ArchiveConfig struct {
	Resolution int64
	Retention  int64
}

Resolution and retention specified in seconds. Use package-level enum (SECOND, TEN_SECOND ...) to ensure all resolutions divide evenly

type Rollup

type Rollup struct {
	Total float64
	Count int64
	Min   float64
	Max   float64
}

type TimeSeries

type TimeSeries struct {
	LastWritten int64
	// contains filtered or unexported fields
}

A TimeSeries can track multiple key/value pairs over time. TimeSeries are append-only,and data can be rolled up to a number of loawer-resolution archives.

func NewTimeSeries

func NewTimeSeries(dir string, config TimeSeriesConfig) (*TimeSeries, error)

Construct a new TimeSeries in the given directory, with the given configuration

At least one archive must be provided. If multiple archives are provided, each resolution must be a multiple of the last, and rollups will be populated automatically when new data is inserted.

func OpenTimeSeries

func OpenTimeSeries(dir string) (*TimeSeries, error)

Open an existing TimeSeries in the given directory

func (*TimeSeries) AddValue

func (t *TimeSeries) AddValue(key string, val float64, timestamp int64) error

Add a single key-value pair to the timeseries with the given timestamp. Timeseries are append-only. Timestamps will be normalized to a multiple of the TimeSeries' base resolution.

func (*TimeSeries) AddValues

func (t *TimeSeries) AddValues(vals map[string]float64, timestamp int64) error

Add multiple key-value pairs for the given timestamp. Timestamps will be normalized to a multiple of the TimeSeries' base resolution.

func (*TimeSeries) Averages

func (t *TimeSeries) Averages(startTime, endTime, resolution int64) (map[string][]float64, []int64, error)

For querying rollup archives. Returns average value series for all keys.

func (*TimeSeries) Latest

func (t *TimeSeries) Latest() (val map[string]float64, timestamp int64)

Retrieve the latest key-value pairs

func (*TimeSeries) Maximums

func (t *TimeSeries) Maximums(startTime, endTime, resolution int64) (map[string][]float64, []int64, error)

For querying rollup archives. Returns maximum value series for all keys.

func (*TimeSeries) Minimums

func (t *TimeSeries) Minimums(startTime, endTime, resolution int64) (map[string][]float64, []int64, error)

For querying rollup archives. Returns minimums value series for all keys.

func (*TimeSeries) Rollups

func (t *TimeSeries) Rollups(startTime, endTime, resolution int64) (map[string][]Rollup, []int64, error)

For querying raw daa from rollup archives.

func (*TimeSeries) Write

func (t *TimeSeries) Write() error

Write the TimeSeries to disk, and exercise retention (delete any chunks that are fully expired).

type TimeSeriesConfig

type TimeSeriesConfig struct {
	Archives     []ArchiveConfig
	DefaultValue float64
}

One or more ArchiveConfigs is required. DefaultValue is the value to use for missing data.