tstorage

Documentation

Overview

Package tstorage provides goroutine safe capabilities of insertion into and retrieval from the time-series storage.

Index

• Variables
• type DataPoint
• type Option
  • func WithDataPath(dataPath string) Option
  • func WithLogger(logger Logger) Option
  • func WithPartitionDuration(duration time.Duration) Option
  • func WithRetention(retention time.Duration) Option
  • func WithTimestampPrecision(precision TimestampPrecision) Option
  • func WithWALBufferedSize(size int) Option
  • func WithWriteTimeout(timeout time.Duration) Option
• type Reader
• type Row
• type Storage
  • func NewStorage(opts ...Option) (Storage, error)
• type TimestampPrecision

Examples

• NewStorage (WithDataPath)
• NewStorage (WithPartitionDuration)

Variables

var (
	ErrNoDataPoints = errors.New("no data points found")
)

Types

type DataPoint

type DataPoint struct {
	// The actual value. This field must be set.
	Value float64
	// Unix timestamp.
	Timestamp int64
}

DataPoint represents a data point, the smallest unit of time series data.

type Option

type Option func(*storage)

Option is an optional setting for NewStorage.

func WithDataPath

func WithDataPath(dataPath string) Option

WithDataPath specifies the path to directory that stores time-series data. Use this to make time-series data persistent on disk.

Defaults to empty string which means no data will get persisted.

func WithLogger

func WithLogger(logger Logger) Option

WithLogger specifies the logger to emit verbose output.

Defaults to a logger implementation that does nothing.

func WithPartitionDuration

func WithPartitionDuration(duration time.Duration) Option

WithPartitionDuration specifies the timestamp range of partitions. Once it exceeds the given time range, the new partition gets inserted.

A partition is a chunk of time-series data with the timestamp range. It acts as a fully independent database containing all data points for its time range.

Defaults to 1h

func WithRetention

func WithRetention(retention time.Duration) Option

WithRetention specifies when to remove old data. Data points will get automatically removed from the disk after a specified period of time after a disk partition was created. Defaults to 14d.

func WithTimestampPrecision

func WithTimestampPrecision(precision TimestampPrecision) Option

WithTimestampPrecision specifies the precision of timestamps to be used by all operations.

Defaults to Nanoseconds

func WithWALBufferedSize

func WithWALBufferedSize(size int) Option

WithWAL specifies the buffered byte size before flushing a WAL file. The larger the size, the less frequently the file is written and more write performance at the expense of durability. Giving 0 means it writes to a file whenever data point comes in. Giving -1 disables using WAL.

Defaults to 4096.

func WithWriteTimeout

func WithWriteTimeout(timeout time.Duration) Option

WithWriteTimeout specifies the timeout to wait when workers are busy.

The storage limits the number of concurrent goroutines to prevent from out of memory errors and CPU trashing even if too many goroutines attempt to write.

Defaults to 30s.

type Reader

type Reader interface {
	// Select gives back a list of data points that matches a set of the given metric and
	// labels within the given start-end range. Keep in mind that start is inclusive, end is exclusive,
	// and both must be Unix timestamp. ErrNoDataPoints will be returned if no data points found.
	Select(metric string, labels []Label, start, end int64) (points []*DataPoint, err error)
}

Reader provides reading access to time series data.

type Row

type Row struct {
	// The unique name of metric.
	// This field must be set.
	Metric string
	// An optional key-value properties to further detailed identification.
	Labels []Label
	// This field must be set.
	DataPoint
}

Row includes a data point along with properties to identify a kind of metrics.

type Storage

type Storage interface {
	Reader
	// InsertRows ingests the given rows to the time-series storage.
	// If the timestamp is empty, it uses the machine's local timestamp in UTC.
	// The precision of timestamps is nanoseconds by default. It can be changed using WithTimestampPrecision.
	InsertRows(rows []Row) error
	// Close gracefully shutdowns by flushing any unwritten data to the underlying disk partition.
	Close() error
}

Storage provides goroutine safe capabilities of insertion into and retrieval from the time-series storage.

func NewStorage

func NewStorage(opts ...Option) (Storage, error)

NewStorage gives back a new storage, which stores time-series data in the process memory by default.

Give the WithDataPath option for running as a on-disk storage. Specify a directory with data already exists, then it will be read as the initial data.

Example (WithDataPath)

package main

import (
	"github.com/nakabonne/tstorage"
)

func main() {
	// It will make time-series data persistent under "./data".
	storage, err := tstorage.NewStorage(
		tstorage.WithDataPath("./data"),
	)
	if err != nil {
		panic(err)
	}
	storage.Close()
}

Example (WithPartitionDuration)

package main

import (
	"time"

	"github.com/nakabonne/tstorage"
)

func main() {
	storage, err := tstorage.NewStorage(
		tstorage.WithPartitionDuration(30*time.Minute),
		tstorage.WithTimestampPrecision(tstorage.Seconds),
	)
	if err != nil {
		panic(err)
	}
	defer storage.Close()
}

type TimestampPrecision

type TimestampPrecision string

TimestampPrecision represents precision of timestamps. See WithTimestampPrecision

const (
	Nanoseconds  TimestampPrecision = "ns"
	Microseconds TimestampPrecision = "us"
	Milliseconds TimestampPrecision = "ms"
	Seconds      TimestampPrecision = "s"
)