pilosa

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 8, 2017 License: Apache-2.0 Imports: 46 Imported by: 0

README

Build Status GoDoc Go Report Card license CLA Assistant GitHub release

An open source, distributed bitmap index.

Docs

See our Documentation for information about installing and working with Pilosa.

Getting Started

  1. Install Pilosa.

  2. Start Pilosa with the default configuration:

    pilosa server
    

    and verify that it's running:

    curl localhost:10101/nodes
    
  3. Follow along with the Sample Project to get a better understanding of Pilosa's capabilities.

Data Model

Check out how the Pilosa Data Model works.

Query Language

You can interact with Pilosa directly in the console using the Pilosa Query Language (PQL).

Client Libraries

There are supported libraries for the following languages:

Get Support

There are several channels available for you to reach out to us for support.

Contributing

Pilosa is an open source project. Please see our Contributing Guide for information about how to get involved.

Documentation

Overview

Package pilosa implements the core of the Pilosa distributed bitmap index. It contains all the domain objects, interfaces, and logic that defines pilosa.

Index

Constants

View Source
const (
	AttrTypeString = 1
	AttrTypeInt    = 2
	AttrTypeBool   = 3
	AttrTypeFloat  = 4
)

Attribute data type enum.

View Source
const (
	MessageTypeCreateSlice = 1
	MessageTypeCreateIndex = 2
	MessageTypeDeleteIndex = 3
	MessageTypeCreateFrame = 4
	MessageTypeDeleteFrame = 5
)

Broadcast message types.

View Source
const (
	// DefaultPartitionN is the default number of partitions in a cluster.
	DefaultPartitionN = 256

	// DefaultReplicaN is the default number of replicas per partition.
	DefaultReplicaN = 1
)
View Source
const (
	NodeStateUp   = "UP"
	NodeStateDown = "DOWN"
)

NodeState represents node state returned in /status endpoint for a node in the cluster.

View Source
const (
	// DefaultHost is the default hostname to use.
	DefaultHost = "localhost"

	// DefaultPort is the default port use with the hostname.
	DefaultPort = "10101"

	// DefaultClusterType sets the node intercommunication method.
	DefaultClusterType = "static"

	// DefaultInternalPort the port the nodes intercommunicate on.
	DefaultInternalPort = "14000"

	// DefaultMetrics sets the internal metrics to no op
	DefaultMetrics = "nop"

	// DefaultMaxWritesPerRequest is the default number of writes per request.
	DefaultMaxWritesPerRequest = 5000
)
View Source
const (
	DefaultFrame = "general"

	// MinThreshold is the lowest count to use in a Top-N operation when
	// looking for additional id/count pairs.
	MinThreshold = 1
)

DefaultFrame is the frame used if one is not specified.

View Source
const (
	// SliceWidth is the number of column IDs in a slice.
	SliceWidth = 1048576

	// SnapshotExt is the file extension used for an in-process snapshot.
	SnapshotExt = ".snapshotting"

	// CopyExt is the file extension used for the temp file used while copying.
	CopyExt = ".copying"

	// CacheExt is the file extension for persisted cache ids.
	CacheExt = ".cache"

	// HashBlockSize is the number of rows in a merkle hash block.
	HashBlockSize = 100
)
View Source
const (
	DefaultRowLabel       = "rowID"
	DefaultCacheType      = CacheTypeRanked
	DefaultInverseEnabled = false

	// Default ranked frame cache
	DefaultCacheSize = 50000
)

Default frame settings.

View Source
const (
	CacheTypeLRU    = "lru"
	CacheTypeRanked = "ranked"
)

Cache types.

View Source
const (
	DefaultAntiEntropyInterval = 10 * time.Minute
	DefaultPollingInterval     = 60 * time.Second
)

Default server settings.

View Source
const (
	ViewStandard = "standard"
	ViewInverse  = "inverse"
)

View layout modes.

View Source
const AttrBlockSize = 100

AttrBlockSize is the size of attribute blocks for anti-entropy.

View Source
const DefaultCacheFlushInterval = 1 * time.Minute

DefaultCacheFlushInterval is the default value for Fragment.CacheFlushInterval.

View Source
const (
	DefaultColumnLabel = "columnID"
)

Default index settings.

View Source
const (
	// DefaultFragmentMaxOpN is the default value for Fragment.MaxOpN.
	DefaultFragmentMaxOpN = 2000
)
View Source
const (
	// ThresholdFactor is used to calculate the threshold for new items entering the cache
	ThresholdFactor = 1.1
)
View Source
const TimeFormat = "2006-01-02T15:04"

TimeFormat is the go-style time format used to parse string dates.

Variables

View Source
var (
	ErrHostRequired = errors.New("host required")

	ErrIndexRequired = errors.New("index required")
	ErrIndexExists   = errors.New("index already exists")
	ErrIndexNotFound = errors.New("index not found")

	// ErrFrameRequired is returned when no frame is specified.
	ErrFrameRequired        = errors.New("frame required")
	ErrFrameExists          = errors.New("frame already exists")
	ErrFrameNotFound        = errors.New("frame not found")
	ErrFrameInverseDisabled = errors.New("frame inverse disabled")
	ErrColumnRowLabelEqual  = errors.New("column and row labels cannot be equal")

	ErrInvalidView      = errors.New("invalid view")
	ErrInvalidCacheType = errors.New("invalid cache type")

	ErrName  = errors.New("invalid index or frame's name, must match [a-z0-9_-]")
	ErrLabel = errors.New("invalid row or column label, must match [A-Za-z0-9_-]")

	// ErrFragmentNotFound is returned when a fragment does not exist.
	ErrFragmentNotFound = errors.New("fragment not found")
	ErrQueryRequired    = errors.New("query required")
	ErrTooManyWrites    = errors.New("too many write commands")
)

System errors.

View Source
var BuildTime = "not recorded"
View Source
var ErrInvalidTimeQuantum = errors.New("invalid time quantum")

ErrInvalidTimeQuantum is returned when parsing a time quantum.

View Source
var Expvar = expvar.NewMap("index")

Expvar global expvar map.

View Source
var NopBroadcastReceiver = &nopBroadcastReceiver{}

NopBroadcastReceiver is a no-op implementation of the BroadcastReceiver.

View Source
var Version = "v0.0.0"

Functions

func EncodeIndexes added in v0.4.0

func EncodeIndexes(a []*Index) []*internal.Index

EncodeIndexes converts a into its internal representation.

func IsInverseView

func IsInverseView(name string) bool

IsInverseView returns true if the view is used for storing an inverted representation.

func IsValidCacheType

func IsValidCacheType(v string) bool

IsValidCacheType returns true if v is a valid cache type.

func IsValidView

func IsValidView(name string) bool

IsValidView returns true if name is valid.

func MarshalImportPayload

func MarshalImportPayload(index, frame string, slice uint64, bits []Bit) ([]byte, error)

MarshalImportPayload marshalls the import parameters into a protobuf byte slice.

func MarshalMessage

func MarshalMessage(m proto.Message) ([]byte, error)

MarshalMessage encodes the protobuf message into a byte slice.

func NewRouter

func NewRouter(handler *Handler) *mux.Router

NewRouter creates a Gorilla Mux http router.

func Pos

func Pos(rowID, columnID uint64) uint64

Pos returns the row position of a row/column pair.

func UnionStringSlice

func UnionStringSlice(a, b []string) []string

UnionStringSlice returns a sorted set of tags which combine a & b.

func UnmarshalMessage

func UnmarshalMessage(buf []byte) (proto.Message, error)

UnmarshalMessage decodes the byte slice into a protobuf message.

func ValidateLabel

func ValidateLabel(label string) error

ValidateLabel ensures that the label is a valid format.

func ValidateName

func ValidateName(name string) error

ValidateName ensures that the name is a valid format.

func ViewByTimeUnit

func ViewByTimeUnit(name string, t time.Time, unit rune) string

ViewByTimeUnit returns the view name for time with a given quantum unit.

func ViewsByTime

func ViewsByTime(name string, t time.Time, q TimeQuantum) []string

ViewsByTime returns a list of views for a given timestamp.

func ViewsByTimeRange

func ViewsByTimeRange(name string, start, end time.Time, q TimeQuantum) []string

ViewsByTimeRange returns a list of views to traverse to query a time range.

Types

type AttrBlock

type AttrBlock struct {
	ID       uint64 `json:"id"`
	Checksum []byte `json:"checksum"`
}

AttrBlock represents a checksummed block of the attribute store.

type AttrBlocks

type AttrBlocks []AttrBlock

AttrBlocks represents a list of blocks.

func (AttrBlocks) Diff

func (a AttrBlocks) Diff(other []AttrBlock) []uint64

Diff returns a list of block ids that are different or are new in other. Block lists must be in sorted order.

type AttrStore

type AttrStore struct {
	// contains filtered or unexported fields
}

AttrStore represents a storage layer for attributes.

func NewAttrStore

func NewAttrStore(path string) *AttrStore

NewAttrStore returns a new instance of AttrStore.

func (*AttrStore) Attrs

func (s *AttrStore) Attrs(id uint64) (m map[string]interface{}, err error)

Attrs returns a set of attributes by ID.

func (*AttrStore) BlockData

func (s *AttrStore) BlockData(i uint64) (map[uint64]map[string]interface{}, error)

BlockData returns all data for a single block.

func (*AttrStore) Blocks

func (s *AttrStore) Blocks() ([]AttrBlock, error)

Blocks returns a list of all blocks in the store.

func (*AttrStore) Close

func (s *AttrStore) Close() error

Close closes the store.

func (*AttrStore) Open

func (s *AttrStore) Open() error

Open opens and initializes the store.

func (*AttrStore) Path

func (s *AttrStore) Path() string

Path returns path to the store's data file.

func (*AttrStore) SetAttrs

func (s *AttrStore) SetAttrs(id uint64, m map[string]interface{}) error

SetAttrs sets attribute values for a given ID.

func (*AttrStore) SetBulkAttrs

func (s *AttrStore) SetBulkAttrs(m map[uint64]map[string]interface{}) error

SetBulkAttrs sets attribute values for a set of ids.

type Bit

type Bit struct {
	RowID     uint64
	ColumnID  uint64
	Timestamp int64
}

Bit represents the location of a single bit.

type Bitmap

type Bitmap struct {

	// Attributes associated with the bitmap.
	Attrs map[string]interface{}
	// contains filtered or unexported fields
}

Bitmap represents a set of bits.

func NewBitmap

func NewBitmap(bits ...uint64) *Bitmap

NewBitmap returns a new instance of Bitmap.

func Union

func Union(bitmaps []*Bitmap) *Bitmap

Union performs a union on a slice of bitmaps.

func (*Bitmap) Bits

func (b *Bitmap) Bits() []uint64

Bits returns the bits in b as a slice of ints.

func (*Bitmap) ClearBit

func (b *Bitmap) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*Bitmap) Count

func (b *Bitmap) Count() uint64

Count returns the number of set bits in the bitmap.

func (*Bitmap) DecrementCount

func (b *Bitmap) DecrementCount(i uint64)

DecrementCount decrements the bitmap cached counter.

func (*Bitmap) Difference

func (b *Bitmap) Difference(other *Bitmap) *Bitmap

Difference returns the diff of b and other.

func (*Bitmap) IncrementCount

func (b *Bitmap) IncrementCount(i uint64)

IncrementCount increments the bitmap cached counter, note this is an optimization that assumes that the caller is aware the size increased.

func (*Bitmap) Intersect

func (b *Bitmap) Intersect(other *Bitmap) *Bitmap

Intersect returns the itersection of b and other.

func (*Bitmap) IntersectionCount

func (b *Bitmap) IntersectionCount(other *Bitmap) uint64

IntersectionCount returns the number of intersections between b and other.

func (*Bitmap) InvalidateCount

func (b *Bitmap) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*Bitmap) MarshalJSON

func (b *Bitmap) MarshalJSON() ([]byte, error)

MarshalJSON returns a JSON-encoded byte slice of b.

func (*Bitmap) Merge

func (b *Bitmap) Merge(other *Bitmap)

Merge merges data from other into b.

func (*Bitmap) SetBit

func (b *Bitmap) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*Bitmap) Union

func (b *Bitmap) Union(other *Bitmap) *Bitmap

Union returns the bitwise union of b and other.

type BitmapCache

type BitmapCache interface {
	Fetch(id uint64) (*Bitmap, bool)
	Add(id uint64, b *Bitmap)
}

BitmapCache provides an interface for caching full bitmaps.

type BitmapPair

type BitmapPair struct {
	ID    uint64
	Count uint64
}

BitmapPair represents a id/count pair with an associated identifier.

type BitmapPairs

type BitmapPairs []BitmapPair

BitmapPairs is a sortable list of BitmapPair objects.

func (BitmapPairs) Len

func (p BitmapPairs) Len() int

func (BitmapPairs) Less

func (p BitmapPairs) Less(i, j int) bool

func (BitmapPairs) Swap

func (p BitmapPairs) Swap(i, j int)

type BitmapSegment

type BitmapSegment struct {
	// contains filtered or unexported fields
}

BitmapSegment holds a subset of a bitmap. This could point to a mmapped roaring bitmap or an in-memory bitmap. The width of the segment will always match the slice width.

func (*BitmapSegment) Bits

func (s *BitmapSegment) Bits() []uint64

Bits returns a list of all bits set in the segment.

func (*BitmapSegment) ClearBit

func (s *BitmapSegment) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*BitmapSegment) Count

func (s *BitmapSegment) Count() uint64

Count returns the number of set bits in the bitmap.

func (*BitmapSegment) Difference

func (s *BitmapSegment) Difference(other *BitmapSegment) *BitmapSegment

Difference returns the diff of s and other.

func (*BitmapSegment) Intersect

func (s *BitmapSegment) Intersect(other *BitmapSegment) *BitmapSegment

Intersect returns the itersection of s and other.

func (*BitmapSegment) IntersectionCount

func (s *BitmapSegment) IntersectionCount(other *BitmapSegment) uint64

IntersectionCount returns the number of intersections between s and other.

func (*BitmapSegment) InvalidateCount

func (s *BitmapSegment) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*BitmapSegment) Merge

func (s *BitmapSegment) Merge(other *BitmapSegment)

Merge adds chunks from other to s. Chunks in s are overwritten if they exist in other.

func (*BitmapSegment) SetBit

func (s *BitmapSegment) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*BitmapSegment) Union

func (s *BitmapSegment) Union(other *BitmapSegment) *BitmapSegment

Union returns the bitwise union of s and other.

type Bits

type Bits []Bit

Bits represents a slice of bits.

func (Bits) ColumnIDs

func (p Bits) ColumnIDs() []uint64

ColumnIDs returns a slice of all the column IDs.

func (Bits) GroupBySlice

func (p Bits) GroupBySlice() map[uint64][]Bit

GroupBySlice returns a map of bits by slice.

func (Bits) Len

func (p Bits) Len() int

func (Bits) Less

func (p Bits) Less(i, j int) bool

func (Bits) RowIDs

func (p Bits) RowIDs() []uint64

RowIDs returns a slice of all the row IDs.

func (Bits) Swap

func (p Bits) Swap(i, j int)

func (Bits) Timestamps

func (p Bits) Timestamps() []int64

Timestamps returns a slice of all the timestamps.

type BitsByPos

type BitsByPos []Bit

BitsByPos represents a slice of bits sorted by internal position.

func (BitsByPos) Len

func (p BitsByPos) Len() int

func (BitsByPos) Less

func (p BitsByPos) Less(i, j int) bool

func (BitsByPos) Swap

func (p BitsByPos) Swap(i, j int)

type BroadcastHandler

type BroadcastHandler interface {
	ReceiveMessage(pb proto.Message) error
}

BroadcastHandler is the interface for the pilosa object which knows how to handle broadcast messages. (Hint: this is implemented by pilosa.Server)

type BroadcastReceiver

type BroadcastReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(BroadcastHandler) error
}

BroadcastReceiver is the interface for the object which will listen for and decode broadcast messages before passing them to pilosa to handle. The implementation of this could be an http server which listens for messages, gets the protobuf payload, and then passes it to BroadcastHandler.ReceiveMessage.

type Broadcaster

type Broadcaster interface {
	SendSync(pb proto.Message) error
	SendAsync(pb proto.Message) error
}

Broadcaster is an interface for broadcasting messages.

var NopBroadcaster Broadcaster

NopBroadcaster represents a Broadcaster that doesn't do anything.

type BufIterator

type BufIterator struct {
	// contains filtered or unexported fields
}

BufIterator wraps an iterator to provide the ability to unread values.

func NewBufIterator

func NewBufIterator(itr Iterator) *BufIterator

NewBufIterator returns a buffered iterator that wraps itr.

func (*BufIterator) Next

func (itr *BufIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next pair in the row. If a value has been buffered then it is returned and the buffer is cleared.

func (*BufIterator) Peek

func (itr *BufIterator) Peek() (rowID, columnID uint64, eof bool)

Peek reads the next value but leaves it on the buffer.

func (*BufIterator) Seek

func (itr *BufIterator) Seek(rowID, columnID uint64)

Seek moves to the first pair equal to or greater than pseek/bseek.

func (*BufIterator) Unread

func (itr *BufIterator) Unread()

Unread pushes previous pair on to the buffer. Panics if the buffer is already full.

type Cache

type Cache interface {
	Add(id uint64, n uint64)
	BulkAdd(id uint64, n uint64)
	Get(id uint64) uint64
	Len() int

	// Returns a list of all IDs.
	IDs() []uint64

	// Updates the cache, if necessary.
	Invalidate()

	// Rebuilds the cache
	Recalculate()

	// Returns an ordered list of the top ranked bitmaps.
	Top() []BitmapPair

	// SetStats defines the stats client used in the cache.
	SetStats(s StatsClient)
}

Cache represents a cache of counts.

type Client

type Client struct {

	// The client to use for HTTP communication.
	// Defaults to the http.DefaultClient.
	HTTPClient *http.Client
	// contains filtered or unexported fields
}

Client represents a client to the Pilosa cluster.

func NewClient

func NewClient(host string) (*Client, error)

NewClient returns a new instance of Client to connect to host.

func (*Client) BackupSlice

func (c *Client) BackupSlice(ctx context.Context, index, frame, view string, slice uint64) (io.ReadCloser, error)

BackupSlice retrieves a streaming backup from a single slice. This function tries slice owners until one succeeds.

func (*Client) BackupTo

func (c *Client) BackupTo(ctx context.Context, w io.Writer, index, frame, view string) error

BackupTo backs up an entire frame from a cluster to w.

func (*Client) BlockData

func (c *Client) BlockData(ctx context.Context, index, frame, view string, slice uint64, block int) ([]uint64, []uint64, error)

BlockData returns row/column id pairs for a block.

func (*Client) ColumnAttrDiff

func (c *Client) ColumnAttrDiff(ctx context.Context, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

ColumnAttrDiff returns data from differing blocks on a remote host.

func (*Client) CreateFrame

func (c *Client) CreateFrame(ctx context.Context, index, frame string, opt FrameOptions) error

CreateFrame creates a new frame on the server.

func (*Client) CreateIndex

func (c *Client) CreateIndex(ctx context.Context, index string, opt IndexOptions) error

CreateIndex creates a new index on the server.

func (*Client) ExecutePQL

func (c *Client) ExecutePQL(ctx context.Context, index, query string) (interface{}, error)

ExecutePQL executes query string against index on the server.

func (*Client) ExecuteQuery

func (c *Client) ExecuteQuery(ctx context.Context, index, query string, allowRedirect bool) (result interface{}, err error)

ExecuteQuery executes query against index on the server.

func (*Client) ExportCSV

func (c *Client) ExportCSV(ctx context.Context, index, frame, view string, slice uint64, w io.Writer) error

ExportCSV bulk exports data for a single slice from a host to CSV format.

func (*Client) FragmentBlocks

func (c *Client) FragmentBlocks(ctx context.Context, index, frame, view string, slice uint64) ([]FragmentBlock, error)

FragmentBlocks returns a list of block checksums for a fragment on a host. Only returns blocks which contain data.

func (*Client) FragmentNodes

func (c *Client) FragmentNodes(ctx context.Context, index string, slice uint64) ([]*Node, error)

FragmentNodes returns a list of nodes that own a slice.

func (*Client) FrameViews

func (c *Client) FrameViews(ctx context.Context, index, frame string) ([]string, error)

FrameViews returns a list of view names for a frame.

func (*Client) Host

func (c *Client) Host() string

Host returns the host the client was initialized with.

func (*Client) Import

func (c *Client) Import(ctx context.Context, index, frame string, slice uint64, bits []Bit) error

Import bulk imports bits for a single slice to a host.

func (*Client) MaxInverseSliceByIndex

func (c *Client) MaxInverseSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxInverseSliceByIndex returns the number of inverse slices on a server by index.

func (*Client) MaxSliceByIndex

func (c *Client) MaxSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxSliceByIndex returns the number of slices on a server by index.

func (*Client) RestoreFrame

func (c *Client) RestoreFrame(ctx context.Context, host, index, frame string) error

RestoreFrame restores an entire frame from a host in another cluster.

func (*Client) RestoreFrom

func (c *Client) RestoreFrom(ctx context.Context, r io.Reader, index, frame, view string) error

RestoreFrom restores a frame from a backup file to an entire cluster.

func (*Client) RowAttrDiff

func (c *Client) RowAttrDiff(ctx context.Context, index, frame string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

RowAttrDiff returns data from differing blocks on a remote host.

func (*Client) Schema

func (c *Client) Schema(ctx context.Context) ([]*IndexInfo, error)

Schema returns all index and frame schema information.

type Cluster

type Cluster struct {
	Nodes   []*Node
	NodeSet NodeSet

	// Hashing algorithm used to assign partitions to nodes.
	Hasher Hasher

	// The number of partitions in the cluster.
	PartitionN int

	// The number of replicas a partition has.
	ReplicaN int

	// Threshold for logging long-running queries
	LongQueryTime time.Duration
}

Cluster represents a collection of nodes.

func NewCluster

func NewCluster() *Cluster

NewCluster returns a new instance of Cluster with defaults.

func (*Cluster) FragmentNodes

func (c *Cluster) FragmentNodes(index string, slice uint64) []*Node

FragmentNodes returns a list of nodes that own a fragment.

func (*Cluster) NodeByHost

func (c *Cluster) NodeByHost(host string) *Node

NodeByHost returns a node reference by host.

func (*Cluster) NodeSetHosts

func (c *Cluster) NodeSetHosts() []string

NodeSetHosts returns the list of host strings for NodeSet members.

func (*Cluster) NodeStates

func (c *Cluster) NodeStates() map[string]string

NodeStates returns a map of nodes in the cluster with each node's state (UP/DOWN) as the value.

func (*Cluster) OwnsFragment

func (c *Cluster) OwnsFragment(host string, index string, slice uint64) bool

OwnsFragment returns true if a host owns a fragment.

func (*Cluster) OwnsSlices

func (c *Cluster) OwnsSlices(index string, maxSlice uint64, host string) []uint64

OwnsSlices find the set of slices owned by the node per Index

func (*Cluster) Partition

func (c *Cluster) Partition(index string, slice uint64) int

Partition returns the partition that a slice belongs to.

func (*Cluster) PartitionNodes

func (c *Cluster) PartitionNodes(partitionID int) []*Node

PartitionNodes returns a list of nodes that own a partition.

func (*Cluster) Status

func (c *Cluster) Status() *internal.ClusterStatus

Status returns the internal ClusterStatus representation.

type CmdIO

type CmdIO struct {
	Stdin  io.Reader
	Stdout io.Writer
	Stderr io.Writer
}

CmdIO holds standard unix inputs and outputs.

func NewCmdIO

func NewCmdIO(stdin io.Reader, stdout, stderr io.Writer) *CmdIO

NewCmdIO returns a new instance of CmdIO with inputs and outputs set to the arguments.

type ColumnAttrSet

type ColumnAttrSet struct {
	ID    uint64                 `json:"id"`
	Attrs map[string]interface{} `json:"attrs,omitempty"`
}

ColumnAttrSet represents a set of attributes for a vertical column in an index. Can have a set of attributes attached to it.

type Config

type Config struct {
	DataDir string `toml:"data-dir"`
	Host    string `toml:"host"`

	Cluster struct {
		ReplicaN        int      `toml:"replicas"`
		Type            string   `toml:"type"`
		Hosts           []string `toml:"hosts"`
		InternalHosts   []string `toml:"internal-hosts"`
		PollingInterval Duration `toml:"polling-interval"`
		InternalPort    string   `toml:"internal-port"`
		GossipSeed      string   `toml:"gossip-seed"`
		LongQueryTime   Duration `toml:"long-query-time"`
	} `toml:"cluster"`

	Plugins struct {
		Path string `toml:"path"`
	} `toml:"plugins"`

	AntiEntropy struct {
		Interval Duration `toml:"interval"`
	} `toml:"anti-entropy"`

	// Limits the number of mutating commands that can be in a single request to
	// the server. This includes SetBit, ClearBit, SetRowAttrs & SetColumnAttrs.
	MaxWritesPerRequest int `toml:"max-writes-per-request"`

	LogPath string `toml:"log-path"`

	Metric struct {
		Service         string   `toml:"service"`
		Host            string   `toml:"host"`
		PollingInterval Duration `toml:"interval"`
	} `toml:"metrics"`
}

Config represents the configuration for the command.

func NewConfig

func NewConfig() *Config

NewConfig returns an instance of Config with default options.

type Duration

type Duration time.Duration

Duration is a TOML wrapper type for time.Duration.

func (Duration) MarshalTOML added in v0.4.0

func (d Duration) MarshalTOML() ([]byte, error)

func (Duration) MarshalText

func (d Duration) MarshalText() (text []byte, err error)

MarshalText writes duration value in text format.

func (Duration) String

func (d Duration) String() string

String returns the string representation of the duration.

func (*Duration) UnmarshalText

func (d *Duration) UnmarshalText(text []byte) error

UnmarshalText parses a TOML value into a duration value.

type ExecOptions

type ExecOptions struct {
	Remote bool
}

ExecOptions represents an execution context for a single Execute() call.

type Executor

type Executor struct {
	Holder *Holder

	// Local hostname & cluster configuration.
	Host    string
	Cluster *Cluster

	// Client used for remote HTTP requests.
	HTTPClient *http.Client

	// Maximum number of SetBit() or ClearBit() commands per request.
	MaxWritesPerRequest int
}

Executor recursively executes calls in a PQL query across all slices.

func NewExecutor

func NewExecutor() *Executor

NewExecutor returns a new instance of Executor.

func (*Executor) Execute

func (e *Executor) Execute(ctx context.Context, index string, q *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)

Execute executes a PQL query.

type ExpvarStatsClient

type ExpvarStatsClient struct {
	// contains filtered or unexported fields
}

ExpvarStatsClient writes stats out to expvars.

func NewExpvarStatsClient

func NewExpvarStatsClient() *ExpvarStatsClient

NewExpvarStatsClient returns a new instance of ExpvarStatsClient. This client points at the root of the expvar index map.

func (*ExpvarStatsClient) Count

func (c *ExpvarStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs.

func (*ExpvarStatsClient) CountWithCustomTags added in v0.4.0

func (c *ExpvarStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (*ExpvarStatsClient) Gauge

func (c *ExpvarStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric.

func (*ExpvarStatsClient) Histogram

func (c *ExpvarStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric. This works the same as gauge for this client.

func (*ExpvarStatsClient) Set

func (c *ExpvarStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements.

func (*ExpvarStatsClient) SetLogger added in v0.4.0

func (c *ExpvarStatsClient) SetLogger(logger io.Writer)

SetLogger has no logger

func (*ExpvarStatsClient) Tags

func (c *ExpvarStatsClient) Tags() []string

Tags returns a sorted list of tags on the client.

func (*ExpvarStatsClient) Timing

func (c *ExpvarStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric.

func (*ExpvarStatsClient) WithTags

func (c *ExpvarStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new client with additional tags appended.

type Fragment

type Fragment struct {

	// Cache for row counts.
	CacheType string // passed in by frame

	CacheSize uint32

	// Number of operations performed before performing a snapshot.
	// This limits the size of fragments on the heap and flushes them to disk
	// so that they can be mmapped and heap utilization can be kept low.
	MaxOpN int

	// Writer used for out-of-band log entries.
	LogOutput io.Writer

	// Row attribute storage.
	// This is set by the parent frame unless overridden for testing.
	RowAttrStore *AttrStore
	// contains filtered or unexported fields
}

Fragment represents the intersection of a frame and slice in an index.

func NewFragment

func NewFragment(path, index, frame, view string, slice uint64) *Fragment

NewFragment returns a new instance of Fragment.

func (*Fragment) BlockData

func (f *Fragment) BlockData(id int) (rowIDs, columnIDs []uint64)

BlockData returns bits in a block as row & column ID pairs.

func (*Fragment) BlockN

func (f *Fragment) BlockN() int

BlockN returns the number of blocks in the fragment.

func (*Fragment) Blocks

func (f *Fragment) Blocks() []FragmentBlock

Blocks returns info for all blocks containing data.

func (*Fragment) Cache

func (f *Fragment) Cache() Cache

Cache returns the fragment's cache. This is not safe for concurrent use.

func (*Fragment) CachePath

func (f *Fragment) CachePath() string

CachePath returns the path to the fragment's cache data.

func (*Fragment) Checksum

func (f *Fragment) Checksum() []byte

Checksum returns a checksum for the entire fragment. If two fragments have the same checksum then they have the same data.

func (*Fragment) ClearBit

func (f *Fragment) ClearBit(rowID, columnID uint64) (bool, error)

ClearBit clears a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) Close

func (f *Fragment) Close() error

Close flushes the underlying storage, closes the file and unlocks it.

func (*Fragment) FlushCache

func (f *Fragment) FlushCache() error

FlushCache writes the cache data to disk.

func (*Fragment) ForEachBit

func (f *Fragment) ForEachBit(fn func(rowID, columnID uint64) error) error

ForEachBit executes fn for every bit set in the fragment. Errors returned from fn are passed through.

func (*Fragment) Frame

func (f *Fragment) Frame() string

Frame returns the frame the fragment was initialized with.

func (*Fragment) Import

func (f *Fragment) Import(rowIDs, columnIDs []uint64) error

Import bulk imports a set of bits and then snapshots the storage. This does not affect the fragment's cache.

func (*Fragment) Index

func (f *Fragment) Index() string

Index returns the index that the fragment was initialized with.

func (*Fragment) InvalidateChecksums

func (f *Fragment) InvalidateChecksums()

InvalidateChecksums clears all cached block checksums.

func (*Fragment) MergeBlock

func (f *Fragment) MergeBlock(id int, data []PairSet) (sets, clears []PairSet, err error)

MergeBlock compares the block's bits and computes a diff with another set of block bits. The state of a bit is determined by consensus from all blocks being considered.

For example, if 3 blocks are compared and two have a set bit and one has a cleared bit then the bit is considered cleared. The function returns the diff per incoming block so that all can be in sync.

func (*Fragment) Open

func (f *Fragment) Open() error

Open opens the underlying storage.

func (*Fragment) Path

func (f *Fragment) Path() string

Path returns the path the fragment was initialized with.

func (*Fragment) ReadFrom

func (f *Fragment) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom reads a data file from r and loads it into the fragment.

func (*Fragment) RecalculateCache

func (f *Fragment) RecalculateCache()

RecalculateCache rebuilds the cache regardless of invalidate time delay.

func (*Fragment) Row

func (f *Fragment) Row(rowID uint64) *Bitmap

Row returns a row by ID.

func (*Fragment) SetBit

func (f *Fragment) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) Slice

func (f *Fragment) Slice() uint64

Slice returns the slice the fragment was initialized with.

func (*Fragment) Snapshot

func (f *Fragment) Snapshot() error

Snapshot writes the storage bitmap to disk and reopens it.

func (*Fragment) Top

func (f *Fragment) Top(opt TopOptions) ([]Pair, error)

Top returns the top rows from the fragment. If opt.Src is specified then only rows which intersect src are returned. If opt.FilterValues exist then the row attribute specified by field is matched.

func (*Fragment) View

func (f *Fragment) View() string

View returns the view the fragment was initialized with.

func (*Fragment) WriteTo

func (f *Fragment) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the fragment's data to w.

type FragmentBlock

type FragmentBlock struct {
	ID       int    `json:"id"`
	Checksum []byte `json:"checksum"`
}

FragmentBlock represents info about a subsection of the rows in a block. This is used for comparing data in remote blocks for active anti-entropy.

type FragmentSyncer

type FragmentSyncer struct {
	Fragment *Fragment

	Host    string
	Cluster *Cluster

	Closing <-chan struct{}
}

FragmentSyncer syncs a local fragment to one on a remote host.

func (*FragmentSyncer) SyncFragment

func (s *FragmentSyncer) SyncFragment() error

SyncFragment compares checksums for the local and remote fragments and then merges any blocks which have differences.

type Frame

type Frame struct {
	Stats StatsClient

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Frame represents a container for views.

func NewFrame

func NewFrame(path, index, name string) (*Frame, error)

NewFrame returns a new instance of frame.

func (*Frame) CacheSize

func (f *Frame) CacheSize() uint32

CacheSize returns the ranked frame cache size.

func (*Frame) CacheType

func (f *Frame) CacheType() string

CacheType returns the caching mode for the frame.

func (*Frame) ClearBit

func (f *Frame) ClearBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

ClearBit clears a bit within the frame.

func (*Frame) Close

func (f *Frame) Close() error

Close closes the frame and its views.

func (*Frame) CreateViewIfNotExists

func (f *Frame) CreateViewIfNotExists(name string) (*View, error)

CreateViewIfNotExists returns the named view, creating it if necessary.

func (*Frame) Import

func (f *Frame) Import(rowIDs, columnIDs []uint64, timestamps []*time.Time) error

Import bulk imports data.

func (*Frame) Index

func (f *Frame) Index() string

Index returns the index name the frame was initialized with.

func (*Frame) InverseEnabled

func (f *Frame) InverseEnabled() bool

InverseEnabled returns true if an inverse view is available.

func (*Frame) MaxInverseSlice

func (f *Frame) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the frame.

func (*Frame) MaxSlice

func (f *Frame) MaxSlice() uint64

MaxSlice returns the max slice in the frame.

func (*Frame) Name

func (f *Frame) Name() string

Name returns the name the frame was initialized with.

func (*Frame) Open

func (f *Frame) Open() error

Open opens and initializes the frame.

func (*Frame) Options

func (f *Frame) Options() FrameOptions

Options returns all options for this frame.

func (*Frame) Path

func (f *Frame) Path() string

Path returns the path the frame was initialized with.

func (*Frame) RowAttrStore

func (f *Frame) RowAttrStore() *AttrStore

RowAttrStore returns the attribute storage.

func (*Frame) RowLabel

func (f *Frame) RowLabel() string

RowLabel returns the row label.

func (*Frame) SetBit

func (f *Frame) SetBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

SetBit sets a bit on a view within the frame.

func (*Frame) SetCacheSize

func (f *Frame) SetCacheSize(v uint32) error

SetCacheSize sets the cache size for ranked fames. Persists to meta file on update. defaults to DefaultCacheSize 50000

func (*Frame) SetRowLabel

func (f *Frame) SetRowLabel(v string) error

SetRowLabel sets the row labels. Persists to meta file on update.

func (*Frame) SetTimeQuantum

func (f *Frame) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the time quantum for the frame.

func (*Frame) TimeQuantum

func (f *Frame) TimeQuantum() TimeQuantum

TimeQuantum returns the time quantum for the frame.

func (*Frame) View

func (f *Frame) View(name string) *View

View returns a view in the frame by name.

func (*Frame) ViewPath

func (f *Frame) ViewPath(name string) string

ViewPath returns the path to a view in the frame.

func (*Frame) Views

func (f *Frame) Views() []*View

Views returns a list of all views in the frame.

type FrameInfo

type FrameInfo struct {
	Name  string      `json:"name"`
	Views []*ViewInfo `json:"views,omitempty"`
}

FrameInfo represents schema information for a frame.

type FrameOptions

type FrameOptions struct {
	RowLabel       string      `json:"rowLabel,omitempty"`
	InverseEnabled bool        `json:"inverseEnabled,omitempty"`
	CacheType      string      `json:"cacheType,omitempty"`
	CacheSize      uint32      `json:"cacheSize,omitempty"`
	TimeQuantum    TimeQuantum `json:"timeQuantum,omitempty"`
}

FrameOptions represents options to set when initializing a frame.

func (*FrameOptions) Encode

func (o *FrameOptions) Encode() *internal.FrameMeta

Encode converts o into its internal representation.

type Handler

type Handler struct {
	Holder        *Holder
	Broadcaster   Broadcaster
	StatusHandler StatusHandler

	// Local hostname & cluster configuration.
	Host    string
	Cluster *Cluster

	Router *mux.Router

	// The execution engine for running queries.
	Executor interface {
		Execute(context context.Context, index string, query *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)
	}

	// The writer for any logging.
	LogOutput io.Writer
}

Handler represents an HTTP handler.

func NewHandler

func NewHandler() *Handler

NewHandler returns a new instance of Handler with a default logger.

func (*Handler) ServeHTTP

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP handles an HTTP request.

type Hasher

type Hasher interface {
	// Hashes the key into a number between [0,N).
	Hash(key uint64, n int) int
}

Hasher represents an interface to hash integers into buckets.

func NewHasher

func NewHasher() Hasher

NewHasher returns a new instance of the default hasher.

type Holder

type Holder struct {
	Broadcaster Broadcaster

	// Stats
	Stats StatsClient

	// Data directory path.
	Path string

	// The interval at which the cached row ids are persisted to disk.
	CacheFlushInterval time.Duration

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Holder represents a container for indexes.

func NewHolder

func NewHolder() *Holder

NewHolder returns a new instance of Holder.

func (*Holder) Close

func (h *Holder) Close() error

Close closes all open fragments.

func (*Holder) CreateIndex

func (h *Holder) CreateIndex(name string, opt IndexOptions) (*Index, error)

CreateIndex creates an index. An error is returned if the index already exists.

func (*Holder) CreateIndexIfNotExists

func (h *Holder) CreateIndexIfNotExists(name string, opt IndexOptions) (*Index, error)

CreateIndexIfNotExists returns an index by name. The index is created if it does not already exist.

func (*Holder) DeleteIndex

func (h *Holder) DeleteIndex(name string) error

DeleteIndex removes an index from the holder.

func (*Holder) Fragment

func (h *Holder) Fragment(index, frame, view string, slice uint64) *Fragment

Fragment returns the fragment for an index, frame & slice.

func (*Holder) Frame

func (h *Holder) Frame(index, name string) *Frame

Frame returns the frame for an index and name.

func (*Holder) Index

func (h *Holder) Index(name string) *Index

Index returns the index by name.

func (*Holder) IndexPath

func (h *Holder) IndexPath(name string) string

IndexPath returns the path where a given index is stored.

func (*Holder) Indexes

func (h *Holder) Indexes() []*Index

Indexes returns a list of all indexes in the holder.

func (*Holder) MaxInverseSlices

func (h *Holder) MaxInverseSlices() map[string]uint64

MaxInverseSlices returns MaxInverseSlice map for all indexes.

func (*Holder) MaxSlices

func (h *Holder) MaxSlices() map[string]uint64

MaxSlices returns MaxSlice map for all indexes.

func (*Holder) Open

func (h *Holder) Open() error

Open initializes the root data directory for the holder.

func (*Holder) Schema

func (h *Holder) Schema() []*IndexInfo

Schema returns schema data for all indexes and frames.

func (*Holder) View

func (h *Holder) View(index, frame, name string) *View

View returns the view for an index, frame, and name.

type HolderSyncer

type HolderSyncer struct {
	Holder *Holder

	Host    string
	Cluster *Cluster

	// Signals that the sync should stop.
	Closing <-chan struct{}
}

HolderSyncer is an active anti-entropy tool that compares the local holder with a remote holder based on block checksums and resolves differences.

func (*HolderSyncer) IsClosing

func (s *HolderSyncer) IsClosing() bool

IsClosing returns true if the syncer has been marked to close.

func (*HolderSyncer) SyncHolder

func (s *HolderSyncer) SyncHolder() error

SyncHolder compares the holder on host with the local holder and resolves differences.

type Index

type Index struct {
	Stats StatsClient

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Index represents a container for frames.

func NewIndex

func NewIndex(path, name string) (*Index, error)

NewIndex returns a new instance of Index.

func (*Index) Close

func (i *Index) Close() error

Close closes the index and its frames.

func (*Index) ColumnAttrStore

func (i *Index) ColumnAttrStore() *AttrStore

ColumnAttrStore returns the storage for column attributes.

func (*Index) ColumnLabel

func (i *Index) ColumnLabel() string

ColumnLabel returns the column label.

func (*Index) CreateFrame

func (i *Index) CreateFrame(name string, opt FrameOptions) (*Frame, error)

CreateFrame creates a frame.

func (*Index) CreateFrameIfNotExists

func (i *Index) CreateFrameIfNotExists(name string, opt FrameOptions) (*Frame, error)

CreateFrameIfNotExists creates a frame with the given options if it doesn't exist.

func (*Index) DeleteFrame

func (i *Index) DeleteFrame(name string) error

DeleteFrame removes a frame from the index.

func (*Index) Frame

func (i *Index) Frame(name string) *Frame

Frame returns a frame in the index by name.

func (*Index) FramePath

func (i *Index) FramePath(name string) string

FramePath returns the path to a frame in the index.

func (*Index) Frames

func (i *Index) Frames() []*Frame

Frames returns a list of all frames in the index.

func (*Index) MaxInverseSlice

func (i *Index) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the index according to this node.

func (*Index) MaxSlice

func (i *Index) MaxSlice() uint64

MaxSlice returns the max slice in the index according to this node.

func (*Index) Name

func (i *Index) Name() string

Name returns name of the index.

func (*Index) Open

func (i *Index) Open() error

Open opens and initializes the index.

func (*Index) Path

func (i *Index) Path() string

Path returns the path the index was initialized with.

func (*Index) SetColumnLabel

func (i *Index) SetColumnLabel(v string) error

SetColumnLabel sets the column label. Persists to meta file on update.

func (*Index) SetRemoteMaxInverseSlice

func (i *Index) SetRemoteMaxInverseSlice(v uint64)

SetRemoteMaxInverseSlice sets the remote max inverse slice value received from another node.

func (*Index) SetRemoteMaxSlice

func (i *Index) SetRemoteMaxSlice(newmax uint64)

SetRemoteMaxSlice sets the remote max slice value received from another node.

func (*Index) SetTimeQuantum

func (i *Index) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the default time quantum for the index.

func (*Index) TimeQuantum

func (i *Index) TimeQuantum() TimeQuantum

TimeQuantum returns the default time quantum for the index.

type IndexInfo

type IndexInfo struct {
	Name   string       `json:"name"`
	Frames []*FrameInfo `json:"frames"`
}

IndexInfo represents schema information for an index.

func MergeSchemas

func MergeSchemas(a, b []*IndexInfo) []*IndexInfo

MergeSchemas combines indexes and frames from a and b into one schema.

type IndexOptions

type IndexOptions struct {
	ColumnLabel string      `json:"columnLabel,omitempty"`
	TimeQuantum TimeQuantum `json:"timeQuantum,omitempty"`
}

IndexOptions represents options to set when initializing an index.

func (*IndexOptions) Encode

func (o *IndexOptions) Encode() *internal.IndexMeta

Encode converts o into its internal representation.

type Iterator

type Iterator interface {
	Seek(rowID, columnID uint64)
	Next() (rowID, columnID uint64, eof bool)
}

Iterator is an interface for looping over row/column pairs.

type LRUCache

type LRUCache struct {
	// contains filtered or unexported fields
}

LRUCache represents a least recently used Cache implemenation.

func NewLRUCache

func NewLRUCache(maxEntries uint32) *LRUCache

NewLRUCache returns a new instance of LRUCache.

func (*LRUCache) Add

func (c *LRUCache) Add(id, n uint64)

Add adds a count to the cache.

func (*LRUCache) BulkAdd

func (c *LRUCache) BulkAdd(id, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*LRUCache) Get

func (c *LRUCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*LRUCache) IDs

func (c *LRUCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*LRUCache) Invalidate

func (c *LRUCache) Invalidate()

Invalidate is a no-op.

func (*LRUCache) Len

func (c *LRUCache) Len() int

Len returns the number of items in the cache.

func (*LRUCache) Recalculate

func (c *LRUCache) Recalculate()

Recalculate is a no-op.

func (*LRUCache) SetStats added in v0.4.0

func (c *LRUCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*LRUCache) Top

func (c *LRUCache) Top() []BitmapPair

Top returns all counts in the cache.

type LimitIterator

type LimitIterator struct {
	// contains filtered or unexported fields
}

LimitIterator wraps an Iterator and limits it to a max column/row pair.

func NewLimitIterator

func NewLimitIterator(itr Iterator, maxRowID, maxColumnID uint64) *LimitIterator

NewLimitIterator returns a new LimitIterator.

func (*LimitIterator) Next

func (itr *LimitIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair. If the underlying iterator returns a pair higher than the max then EOF is returned.

func (*LimitIterator) Seek

func (itr *LimitIterator) Seek(rowID, columnID uint64)

Seek moves the underlying iterator to a column/row pair.

type MultiStatsClient

type MultiStatsClient []StatsClient

MultiStatsClient joins multiple stats clients together.

func (MultiStatsClient) Count

func (a MultiStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs per second on all clients.

func (MultiStatsClient) CountWithCustomTags added in v0.4.0

func (a MultiStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (MultiStatsClient) Gauge

func (a MultiStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric on all clients.

func (MultiStatsClient) Histogram

func (a MultiStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric on all clients.

func (MultiStatsClient) Set

func (a MultiStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements on all clients.

func (MultiStatsClient) SetLogger added in v0.4.0

func (a MultiStatsClient) SetLogger(logger io.Writer)

SetLogger Sets the StatsD logger output type

func (MultiStatsClient) Tags

func (a MultiStatsClient) Tags() []string

Tags returns tags from the first client.

func (MultiStatsClient) Timing

func (a MultiStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric on all clients.

func (MultiStatsClient) WithTags

func (a MultiStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new set of clients with the additional tags.

type Node

type Node struct {
	Host         string `json:"host"`
	InternalHost string `json:"internalHost"`
	// contains filtered or unexported fields
}

Node represents a node in the cluster.

func (*Node) SetState

func (n *Node) SetState(s string)

SetState sets the Node.status.state.

func (*Node) SetStatus

func (n *Node) SetStatus(s *internal.NodeStatus)

SetStatus sets the NodeStatus.

type NodeSet

type NodeSet interface {
	// Returns a list of all Nodes in the cluster
	Nodes() []*Node

	// Open starts any network activity implemented by the NodeSet
	Open() error
}

NodeSet represents an interface for Node membership and inter-node communication.

type Nodes

type Nodes []*Node

Nodes represents a list of nodes.

func (Nodes) Clone

func (a Nodes) Clone() []*Node

Clone returns a shallow copy of nodes.

func (Nodes) Contains

func (a Nodes) Contains(n *Node) bool

Contains returns true if a node exists in the list.

func (Nodes) ContainsHost

func (a Nodes) ContainsHost(host string) bool

ContainsHost returns true if host matches one of the node's host.

func (Nodes) Filter

func (a Nodes) Filter(n *Node) []*Node

Filter returns a new list of nodes with node removed.

func (Nodes) FilterHost

func (a Nodes) FilterHost(host string) []*Node

FilterHost returns a new list of nodes with host removed.

func (Nodes) Hosts

func (a Nodes) Hosts() []string

Hosts returns a list of all hostnames.

type Pair

type Pair struct {
	ID    uint64 `json:"id"`
	Count uint64 `json:"count"`
}

Pair holds an id/count pair.

type PairHeap

type PairHeap struct {
	Pairs
}

PairHeap is a heap implementation over a group of Pairs.

func (PairHeap) Less

func (p PairHeap) Less(i, j int) bool

Less implemets the Sort interface. reports whether the element with index i should sort before the element with index j.

type PairSet

type PairSet struct {
	RowIDs    []uint64
	ColumnIDs []uint64
}

PairSet is a list of equal length row and column id lists.

type Pairs

type Pairs []Pair

Pairs is a sortable slice of Pair objects.

func (Pairs) Add

func (p Pairs) Add(other []Pair) []Pair

Add merges other into p and returns a new slice.

func (Pairs) Keys

func (p Pairs) Keys() []uint64

Keys returns a slice of all keys in p.

func (Pairs) Len

func (p Pairs) Len() int

func (Pairs) Less

func (p Pairs) Less(i, j int) bool

func (*Pairs) Pop

func (p *Pairs) Pop() interface{}

Pop removes the minimum element from the Pair slice.

func (*Pairs) Push

func (p *Pairs) Push(x interface{})

Push appends the element onto the Pair slice.

func (Pairs) String

func (p Pairs) String() string

func (Pairs) Swap

func (p Pairs) Swap(i, j int)

type QueryRequest

type QueryRequest struct {
	// Index to execute query against.
	Index string

	// The query string to parse and execute.
	Query string

	// The slices to include in the query execution.
	// If empty, all slices are included.
	Slices []uint64

	// Return column attributes, if true.
	ColumnAttrs bool

	// Time granularity to use with the timestamp.
	Quantum TimeQuantum

	// If true, indicates that query is part of a larger distributed query.
	// If false, this request is on the originating node.
	Remote bool
}

QueryRequest represent a request to process a query.

type QueryResponse

type QueryResponse struct {
	// Result for each top-level query call.
	// Can be a Bitmap, Pairs, or uint64.
	Results []interface{}

	// Set of column attribute objects matching IDs returned in Result.
	ColumnAttrSets []*ColumnAttrSet

	// Error during parsing or execution.
	Err error
}

QueryResponse represent a response from a processed query.

func (*QueryResponse) MarshalJSON

func (resp *QueryResponse) MarshalJSON() ([]byte, error)

MarshalJSON marshals QueryResponse into a JSON-encoded byte slice

type RankCache

type RankCache struct {
	// contains filtered or unexported fields
}

RankCache represents a cache with sorted entries.

func NewRankCache

func NewRankCache(maxEntries uint32) *RankCache

NewRankCache returns a new instance of RankCache.

func (*RankCache) Add

func (c *RankCache) Add(id uint64, n uint64)

Add adds a count to the cache.

func (*RankCache) BulkAdd

func (c *RankCache) BulkAdd(id uint64, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*RankCache) Get

func (c *RankCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*RankCache) IDs

func (c *RankCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*RankCache) Invalidate

func (c *RankCache) Invalidate()

Invalidate recalculates the the entries by rank.

func (*RankCache) Len

func (c *RankCache) Len() int

Len returns the number of items in the cache.

func (*RankCache) ReadFrom

func (c *RankCache) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom read from r into the cache.

func (*RankCache) Recalculate

func (c *RankCache) Recalculate()

Recalculate rebuilds the cache.

func (*RankCache) SetStats added in v0.4.0

func (c *RankCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*RankCache) Top

func (c *RankCache) Top() []BitmapPair

Top returns an ordered list of pairs.

func (*RankCache) WriteTo

func (c *RankCache) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the cache to w.

type RoaringIterator

type RoaringIterator struct {
	// contains filtered or unexported fields
}

RoaringIterator converts a roaring.Iterator to output column/row pairs.

func NewRoaringIterator

func NewRoaringIterator(itr *roaring.Iterator) *RoaringIterator

NewRoaringIterator returns a new iterator wrapping itr.

func (*RoaringIterator) Next

func (itr *RoaringIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next column/row ID pair.

func (*RoaringIterator) Seek

func (itr *RoaringIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a pair matching bseek/pseek. If the pair is not found then it moves to the next pair.

type Server

type Server struct {

	// Data storage and HTTP interface.
	Holder            *Holder
	Handler           *Handler
	Broadcaster       Broadcaster
	BroadcastReceiver BroadcastReceiver

	// Cluster configuration.
	// Host is replaced with actual host after opening if port is ":0".
	Network string
	Host    string
	Cluster *Cluster

	// Background monitoring intervals.
	AntiEntropyInterval time.Duration
	PollingInterval     time.Duration
	MetricInterval      time.Duration

	// Misc options.
	MaxWritesPerRequest int

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Server represents a holder wrapped by a running HTTP server.

func NewServer

func NewServer() *Server

NewServer returns a new instance of Server.

func (*Server) Addr

func (s *Server) Addr() net.Addr

Addr returns the address of the listener.

func (*Server) Close

func (s *Server) Close() error

Close closes the server and waits for it to shutdown.

func (*Server) ClusterStatus

func (s *Server) ClusterStatus() (proto.Message, error)

ClusterStatus returns the NodeState for all nodes in the cluster.

func (*Server) HandleRemoteStatus

func (s *Server) HandleRemoteStatus(pb proto.Message) error

HandleRemoteStatus receives incoming NodeState from remote nodes.

func (*Server) LocalStatus

func (s *Server) LocalStatus() (proto.Message, error)

LocalStatus returns the state of the local node as well as the holder (indexes/frames) according to the local node. In a gossip implementation, memberlist.Delegate.LocalState() uses this. Server implements StatusHandler.

func (*Server) Open

func (s *Server) Open() error

Open opens and initializes the server.

func (*Server) ReceiveMessage

func (s *Server) ReceiveMessage(pb proto.Message) error

ReceiveMessage represents an implementation of BroadcastHandler.

type SimpleCache

type SimpleCache struct {
	// contains filtered or unexported fields
}

SimpleCache implements BitmapCache it is meant to be a short-lived cache for cases where writes are continuing to access the same bit within a short time frame (i.e. good for write-heavy loads) A read-heavy use case would cause the cache to get bigger, potentially causing the node to run out of memory.

func (*SimpleCache) Add

func (s *SimpleCache) Add(id uint64, b *Bitmap)

Add adds the bitmap to the cache, keyed on the id.

func (*SimpleCache) Fetch

func (s *SimpleCache) Fetch(id uint64) (*Bitmap, bool)

Fetch retrieves the bitmap at the id in the cache.

type SliceIterator

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator iterates over a pair of row/column ID slices.

func NewSliceIterator

func NewSliceIterator(rowIDs, columnIDs []uint64) *SliceIterator

NewSliceIterator returns an iterator to iterate over a set of row/column ID pairs. Both slices MUST have an equal length. Otherwise the function will panic.

func (*SliceIterator) Next

func (itr *SliceIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair.

func (*SliceIterator) Seek

func (itr *SliceIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a given pair. If the pair is not found, the iterator seeks to the next pair.

type StaticNodeSet

type StaticNodeSet struct {
	// contains filtered or unexported fields
}

StaticNodeSet represents a basic NodeSet for testing.

func NewStaticNodeSet

func NewStaticNodeSet() *StaticNodeSet

NewStaticNodeSet creates a statically defined NodeSet.

func (*StaticNodeSet) Join

func (s *StaticNodeSet) Join(nodes []*Node) error

Join sets the NodeSet nodes to the slice of Nodes passed in.

func (*StaticNodeSet) Nodes

func (s *StaticNodeSet) Nodes() []*Node

Nodes implements the NodeSet interface and returns a list of nodes in the cluster.

func (*StaticNodeSet) Open

func (s *StaticNodeSet) Open() error

Open implements the NodeSet interface to start network activity, but for a static NodeSet it does nothing.

type StatsClient

type StatsClient interface {
	// Returns a sorted list of tags on the client.
	Tags() []string

	// Returns a new client with additional tags appended.
	WithTags(tags ...string) StatsClient

	// Tracks the number of times something occurs per second.
	Count(name string, value int64, rate float64)

	// Tracks the number of times something occurs per second with custom tags
	CountWithCustomTags(name string, value int64, rate float64, tags []string)

	// Sets the value of a metric.
	Gauge(name string, value float64, rate float64)

	// Tracks statistical distribution of a metric.
	Histogram(name string, value float64, rate float64)

	// Tracks number of unique elements.
	Set(name string, value string, rate float64)

	// Tracks timing information for a metric.
	Timing(name string, value time.Duration, rate float64)

	// SetLogger Set the logger output type
	SetLogger(logger io.Writer)
}

StatsClient represents a client to a stats server.

var NopStatsClient StatsClient

NopStatsClient represents a client that doesn't do anything.

type StatusHandler

type StatusHandler interface {
	LocalStatus() (proto.Message, error)
	ClusterStatus() (proto.Message, error)
	HandleRemoteStatus(proto.Message) error
}

StatusHandler specifies two methods which an object must implement to share state in the cluster. These are used by the GossipNodeSet to implement the LocalState and MergeRemoteState methods of memberlist.Delegate

type TimeQuantum

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func ParseTimeQuantum

func ParseTimeQuantum(v string) (TimeQuantum, error)

ParseTimeQuantum parses v into a time quantum.

func (TimeQuantum) HasDay

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (TimeQuantum) Valid

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type TopOptions

type TopOptions struct {
	// Number of rows to return.
	N int

	// Bitmap to intersect with.
	Src *Bitmap

	// Specific rows to filter against.
	RowIDs       []uint64
	MinThreshold uint64

	// Filter field name & values.
	FilterField       string
	FilterValues      []interface{}
	TanimotoThreshold uint64
}

TopOptions represents options passed into the Top() function.

type View

type View struct {
	RowAttrStore *AttrStore
	LogOutput    io.Writer
	// contains filtered or unexported fields
}

View represents a container for frame data.

func NewView

func NewView(path, index, frame, name string, cacheSize uint32) *View

NewView returns a new instance of View.

func (*View) ClearBit

func (v *View) ClearBit(rowID, columnID uint64) (changed bool, err error)

ClearBit clears a bit within the view.

func (*View) Close

func (v *View) Close() error

Close closes the view and its fragments.

func (*View) CreateFragmentIfNotExists

func (v *View) CreateFragmentIfNotExists(slice uint64) (*Fragment, error)

CreateFragmentIfNotExists returns a fragment in the view by slice.

func (*View) Fragment

func (v *View) Fragment(slice uint64) *Fragment

Fragment returns a fragment in the view by slice.

func (*View) FragmentPath

func (v *View) FragmentPath(slice uint64) string

FragmentPath returns the path to a fragment in the view.

func (*View) Fragments

func (v *View) Fragments() []*Fragment

Fragments returns a list of all fragments in the view.

func (*View) Frame

func (v *View) Frame() string

Frame returns the frame name the view was initialized with.

func (*View) Index

func (v *View) Index() string

Index returns the index name the view was initialized with.

func (*View) MaxSlice

func (v *View) MaxSlice() uint64

MaxSlice returns the max slice in the view.

func (*View) Name

func (v *View) Name() string

Name returns the name the view was initialized with.

func (*View) Open

func (v *View) Open() error

Open opens and initializes the view.

func (*View) Path

func (v *View) Path() string

Path returns the path the view was initialized with.

func (*View) SetBit

func (v *View) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit within the view.

type ViewInfo

type ViewInfo struct {
	Name string `json:"name"`
}

ViewInfo represents schema information for a view.

Directories

Path Synopsis
cmd
Package cmd contains all the pilosa subcommand definitions (1 per file).
Package cmd contains all the pilosa subcommand definitions (1 per file).
pilosa
This is the entrypoint for the Pilosa binary.
This is the entrypoint for the Pilosa binary.
package ctl contains all pilosa subcommands other than 'server'.
package ctl contains all pilosa subcommands other than 'server'.
Package internal is a generated protocol buffer package.
Package internal is a generated protocol buffer package.
Package pql defines the Pilosa Query Language.
Package pql defines the Pilosa Query Language.
package roaring implements roaring bitmaps with support for incremental changes.
package roaring implements roaring bitmaps with support for incremental changes.
Package server contains the `pilosa server` subcommand which runs Pilosa itself.
Package server contains the `pilosa server` subcommand which runs Pilosa itself.
Package statik contains static assets for the Web UI.
Package statik contains static assets for the Web UI.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL