Help us shape the future of Go! Take the 2026 Go Developer Survey

pilosa

package module
v0.8.1-0...-fa4e543 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 8, 2018 License: Apache-2.0 Imports: 52 Imported by: 0

README

Build Status GoDoc Go Report Card license CLA Assistant GitHub release

An open source, distributed bitmap index.

Want to contribute? One of the easiest ways is to tell us how you're using (or want to use) Pilosa. We learn from every discussion!

Docs

See our Documentation for information about installing and working with Pilosa.

Getting Started

  1. Install Pilosa.

  2. Start Pilosa with the default configuration:

    pilosa server

    and verify that it's running:

    curl localhost:10101/nodes

  3. Follow along with the Sample Project to get a better understanding of Pilosa's capabilities.

Data Model

Check out how the Pilosa Data Model works.

Query Language

You can interact with Pilosa directly in the console using the Pilosa Query Language (PQL).

Client Libraries

There are supported libraries for the following languages:

Get Support

There are several channels available for you to reach out to us for support.

Contributing

Pilosa is an open source project. Please see our Contributing Guide for information about how to get involved.

Documentation

Overview

Package pilosa implements the core of the Pilosa distributed bitmap index. It contains all the domain objects, interfaces, and logic that defines pilosa.

Index

Constants

View Source 
const (
	AttrTypeString = 1
	AttrTypeInt    = 2
	AttrTypeBool   = 3
	AttrTypeFloat  = 4
)

Attribute data type enum.

View Source 
const (
	MessageTypeCreateSlice = iota
	MessageTypeCreateIndex
	MessageTypeDeleteIndex
	MessageTypeCreateFrame
	MessageTypeDeleteFrame
	MessageTypeCreateView
	MessageTypeDeleteView
	MessageTypeCreateField
	MessageTypeDeleteField
	MessageTypeCreateInputDefinition
	MessageTypeDeleteInputDefinition
	MessageTypeClusterStatus
	MessageTypeResizeInstruction
	MessageTypeResizeInstructionComplete
	MessageTypeSetCoordinator
	MessageTypeNodeState
	MessageTypeRecalculateCaches
	MessageTypeNodeEvent
)

Broadcast message types.

View Source 
const (
	// DefaultPartitionN is the default number of partitions in a cluster.
	DefaultPartitionN = 256

	// DefaultReplicaN is the default number of replicas per partition.
	DefaultReplicaN = 1

	// ClusterState represents the state returned in the /status endpoint.
	ClusterStateStarting = "STARTING"
	ClusterStateNormal   = "NORMAL"
	ClusterStateResizing = "RESIZING"

	// NodeState represents the state of a node during startup.
	NodeStateLoading = "LOADING"
	NodeStateReady   = "READY"

	// ResizeJob states.
	ResizeJobStateRunning = "RUNNING"
	// Final states.
	ResizeJobStateDone    = "DONE"
	ResizeJobStateAborted = "ABORTED"

	ResizeJobActionAdd    = "ADD"
	ResizeJobActionRemove = "REMOVE"
)
View Source 
const (
	ClusterNone   = ""
	ClusterStatic = "static"
	ClusterGossip = "gossip"
)

Cluster types.

View Source 
const (
	// DefaultDataDir is the default data directory.
	DefaultDataDir = "~/.pilosa"

	// DefaultHost is the default hostname to use.
	DefaultHost = "localhost"

	// DefaultPort is the default port to use with the hostname.
	DefaultPort = "10101"

	// DefaultClusterDisabled sets the node intercommunication method.
	DefaultClusterDisabled = false

	// DefaultMetrics sets the internal metrics to no-op.
	DefaultMetrics = "nop"

	// DefaultMaxWritesPerRequest is the default number of writes per request.
	DefaultMaxWritesPerRequest = 5000

	// Port indicates the port to which pilosa should bind for internal state sharing.
	DefaultGossipPort = "14000"

	// StreamTimeout is the timeout for establishing a stream connection with
	// a remote node for a full state sync, and for stream read and write
	// operations. Maps to memberlist TCPTimeout.
	DefaultGossipStreamTimeout = 10 * time.Second

	// SuspicionMult is the multiplier for determining the time an
	// inaccessible node is considered suspect before declaring it dead.
	// The actual timeout is calculated using the formula:
	//
	//   SuspicionTimeout = SuspicionMult * log(N+1) * ProbeInterval
	//
	// This allows the timeout to scale properly with expected propagation
	// delay with a larger cluster size. The higher the multiplier, the longer
	// an inaccessible node is considered part of the cluster before declaring
	// it dead, giving that suspect node more time to refute if it is indeed
	// still alive.
	DefaultGossipSuspicionMult = 4

	// PushPullInterval is the interval between complete state syncs.
	// Complete state syncs are done with a single node over TCP and are
	// quite expensive relative to standard gossiped messages. Setting this
	// to zero will disable state push/pull syncs completely.
	//
	// Setting this interval lower (more frequent) will increase convergence
	// speeds across larger clusters at the expense of increased bandwidth
	// usage.
	DefaultGossipPushPullInterval = 30 * time.Second

	// ProbeInterval and ProbeTimeout are used to configure probing behavior
	// for memberlist.
	//
	// ProbeInterval is the interval between random node probes. Setting
	// this lower (more frequent) will cause the memberlist cluster to detect
	// failed nodes more quickly at the expense of increased bandwidth usage.
	//
	// ProbeTimeout is the timeout to wait for an ack from a probed node
	// before assuming it is unhealthy. This should be set to 99-percentile
	// of RTT (round-trip time) on your network.
	DefaultGossipProbeInterval = 1 * time.Second
	DefaultGossipProbeTimeout  = 500 * time.Millisecond

	// GossipInterval and GossipNodes are used to configure the gossip
	// behavior of memberlist.
	//
	// GossipInterval is the interval between sending messages that need
	// to be gossiped that haven't been able to piggyback on probing messages.
	// If this is set to zero, non-piggyback gossip is disabled. By lowering
	// this value (more frequent) gossip messages are propagated across
	// the cluster more quickly at the expense of increased bandwidth.
	//
	// GossipNodes is the number of random nodes to send gossip messages to
	// per GossipInterval. Increasing this number causes the gossip messages
	// to propagate across the cluster more quickly at the expense of
	// increased bandwidth.
	//
	// GossipToTheDeadTime is the interval after which a node has died that
	// we will still try to gossip to it. This gives it a chance to refute.
	DefaultGossipGossipInterval      = 200 * time.Millisecond
	DefaultGossipGossipNodes         = 3
	DefaultGossipGossipToTheDeadTime = 30 * time.Second

	DefaultMetricPollInterval = 0 * time.Minute
)
View Source 
const (
	DefaultFrame = "general"

	// MinThreshold is the lowest count to use in a Top-N operation when
	// looking for additional id/count pairs.
	MinThreshold = 1
)

DefaultFrame is the frame used if one is not specified.

View Source 
const (
	// SliceWidth is the number of column IDs in a slice.
	SliceWidth = 1048576

	// SnapshotExt is the file extension used for an in-process snapshot.
	SnapshotExt = ".snapshotting"

	// CopyExt is the file extension used for the temp file used while copying.
	CopyExt = ".copying"

	// CacheExt is the file extension for persisted cache ids.
	CacheExt = ".cache"

	// HashBlockSize is the number of rows in a merkle hash block.
	HashBlockSize = 100
)
View Source 
const (
	DefaultRowLabel       = "rowID"
	DefaultCacheType      = CacheTypeRanked
	DefaultInverseEnabled = false
	DefaultRangeEnabled   = false

	// Default ranked frame cache
	DefaultCacheSize = 50000
)

Default frame settings.

View Source 
const (
	CacheTypeLRU    = "lru"
	CacheTypeRanked = "ranked"
	CacheTypeNone   = "none"
)

Cache types.

View Source 
const (
	QueryResultTypeNil uint32 = iota
	QueryResultTypeBitmap
	QueryResultTypePairs
	QueryResultTypeSumCount
	QueryResultTypeUint64
	QueryResultTypeBool
)

QueryResult types.

View Source 
const (
	// DefaultCacheFlushInterval is the default value for Fragment.CacheFlushInterval.
	DefaultCacheFlushInterval = 1 * time.Minute

	// FileLimit is the maximum open file limit (ulimit -n) to automatically set.
	FileLimit = 262144 // (512^2)
)
View Source 
const (
	DefaultColumnLabel = "columnID"
	InputDefinitionDir = ".input-definitions"
)

Default index settings.

View Source 
const (
	InputMapping       = "mapping"
	InputValueToRow    = "value-to-row"
	InputSingleRowBool = "single-row-boolean"
	InputSetTimestamp  = "set-timestamp"
)

Action types.

View Source 
const (
	DefaultAntiEntropyInterval = 10 * time.Minute
	DefaultDiagnosticServer    = "https://diagnostics.pilosa.com/v0/diagnostics"
)

Default server settings.

View Source 
const (
	ViewStandard = "standard"
	ViewInverse  = "inverse"

	ViewFieldPrefix = "field_"
)

View layout modes.

View Source 
const AttrBlockSize = 100

AttrBlockSize is the size of attribute blocks for anti-entropy.

View Source 
const (
	// DefaultFragmentMaxOpN is the default value for Fragment.MaxOpN.
	DefaultFragmentMaxOpN = 2000
)
View Source 
const (
	FieldTypeInt = "int"
)

List of field data types.

View Source 
const (
	// ThresholdFactor is used to calculate the threshold for new items entering the cache
	ThresholdFactor = 1.1
)
View Source 
const TimeFormat = "2006-01-02T15:04"

TimeFormat is the go-style time format used to parse string dates.

Variables

View Source 
var (
	ErrHostRequired = errors.New("host required")

	ErrIndexRequired = errors.New("index required")
	ErrIndexExists   = errors.New("index already exists")
	ErrIndexNotFound = errors.New("index not found")

	// ErrFrameRequired is returned when no frame is specified.
	ErrFrameRequired        = errors.New("frame required")
	ErrFrameExists          = errors.New("frame already exists")
	ErrFrameNotFound        = errors.New("frame not found")
	ErrFrameInverseDisabled = errors.New("frame inverse disabled")
	ErrColumnRowLabelEqual  = errors.New("column and row labels cannot be equal")

	ErrInputDefinitionExists         = errors.New("input-definition already exists")
	ErrInputDefinitionHasPrimaryKey  = errors.New("input-definition must contain one PrimaryKey")
	ErrInputDefinitionDupePrimaryKey = errors.New("input-definition can only contain one PrimaryKey")
	ErrInputDefinitionColumnLabel    = errors.New("PrimaryKey field name does not match columnLabel")
	ErrInputDefinitionNameRequired   = errors.New("input-definition name required")
	ErrInputDefinitionAttrsRequired  = errors.New("frames and fields are required")
	ErrInputDefinitionValueMap       = errors.New("valueMap required for map")
	ErrInputDefinitionActionRequired = errors.New("field definitions require an action")
	ErrInputDefinitionNotFound       = errors.New("input-definition not found")

	ErrFieldNotFound          = errors.New("field not found")
	ErrFieldExists            = errors.New("field already exists")
	ErrFieldNameRequired      = errors.New("field name required")
	ErrInvalidFieldType       = errors.New("invalid field type")
	ErrInvalidFieldRange      = errors.New("invalid field range")
	ErrInverseRangeNotAllowed = errors.New("inverse range not allowed")
	ErrRangeCacheNotAllowed   = errors.New("range cache not allowed")
	ErrFrameFieldsNotAllowed  = errors.New("frame fields not allowed")
	ErrInvalidFieldValueType  = errors.New("invalid field value type")
	ErrFieldValueTooLow       = errors.New("field value too low")
	ErrFieldValueTooHigh      = errors.New("field value too high")
	ErrInvalidRangeOperation  = errors.New("invalid range operation")
	ErrInvalidBetweenValue    = errors.New("invalid value for between operation")

	ErrInvalidView      = errors.New("invalid view")
	ErrInvalidCacheType = errors.New("invalid cache type")

	ErrName  = errors.New("invalid index or frame's name, must match [a-z0-9_-]")
	ErrLabel = errors.New("invalid row or column label, must match [A-Za-z0-9_-]")

	// ErrFragmentNotFound is returned when a fragment does not exist.
	ErrFragmentNotFound = errors.New("fragment not found")
	ErrQueryRequired    = errors.New("query required")
	ErrTooManyWrites    = errors.New("too many write commands")

	ErrConfigClusterEnabledHosts = errors.New("providing hosts to a non-disabled cluster is not allowed")
)

System errors.

View Source 
var BuildTime = "not recorded"
View Source 
var ErrInvalidTimeQuantum = errors.New("invalid time quantum")

ErrInvalidTimeQuantum is returned when parsing a time quantum.

View Source 
var Expvar = expvar.NewMap("index")

Expvar global expvar map.

View Source 
var NopBroadcastReceiver = &nopBroadcastReceiver{}

NopBroadcastReceiver is a no-op implementation of the BroadcastReceiver.

View Source 
var NopEventReceiver = &nopEventReceiver{}

NopEventReceiver is a no-op implementation of the EventReceiver.

View Source 
var Version = "v0.0.0"

Functions

func BuildRouters 

func BuildRouters(handler *Handler)

BuildRouters creates Gorilla Mux http routers for both normal and restricted endpoints.

func ContainsSubstring added in v0.5.0 

func ContainsSubstring(a string, list []string) bool

ContainsSubstring checks to see if substring a is contained in any string in the slice.

func CountOpenFiles added in v0.5.0 

func CountOpenFiles() (int, error)

CountOpenFiles on operating systems that support lsof.

func EncodeIndexes added in v0.4.0 

func EncodeIndexes(a []*Index) []*internal.Index

EncodeIndexes converts a into its internal representation.

func EncodeNode added in v0.9.0 

func EncodeNode(n *Node) *internal.Node

EncodeNode converts a Node into its internal representation.

func EncodeNodes added in v0.9.0 

func EncodeNodes(a []*Node) []*internal.Node

EncodeNodes converts a slice of Nodes into its internal representation.

func GetHTTPClient added in v0.8.2 

func GetHTTPClient(t *tls.Config) *http.Client

func GetTimeStamp added in v0.5.0 

func GetTimeStamp(data map[string]interface{}, timeField string) (int64, error)

GetTimeStamp retrieves unix timestamp from Input data.

func HostToIP added in v0.6.0 

func HostToIP(host string) string

HostToIP converts host to an IP4 address based on net.LookupIP().

func IsInverseView 

func IsInverseView(name string) bool

IsInverseView returns true if the view is used for storing an inverted representation.

func IsValidCacheType 

func IsValidCacheType(v string) bool

IsValidCacheType returns true if v is a valid cache type.

func IsValidFieldType added in v0.5.0 

func IsValidFieldType(v string) bool

func IsValidView 

func IsValidView(name string) bool

IsValidView returns true if name is valid.

func MarshalMessage 

func MarshalMessage(m proto.Message) ([]byte, error)

MarshalMessage encodes the protobuf message into a byte slice.

func Pos 

func Pos(rowID, columnID uint64) uint64

Pos returns the row position of a row/column pair.

func QueryValidationSpecRequired 

func QueryValidationSpecRequired(requiredArgs ...string) *queryValidationSpec

func SliceDiff added in v0.9.0 

func SliceDiff(a, b []uint64) []uint64

SliceDiff returns the difference between two uint64 slices.

func StringInSlice added in v0.5.0 

func StringInSlice(a string, list []string) bool

StringInSlice checks for substring a in the slice.

func StringSlicesAreEqual added in v0.9.0 

func StringSlicesAreEqual(a, b []string) bool

StringSlicesAreEqual determines if two string slices are equal.

func UnionStringSlice 

func UnionStringSlice(a, b []string) []string

UnionStringSlice returns a sorted set of tags which combine a & b.

func UnmarshalMessage 

func UnmarshalMessage(buf []byte) (proto.Message, error)

UnmarshalMessage decodes the byte slice into a protobuf message.

func ValidateField added in v0.5.0 

func ValidateField(f *Field) error

func ValidateLabel 

func ValidateLabel(label string) error

ValidateLabel ensures that the label is a valid format.

func ValidateName 

func ValidateName(name string) error

ValidateName ensures that the name is a valid format.

func ViewByTimeUnit 

func ViewByTimeUnit(name string, t time.Time, unit rune) string

ViewByTimeUnit returns the view name for time with a given quantum unit.

func ViewsByTime 

func ViewsByTime(name string, t time.Time, q TimeQuantum) []string

ViewsByTime returns a list of views for a given timestamp.

func ViewsByTimeRange 

func ViewsByTimeRange(name string, start, end time.Time, q TimeQuantum) []string

ViewsByTimeRange returns a list of views to traverse to query a time range.

Types

type Action added in v0.5.0 

type Action struct {
	Frame            string            `json:"frame,omitempty"`
	ValueDestination string            `json:"valueDestination,omitempty"`
	ValueMap         map[string]uint64 `json:"valueMap,omitempty"`
	RowID            *uint64           `json:"rowID,omitempty"`
}

Action describes the mapping method for the field in the InputDefinition.

func (*Action) Encode added in v0.5.0 

func (a *Action) Encode() *internal.InputDefinitionAction

Encode converts Action into its internal representation.

func (*Action) Validate added in v0.5.0 

func (a *Action) Validate() error

Validate ensures the input definition action conforms to our specification.

type AttrBlock 

type AttrBlock struct {
	ID       uint64 `json:"id"`
	Checksum []byte `json:"checksum"`
}

AttrBlock represents a checksummed block of the attribute store.

type AttrBlocks 

type AttrBlocks []AttrBlock

AttrBlocks represents a list of blocks.

func (AttrBlocks) Diff 

func (a AttrBlocks) Diff(other []AttrBlock) []uint64

Diff returns a list of block ids that are different or are new in other. Block lists must be in sorted order.

type AttrCache added in v0.5.0 

type AttrCache struct {
	// contains filtered or unexported fields
}

AttrCache represents a cache for attributes.

func NewAttrCache added in v0.5.0 

func NewAttrCache() *AttrCache

NewAttrCache returns a new instance of AttrCache.

func (*AttrCache) Get added in v0.5.0 

func (c *AttrCache) Get(id uint64) map[string]interface{}

Get returns the cached attributes for a given id.

func (*AttrCache) Set added in v0.5.0 

func (c *AttrCache) Set(id uint64, attrs map[string]interface{})

Set updates the cached attributes for a given id.

type AttrStore 

type AttrStore struct {
	// contains filtered or unexported fields
}

AttrStore represents a storage layer for attributes.

func NewAttrStore 

func NewAttrStore(path string) *AttrStore

NewAttrStore returns a new instance of AttrStore.

func (*AttrStore) Attrs 

func (s *AttrStore) Attrs(id uint64) (m map[string]interface{}, err error)

Attrs returns a set of attributes by ID.

func (*AttrStore) BlockData 

func (s *AttrStore) BlockData(i uint64) (map[uint64]map[string]interface{}, error)

BlockData returns all data for a single block.

func (*AttrStore) Blocks 

func (s *AttrStore) Blocks() ([]AttrBlock, error)

Blocks returns a list of all blocks in the store.

func (*AttrStore) Close 

func (s *AttrStore) Close() error

Close closes the store.

func (*AttrStore) Open 

func (s *AttrStore) Open() error

Open opens and initializes the store.

func (*AttrStore) Path 

func (s *AttrStore) Path() string

Path returns path to the store's data file.

func (*AttrStore) SetAttrs 

func (s *AttrStore) SetAttrs(id uint64, m map[string]interface{}) error

SetAttrs sets attribute values for a given ID.

func (*AttrStore) SetBulkAttrs 

func (s *AttrStore) SetBulkAttrs(m map[uint64]map[string]interface{}) error

SetBulkAttrs sets attribute values for a set of ids.

type Bit 

type Bit struct {
	RowID     uint64
	ColumnID  uint64
	RowKey    string
	ColumnKey string
	Timestamp int64
}

Bit represents the location of a single bit.

func HandleAction added in v0.5.0 

func HandleAction(a Action, value interface{}, colID uint64, timestamp int64) (*Bit, error)

HandleAction Process the input data with its action and return a bit to be imported later Note: if the Bit should not be set then nil is returned with no error From the JSON marshalling the possible types are: float64, boolean, string

type Bitmap 

type Bitmap struct {

	// Attributes associated with the bitmap.
	Attrs map[string]interface{}
	// contains filtered or unexported fields
}

Bitmap represents a set of bits.

func NewBitmap 

func NewBitmap(bits ...uint64) *Bitmap

NewBitmap returns a new instance of Bitmap.

func Union 

func Union(bitmaps []*Bitmap) *Bitmap

Union performs a union on a slice of bitmaps.

func (*Bitmap) Bits 

func (b *Bitmap) Bits() []uint64

Bits returns the bits in b as a slice of ints.

func (*Bitmap) ClearBit 

func (b *Bitmap) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*Bitmap) Count 

func (b *Bitmap) Count() uint64

Count returns the number of set bits in the bitmap.

func (*Bitmap) DecrementCount 

func (b *Bitmap) DecrementCount(i uint64)

DecrementCount decrements the bitmap cached counter.

func (*Bitmap) Difference 

func (b *Bitmap) Difference(other *Bitmap) *Bitmap

Difference returns the diff of b and other.

func (*Bitmap) IncrementCount 

func (b *Bitmap) IncrementCount(i uint64)

IncrementCount increments the bitmap cached counter, note this is an optimization that assumes that the caller is aware the size increased.

func (*Bitmap) Intersect 

func (b *Bitmap) Intersect(other *Bitmap) *Bitmap

Intersect returns the itersection of b and other.

func (*Bitmap) IntersectionCount 

func (b *Bitmap) IntersectionCount(other *Bitmap) uint64

IntersectionCount returns the number of intersections between b and other.

func (*Bitmap) InvalidateCount 

func (b *Bitmap) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*Bitmap) MarshalJSON 

func (b *Bitmap) MarshalJSON() ([]byte, error)

MarshalJSON returns a JSON-encoded byte slice of b.

func (*Bitmap) Merge 

func (b *Bitmap) Merge(other *Bitmap)

Merge merges data from other into b.

func (*Bitmap) SetBit 

func (b *Bitmap) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*Bitmap) Union 

func (b *Bitmap) Union(other *Bitmap) *Bitmap

Union returns the bitwise union of b and other.

func (*Bitmap) Xor added in v0.7.0 

func (b *Bitmap) Xor(other *Bitmap) *Bitmap

Xor returns the xor of b and other.

type BitmapCache 

type BitmapCache interface {
	Fetch(id uint64) (*Bitmap, bool)
	Add(id uint64, b *Bitmap)
}

BitmapCache provides an interface for caching full bitmaps.

type BitmapPair 

type BitmapPair struct {
	ID    uint64
	Count uint64
}

BitmapPair represents a id/count pair with an associated identifier.

type BitmapPairs 

type BitmapPairs []BitmapPair

BitmapPairs is a sortable list of BitmapPair objects.

func (BitmapPairs) Len 

func (p BitmapPairs) Len() int

func (BitmapPairs) Less 

func (p BitmapPairs) Less(i, j int) bool

func (BitmapPairs) Swap 

func (p BitmapPairs) Swap(i, j int)

type BitmapSegment 

type BitmapSegment struct {
	// contains filtered or unexported fields
}

BitmapSegment holds a subset of a bitmap. This could point to a mmapped roaring bitmap or an in-memory bitmap. The width of the segment will always match the slice width.

func (*BitmapSegment) Bits 

func (s *BitmapSegment) Bits() []uint64

Bits returns a list of all bits set in the segment.

func (*BitmapSegment) ClearBit 

func (s *BitmapSegment) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th bit of the bitmap.

func (*BitmapSegment) Count 

func (s *BitmapSegment) Count() uint64

Count returns the number of set bits in the bitmap.

func (*BitmapSegment) Difference 

func (s *BitmapSegment) Difference(other *BitmapSegment) *BitmapSegment

Difference returns the diff of s and other.

func (*BitmapSegment) Intersect 

func (s *BitmapSegment) Intersect(other *BitmapSegment) *BitmapSegment

Intersect returns the itersection of s and other.

func (*BitmapSegment) IntersectionCount 

func (s *BitmapSegment) IntersectionCount(other *BitmapSegment) uint64

IntersectionCount returns the number of intersections between s and other.

func (*BitmapSegment) InvalidateCount 

func (s *BitmapSegment) InvalidateCount()

InvalidateCount updates the cached count in the bitmap.

func (*BitmapSegment) Merge 

func (s *BitmapSegment) Merge(other *BitmapSegment)

Merge adds chunks from other to s. Chunks in s are overwritten if they exist in other.

func (*BitmapSegment) SetBit 

func (s *BitmapSegment) SetBit(i uint64) (changed bool)

SetBit sets the i-th bit of the bitmap.

func (*BitmapSegment) Union 

func (s *BitmapSegment) Union(other *BitmapSegment) *BitmapSegment

Union returns the bitwise union of s and other.

func (*BitmapSegment) Xor added in v0.7.0 

func (s *BitmapSegment) Xor(other *BitmapSegment) *BitmapSegment

Xor returns the xor of s and other.

type Bits 

type Bits []Bit

Bits represents a slice of bits.

func (Bits) ColumnIDs 

func (p Bits) ColumnIDs() []uint64

ColumnIDs returns a slice of all the column IDs.

func (Bits) ColumnKeys added in v0.9.0 

func (p Bits) ColumnKeys() []string

ColumnKeys returns a slice of all the column keys.

func (Bits) GroupBySlice 

func (p Bits) GroupBySlice() map[uint64][]Bit

GroupBySlice returns a map of bits by slice.

func (Bits) Len 

func (p Bits) Len() int

func (Bits) Less 

func (p Bits) Less(i, j int) bool

func (Bits) RowIDs 

func (p Bits) RowIDs() []uint64

RowIDs returns a slice of all the row IDs.

func (Bits) RowKeys added in v0.9.0 

func (p Bits) RowKeys() []string

RowKeys returns a slice of all the row keys.

func (Bits) Swap 

func (p Bits) Swap(i, j int)

func (Bits) Timestamps 

func (p Bits) Timestamps() []int64

Timestamps returns a slice of all the timestamps.

type BitsByPos 

type BitsByPos []Bit

BitsByPos represents a slice of bits sorted by internal position.

func (BitsByPos) Len 

func (p BitsByPos) Len() int

func (BitsByPos) Less 

func (p BitsByPos) Less(i, j int) bool

func (BitsByPos) Swap 

func (p BitsByPos) Swap(i, j int)

type BroadcastHandler 

type BroadcastHandler interface {
	ReceiveMessage(pb proto.Message) error
}

BroadcastHandler is the interface for the pilosa object which knows how to handle broadcast messages. (Hint: this is implemented by pilosa.Server)

type BroadcastReceiver 

type BroadcastReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(BroadcastHandler) error
}

BroadcastReceiver is the interface for the object which will listen for and decode broadcast messages before passing them to pilosa to handle. The implementation of this could be an http server which listens for messages, gets the protobuf payload, and then passes it to BroadcastHandler.ReceiveMessage.

type Broadcaster 

type Broadcaster interface {
	SendSync(pb proto.Message) error
	SendAsync(pb proto.Message) error
	SendTo(to *Node, pb proto.Message) error
}

Broadcaster is an interface for broadcasting messages. 

var NopBroadcaster Broadcaster

NopBroadcaster represents a Broadcaster that doesn't do anything.

type BufIterator 

type BufIterator struct {
	// contains filtered or unexported fields
}

BufIterator wraps an iterator to provide the ability to unread values.

func NewBufIterator 

func NewBufIterator(itr Iterator) *BufIterator

NewBufIterator returns a buffered iterator that wraps itr.

func (*BufIterator) Next 

func (itr *BufIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next pair in the row. If a value has been buffered then it is returned and the buffer is cleared.

func (*BufIterator) Peek 

func (itr *BufIterator) Peek() (rowID, columnID uint64, eof bool)

Peek reads the next value but leaves it on the buffer.

func (*BufIterator) Seek 

func (itr *BufIterator) Seek(rowID, columnID uint64)

Seek moves to the first pair equal to or greater than pseek/bseek.

func (*BufIterator) Unread 

func (itr *BufIterator) Unread()

Unread pushes previous pair on to the buffer. Panics if the buffer is already full.

type Cache 

type Cache interface {
	Add(id uint64, n uint64)
	BulkAdd(id uint64, n uint64)
	Get(id uint64) uint64
	Len() int

	// Returns a list of all IDs.
	IDs() []uint64

	// Updates the cache, if necessary.
	Invalidate()

	// Rebuilds the cache
	Recalculate()

	// Returns an ordered list of the top ranked bitmaps.
	Top() []BitmapPair

	// SetStats defines the stats client used in the cache.
	SetStats(s StatsClient)
}

Cache represents a cache of counts.

type ClientOptions added in v0.8.0 

type ClientOptions struct {
	TLS *tls.Config
}

ClientOptions represents the configuration for a InternalHTTPClient

type Cluster 

type Cluster struct {
	ID        string
	Node      *Node
	Nodes     []*Node // TODO phase this out?
	MemberSet MemberSet

	// Hashing algorithm used to assign partitions to nodes.
	Hasher Hasher

	// The number of partitions in the cluster.
	PartitionN int

	// The number of replicas a partition has.
	ReplicaN int

	// Threshold for logging long-running queries
	LongQueryTime time.Duration

	// Maximum number of SetBit() or ClearBit() commands per request.
	MaxWritesPerRequest int

	// EventReceiver receives NodeEvents pertaining to node membership.
	EventReceiver EventReceiver

	// Data directory path.
	Path     string
	Topology *Topology

	// Required for cluster Resize.
	Static bool // Static is primarily used for testing in a non-gossip environment.

	Coordinator string
	Holder      *Holder
	Broadcaster Broadcaster

	// The writer for any logging.
	LogOutput io.Writer

	//
	RemoteClient *http.Client
	// contains filtered or unexported fields
}

Cluster represents a collection of nodes.

func NewCluster 

func NewCluster() *Cluster

NewCluster returns a new instance of Cluster with defaults.

func (*Cluster) AddNode added in v0.9.0 

func (c *Cluster) AddNode(node *Node) error

AddNode adds a node to the Cluster and updates and saves the new topology.

func (*Cluster) Close added in v0.9.0 

func (c *Cluster) Close() error

func (*Cluster) CompleteCurrentJob added in v0.9.0 

func (c *Cluster) CompleteCurrentJob(state string) error

CompleteCurrentJob sets the state of the current ResizeJob then removes the pointer to currentJob.

func (*Cluster) ContainsSlices added in v0.9.0 

func (c *Cluster) ContainsSlices(index string, maxSlice uint64, node *Node) []uint64

ContainsSlices is like OwnsSlices, but it includes replicas.

func (*Cluster) CoordinatorNode added in v0.9.0 

func (c *Cluster) CoordinatorNode() *Node

Coordinator returns the coordinator node.

func (*Cluster) FollowResizeInstruction added in v0.9.0 

func (c *Cluster) FollowResizeInstruction(instr *internal.ResizeInstruction) error

FollowResizeInstruction is run by any node that receives a ResizeInstruction.

func (*Cluster) FragmentNodes 

func (c *Cluster) FragmentNodes(index string, slice uint64) []*Node

FragmentNodes returns a list of nodes that own a fragment.

func (*Cluster) IsCoordinator added in v0.9.0 

func (c *Cluster) IsCoordinator() bool

IsCoordinator is true if this node is the coordinator.

func (*Cluster) Job added in v0.9.0 

func (c *Cluster) Job(id int64) *ResizeJob

Job returns a ResizeJob by id.

func (*Cluster) ListenForJoins added in v0.9.0 

func (c *Cluster) ListenForJoins()

ListenForJoins handles cluster-resize events.

func (*Cluster) MarkResizeInstructionComplete added in v0.9.0 

func (c *Cluster) MarkResizeInstructionComplete(complete *internal.ResizeInstructionComplete) error

func (*Cluster) MergeClusterStatus added in v0.9.0 

func (c *Cluster) MergeClusterStatus(cs *internal.ClusterStatus) error

func (*Cluster) NodeIDs added in v0.9.0 

func (c *Cluster) NodeIDs() []string

NodeIDs returns the list of IDs in the cluster.

func (*Cluster) NodeLeave added in v0.9.0 

func (c *Cluster) NodeLeave(node *Node) error

NodeLeave initiates the removal of a node from the cluster.

func (*Cluster) Open added in v0.9.0 

func (c *Cluster) Open() error

func (*Cluster) OwnsFragment 

func (c *Cluster) OwnsFragment(nodeID string, index string, slice uint64) bool

OwnsFragment returns true if a host owns a fragment.

func (*Cluster) OwnsSlices 

func (c *Cluster) OwnsSlices(index string, maxSlice uint64, uri URI) []uint64

OwnsSlices finds the set of slices owned by the node per Index

func (*Cluster) Partition 

func (c *Cluster) Partition(index string, slice uint64) int

Partition returns the partition that a slice belongs to.

func (*Cluster) PartitionNodes 

func (c *Cluster) PartitionNodes(partitionID int) []*Node

PartitionNodes returns a list of nodes that own a partition.

func (*Cluster) ReceiveEvent added in v0.9.0 

func (c *Cluster) ReceiveEvent(e *NodeEvent) error

ReceiveEvent represents an implementation of EventHandler.

func (*Cluster) ReceiveNodeState added in v0.9.0 

func (c *Cluster) ReceiveNodeState(nodeID string, state string) error

ReceiveNodeState sets node state in Topology in order for the Coordinator to keep track of, during startup, which nodes have finished opening their Holder.

func (*Cluster) RemoveNode added in v0.9.0 

func (c *Cluster) RemoveNode(node *Node) error

RemoveNode removes a node from the Cluster and updates and saves the new topology.

func (*Cluster) SetCoordinator added in v0.9.0 

func (c *Cluster) SetCoordinator(n *Node) bool

SetCoordinator updates the Coordinator to n. Returns true if the Coordinator changed.

func (*Cluster) SetNodeState added in v0.9.0 

func (c *Cluster) SetNodeState(state string) error

func (*Cluster) SetState added in v0.9.0 

func (c *Cluster) SetState(state string)

func (*Cluster) State added in v0.9.0 

func (c *Cluster) State() string

func (*Cluster) Status 

func (c *Cluster) Status() *internal.ClusterStatus

Status returns the internal ClusterStatus representation.

type CmdIO 

type CmdIO struct {
	Stdin  io.Reader
	Stdout io.Writer
	Stderr io.Writer
}

CmdIO holds standard unix inputs and outputs.

func NewCmdIO 

func NewCmdIO(stdin io.Reader, stdout, stderr io.Writer) *CmdIO

NewCmdIO returns a new instance of CmdIO with inputs and outputs set to the arguments.

type ColumnAttrSet 

type ColumnAttrSet struct {
	ID    uint64                 `json:"id"`
	Attrs map[string]interface{} `json:"attrs,omitempty"`
}

ColumnAttrSet represents a set of attributes for a vertical column in an index. Can have a set of attributes attached to it.

type Config 

type Config struct {
	DataDir string `toml:"data-dir"`
	Bind    string `toml:"bind"`

	// Limits the number of mutating commands that can be in a single request to
	// the server. This includes SetBit, ClearBit, SetRowAttrs & SetColumnAttrs.
	MaxWritesPerRequest int `toml:"max-writes-per-request"`

	LogPath string `toml:"log-path"`

	// TLS
	TLS TLSConfig

	Cluster struct {
		Disabled      bool     `toml:"disabled"`
		Coordinator   bool     `toml:"coordinator"`
		ReplicaN      int      `toml:"replicas"`
		Hosts         []string `toml:"hosts"`
		LongQueryTime Duration `toml:"long-query-time"`
	} `toml:"cluster"`

	Gossip struct {
		Port                string   `toml:"port"`
		Seeds               []string `toml:"seeds"`
		Key                 string   `toml:"key"`
		StreamTimeout       Duration `toml:"stream-timeout"`
		SuspicionMult       int      `toml:"suspicion-mult"`
		PushPullInterval    Duration `toml:"push-pull-interval"`
		ProbeTimeout        Duration `toml:"probe-timeout"`
		ProbeInterval       Duration `toml:"probe-interval"`
		GossipNodes         int      `toml:"gossip-nodes"`
		GossipInterval      Duration `toml:"gossip-interval"`
		GossipToTheDeadTime Duration `toml:"gossip-to-the-dead-time"`
	} `toml:"gossip"`

	AntiEntropy struct {
		Interval Duration `toml:"interval"`
	} `toml:"anti-entropy"`

	Metric struct {
		Service      string   `toml:"service"`
		Host         string   `toml:"host"`
		PollInterval Duration `toml:"poll-interval"`
		Diagnostics  bool     `toml:"diagnostics"`
	} `toml:"metric"`
}

Config represents the configuration for the command.

func NewConfig 

func NewConfig() *Config

NewConfig returns an instance of Config with default options.

func (*Config) Validate added in v0.5.0 

func (c *Config) Validate() error

Validate that all configuration permutations are compatible with each other.

type Duration 

type Duration time.Duration

Duration is a TOML wrapper type for time.Duration.

func (Duration) MarshalTOML added in v0.4.0 

func (d Duration) MarshalTOML() ([]byte, error)

MarshalTOML write duration into valid TOML.

func (Duration) MarshalText 

func (d Duration) MarshalText() (text []byte, err error)

MarshalText writes duration value in text format.

func (Duration) String 

func (d Duration) String() string

String returns the string representation of the duration.

func (*Duration) UnmarshalText 

func (d *Duration) UnmarshalText(text []byte) error

UnmarshalText parses a TOML value into a duration value.

type EventHandler added in v0.9.0 

type EventHandler interface {
	ReceiveEvent(e *NodeEvent) error
}

EventHandler is the interface for the pilosa object which knows how to handle broadcast messages. (Hint: this is implemented by pilosa.Server)

type EventReceiver added in v0.9.0 

type EventReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(EventHandler) error
}

EventReceiver is the interface for the object which will listen for and decode broadcast messages before passing them to pilosa to handle. The implementation of this could be an http server which listens for messages, gets the protobuf payload, and then passes it to EventHandler.ReceiveMessage.

type ExecOptions 

type ExecOptions struct {
	Remote       bool
	ExcludeAttrs bool
	ExcludeBits  bool
}

ExecOptions represents an execution context for a single Execute() call.

type Executor 

type Executor struct {
	Holder *Holder

	// Local hostname & cluster configuration.
	Node    *Node
	Cluster *Cluster

	// Maximum number of SetBit() or ClearBit() commands per request.
	MaxWritesPerRequest int
	// contains filtered or unexported fields
}

Executor recursively executes calls in a PQL query across all slices.

func NewExecutor 

func NewExecutor(remoteClient *http.Client) *Executor

NewExecutor returns a new instance of Executor.

func (*Executor) Execute 

func (e *Executor) Execute(ctx context.Context, index string, q *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)

Execute executes a PQL query.

type ExpvarStatsClient 

type ExpvarStatsClient struct {
	// contains filtered or unexported fields
}

ExpvarStatsClient writes stats out to expvars.

func NewExpvarStatsClient 

func NewExpvarStatsClient() *ExpvarStatsClient

NewExpvarStatsClient returns a new instance of ExpvarStatsClient. This client points at the root of the expvar index map.

func (*ExpvarStatsClient) Close added in v0.8.0 

func (c *ExpvarStatsClient) Close() error

Close no-op.

func (*ExpvarStatsClient) Count 

func (c *ExpvarStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs.

func (*ExpvarStatsClient) CountWithCustomTags added in v0.4.0 

func (c *ExpvarStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (*ExpvarStatsClient) Gauge 

func (c *ExpvarStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric.

func (*ExpvarStatsClient) Histogram 

func (c *ExpvarStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric. This works the same as gauge for this client.

func (*ExpvarStatsClient) Open added in v0.8.0 

func (c *ExpvarStatsClient) Open()

Open no-op.

func (*ExpvarStatsClient) Set 

func (c *ExpvarStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements.

func (*ExpvarStatsClient) SetLogger added in v0.4.0 

func (c *ExpvarStatsClient) SetLogger(logger io.Writer)

SetLogger has no logger.

func (*ExpvarStatsClient) Tags 

func (c *ExpvarStatsClient) Tags() []string

Tags returns a sorted list of tags on the client.

func (*ExpvarStatsClient) Timing 

func (c *ExpvarStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric.

func (*ExpvarStatsClient) WithTags 

func (c *ExpvarStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new client with additional tags appended.

type Field added in v0.5.0 

type Field struct {
	Name string `json:"name,omitempty"`
	Type string `json:"type,omitempty"`
	Min  int64  `json:"min,omitempty"`
	Max  int64  `json:"max,omitempty"`
}

Field represents a range field on a frame.

func (*Field) BaseValue added in v0.7.0 

func (f *Field) BaseValue(op pql.Token, value int64) (baseValue uint64, outOfRange bool)

BaseValue adjusts the value to align with the range for Field for a certain operation type. Note: There is an edge case for GT and LT where this returns a baseValue that does not fully encompass the range. ex: Field.Min = 0, Field.Max = 1023 BaseValue(LT, 2000) returns 1023, which will perform "LT 1023" and effectively exclude any columns with value = 1023. Note that in this case (because the range uses the full BitDepth 0 to 1023), we can't simply return 1024. In order to make this work, we effectively need to change the operator to LTE. Executor.executeFieldRangeSlice() takes this into account and returns `frag.FieldNotNull(field.BitDepth())` in such instances.

func (*Field) BaseValueBetween added in v0.7.0 

func (f *Field) BaseValueBetween(min, max int64) (baseValueMin, baseValueMax uint64, outOfRange bool)

BaseValueBetween adjusts the min/max value to align with the range for Field.

func (*Field) BitDepth added in v0.5.0 

func (f *Field) BitDepth() uint

BitDepth returns the number of bits required to store a value between min & max.

type FieldValue added in v0.7.0 

type FieldValue struct {
	ColumnID uint64
	Value    int64
}

FieldValues represents the value for a column within a range-encoded frame.

type FieldValues added in v0.7.0 

type FieldValues []FieldValue

FieldValues represents a slice of field values.

func (FieldValues) ColumnIDs added in v0.7.0 

func (p FieldValues) ColumnIDs() []uint64

ColumnIDs returns a slice of all the column IDs.

func (FieldValues) GroupBySlice added in v0.7.0 

func (p FieldValues) GroupBySlice() map[uint64][]FieldValue

GroupBySlice returns a map of field values by slice.

func (FieldValues) Len added in v0.7.0 

func (p FieldValues) Len() int

func (FieldValues) Less added in v0.7.0 

func (p FieldValues) Less(i, j int) bool

func (FieldValues) Swap added in v0.7.0 

func (p FieldValues) Swap(i, j int)

func (FieldValues) Values added in v0.7.0 

func (p FieldValues) Values() []int64

Values returns a slice of all the values.

type Fragment 

type Fragment struct {

	// Cache for row counts.
	CacheType string // passed in by frame

	CacheSize uint32

	// Number of operations performed before performing a snapshot.
	// This limits the size of fragments on the heap and flushes them to disk
	// so that they can be mmapped and heap utilization can be kept low.
	MaxOpN int

	// Writer used for out-of-band log entries.
	LogOutput io.Writer

	// Row attribute storage.
	// This is set by the parent frame unless overridden for testing.
	RowAttrStore *AttrStore
	// contains filtered or unexported fields
}

Fragment represents the intersection of a frame and slice in an index.

func NewFragment 

func NewFragment(path, index, frame, view string, slice uint64) *Fragment

NewFragment returns a new instance of Fragment.

func (*Fragment) BlockData 

func (f *Fragment) BlockData(id int) (rowIDs, columnIDs []uint64)

BlockData returns bits in a block as row & column ID pairs.

func (*Fragment) BlockN 

func (f *Fragment) BlockN() int

BlockN returns the number of blocks in the fragment.

func (*Fragment) Blocks 

func (f *Fragment) Blocks() []FragmentBlock

Blocks returns info for all blocks containing data.

func (*Fragment) Cache 

func (f *Fragment) Cache() Cache

Cache returns the fragment's cache. This is not safe for concurrent use.

func (*Fragment) CachePath 

func (f *Fragment) CachePath() string

CachePath returns the path to the fragment's cache data.

func (*Fragment) Checksum 

func (f *Fragment) Checksum() []byte

Checksum returns a checksum for the entire fragment. If two fragments have the same checksum then they have the same data.

func (*Fragment) ClearBit 

func (f *Fragment) ClearBit(rowID, columnID uint64) (bool, error)

ClearBit clears a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) Close 

func (f *Fragment) Close() error

Close flushes the underlying storage, closes the file and unlocks it.

func (*Fragment) FieldNotNull added in v0.7.1 

func (f *Fragment) FieldNotNull(bitDepth uint) (*Bitmap, error)

FieldNotNull returns the not-null row (stored at bitDepth).

func (*Fragment) FieldRange added in v0.6.0 

func (f *Fragment) FieldRange(op pql.Token, bitDepth uint, predicate uint64) (*Bitmap, error)

FieldRange returns bitmaps with a field value encoding matching the predicate.

func (*Fragment) FieldRangeBetween added in v0.7.0 

func (f *Fragment) FieldRangeBetween(bitDepth uint, predicateMin, predicateMax uint64) (*Bitmap, error)

FieldRangeBetween returns bitmaps with a field value encoding matching any value between predicateMin and predicateMax.

func (*Fragment) FieldSum added in v0.7.0 

func (f *Fragment) FieldSum(filter *Bitmap, bitDepth uint) (sum, count uint64, err error)

FieldSum returns the sum of a given field as well as the number of columns involved. A bitmap can be passed in to optionally filter the computed columns.

func (*Fragment) FieldValue added in v0.5.0 

func (f *Fragment) FieldValue(columnID uint64, bitDepth uint) (value uint64, exists bool, err error)

FieldValue uses a column of bits to read a multi-bit value.

func (*Fragment) FlushCache 

func (f *Fragment) FlushCache() error

FlushCache writes the cache data to disk.

func (*Fragment) ForEachBit 

func (f *Fragment) ForEachBit(fn func(rowID, columnID uint64) error) error

ForEachBit executes fn for every bit set in the fragment. Errors returned from fn are passed through.

func (*Fragment) Frame 

func (f *Fragment) Frame() string

Frame returns the frame the fragment was initialized with.

func (*Fragment) Import 

func (f *Fragment) Import(rowIDs, columnIDs []uint64) error

Import bulk imports a set of bits and then snapshots the storage. This does not affect the fragment's cache.

func (*Fragment) ImportValue added in v0.7.0 

func (f *Fragment) ImportValue(columnIDs, values []uint64, bitDepth uint) error

ImportValue bulk imports a set of range-encoded values.

func (*Fragment) Index 

func (f *Fragment) Index() string

Index returns the index that the fragment was initialized with.

func (*Fragment) InvalidateChecksums 

func (f *Fragment) InvalidateChecksums()

InvalidateChecksums clears all cached block checksums.

func (*Fragment) MergeBlock 

func (f *Fragment) MergeBlock(id int, data []PairSet) (sets, clears []PairSet, err error)

MergeBlock compares the block's bits and computes a diff with another set of block bits. The state of a bit is determined by consensus from all blocks being considered.

For example, if 3 blocks are compared and two have a set bit and one has a cleared bit then the bit is considered cleared. The function returns the diff per incoming block so that all can be in sync.

func (*Fragment) Open 

func (f *Fragment) Open() error

Open opens the underlying storage.

func (*Fragment) Path 

func (f *Fragment) Path() string

Path returns the path the fragment was initialized with.

func (*Fragment) ReadFrom 

func (f *Fragment) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom reads a data file from r and loads it into the fragment.

func (*Fragment) RecalculateCache 

func (f *Fragment) RecalculateCache()

RecalculateCache rebuilds the cache regardless of invalidate time delay.

func (*Fragment) Row 

func (f *Fragment) Row(rowID uint64) *Bitmap

Row returns a row by ID.

func (*Fragment) SetBit 

func (f *Fragment) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit for a given column & row within the fragment. This updates both the on-disk storage and the in-cache bitmap.

func (*Fragment) SetFieldValue added in v0.5.0 

func (f *Fragment) SetFieldValue(columnID uint64, bitDepth uint, value uint64) (changed bool, err error)

SetFieldValue uses a column of bits to set a multi-bit value.

func (*Fragment) Slice 

func (f *Fragment) Slice() uint64

Slice returns the slice the fragment was initialized with.

func (*Fragment) Snapshot 

func (f *Fragment) Snapshot() error

Snapshot writes the storage bitmap to disk and reopens it.

func (*Fragment) Top 

func (f *Fragment) Top(opt TopOptions) ([]Pair, error)

Top returns the top rows from the fragment. If opt.Src is specified then only rows which intersect src are returned. If opt.FilterValues exist then the row attribute specified by field is matched.

func (*Fragment) View 

func (f *Fragment) View() string

View returns the view the fragment was initialized with.

func (*Fragment) WriteTo 

func (f *Fragment) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the fragment's data to w.

type FragmentBlock 

type FragmentBlock struct {
	ID       int    `json:"id"`
	Checksum []byte `json:"checksum"`
}

FragmentBlock represents info about a subsection of the rows in a block. This is used for comparing data in remote blocks for active anti-entropy.

type FragmentSyncer 

type FragmentSyncer struct {
	Fragment *Fragment

	Node         *Node
	Cluster      *Cluster
	RemoteClient *http.Client

	Closing <-chan struct{}
}

FragmentSyncer syncs a local fragment to one on a remote host.

func (*FragmentSyncer) SyncFragment 

func (s *FragmentSyncer) SyncFragment() error

SyncFragment compares checksums for the local and remote fragments and then merges any blocks which have differences.

type Frame 

type Frame struct {
	Stats StatsClient

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Frame represents a container for views.

func NewFrame 

func NewFrame(path, index, name string) (*Frame, error)

NewFrame returns a new instance of frame.

func (*Frame) CacheSize 

func (f *Frame) CacheSize() uint32

CacheSize returns the ranked frame cache size.

func (*Frame) CacheType 

func (f *Frame) CacheType() string

CacheType returns the caching mode for the frame.

func (*Frame) ClearBit 

func (f *Frame) ClearBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

ClearBit clears a bit within the frame.

func (*Frame) Close 

func (f *Frame) Close() error

Close closes the frame and its views.

func (*Frame) CreateField added in v0.7.0 

func (f *Frame) CreateField(field *Field) error

CreateField creates a new field on the frame.

func (*Frame) CreateViewIfNotExists 

func (f *Frame) CreateViewIfNotExists(name string) (*View, error)

CreateViewIfNotExists returns the named view, creating it if necessary. Additionally, a CreateViewMessage is sent to the cluster.

func (*Frame) DeleteField added in v0.7.0 

func (f *Frame) DeleteField(name string) error

DeleteField deletes an existing field on the schema.

func (*Frame) DeleteView added in v0.7.0 

func (f *Frame) DeleteView(name string) error

DeleteView removes the view from the frame.

func (*Frame) Field added in v0.5.0 

func (f *Frame) Field(name string) *Field

Field returns a field by name.

func (*Frame) FieldRange added in v0.6.0 

func (f *Frame) FieldRange(name string, op pql.Token, predicate int64) (*Bitmap, error)

func (*Frame) FieldRangeBetween added in v0.7.0 

func (f *Frame) FieldRangeBetween(name string, predicateMin, predicateMax int64) (*Bitmap, error)

func (*Frame) FieldSum added in v0.7.0 

func (f *Frame) FieldSum(filter *Bitmap, name string) (sum, count int64, err error)

FieldSum returns the sum and count for a field. An optional filtering bitmap can be provided.

func (*Frame) FieldValue added in v0.5.0 

func (f *Frame) FieldValue(columnID uint64, name string) (value int64, exists bool, err error)

FieldValue reads a field value for a column.

func (*Frame) Fields added in v0.9.0 

func (f *Frame) Fields() []*Field

Fields returns the fields on the frame.

func (*Frame) GetFields added in v0.7.0 

func (f *Frame) GetFields() ([]*Field, error)

GetFields returns a list of all the fields in the frame.

func (*Frame) HasField added in v0.9.0 

func (f *Frame) HasField(name string) bool

HasField returns true if a field exists on the frame.

func (*Frame) Import 

func (f *Frame) Import(rowIDs, columnIDs []uint64, timestamps []*time.Time) error

Import bulk imports data.

func (*Frame) ImportValue added in v0.7.0 

func (f *Frame) ImportValue(fieldName string, columnIDs []uint64, values []int64) error

ImportValue bulk imports range-encoded value data.

func (*Frame) Index 

func (f *Frame) Index() string

Index returns the index name the frame was initialized with.

func (*Frame) InverseEnabled 

func (f *Frame) InverseEnabled() bool

InverseEnabled returns true if an inverse view is available.

func (*Frame) MaxInverseSlice 

func (f *Frame) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the frame.

func (*Frame) MaxSlice 

func (f *Frame) MaxSlice() uint64

MaxSlice returns the max slice in the frame.

func (*Frame) Name 

func (f *Frame) Name() string

Name returns the name the frame was initialized with.

func (*Frame) Open 

func (f *Frame) Open() error

Open opens and initializes the frame.

func (*Frame) Options 

func (f *Frame) Options() FrameOptions

Options returns all options for this frame.

func (*Frame) Path 

func (f *Frame) Path() string

Path returns the path the frame was initialized with.

func (*Frame) RangeEnabled added in v0.5.0 

func (f *Frame) RangeEnabled() bool

RangeEnabled returns true if range fields can be stored on this frame.

func (*Frame) RecalculateCaches added in v0.8.0 

func (f *Frame) RecalculateCaches()

RecalculateCaches recalculates caches on every view in the frame.

func (*Frame) RowAttrStore 

func (f *Frame) RowAttrStore() *AttrStore

RowAttrStore returns the attribute storage.

func (*Frame) RowLabel 

func (f *Frame) RowLabel() string

RowLabel returns the row label.

func (*Frame) SetBit 

func (f *Frame) SetBit(name string, rowID, colID uint64, t *time.Time) (changed bool, err error)

SetBit sets a bit on a view within the frame.

func (*Frame) SetCacheSize 

func (f *Frame) SetCacheSize(v uint32) error

SetCacheSize sets the cache size for ranked fames. Persists to meta file on update. defaults to DefaultCacheSize 50000

func (*Frame) SetFieldValue added in v0.5.0 

func (f *Frame) SetFieldValue(columnID uint64, name string, value int64) (changed bool, err error)

SetFieldValue sets a field value for a column.

func (*Frame) SetRowLabel 

func (f *Frame) SetRowLabel(v string) error

SetRowLabel sets the row labels. Persists to meta file on update.

func (*Frame) SetTimeQuantum 

func (f *Frame) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the time quantum for the frame.

func (*Frame) TimeQuantum 

func (f *Frame) TimeQuantum() TimeQuantum

TimeQuantum returns the time quantum for the frame.

func (*Frame) View 

func (f *Frame) View(name string) *View

View returns a view in the frame by name.

func (*Frame) ViewPath 

func (f *Frame) ViewPath(name string) string

ViewPath returns the path to a view in the frame.

func (*Frame) Views 

func (f *Frame) Views() []*View

Views returns a list of all views in the frame.

type FrameInfo 

type FrameInfo struct {
	Name    string       `json:"name"`
	Options FrameOptions `json:"options"`
	Views   []*ViewInfo  `json:"views,omitempty"`
}

FrameInfo represents schema information for a frame.

type FrameOptions 

type FrameOptions struct {
	RowLabel       string      `json:"rowLabel,omitempty"`
	InverseEnabled bool        `json:"inverseEnabled,omitempty"`
	RangeEnabled   bool        `json:"rangeEnabled,omitempty"`
	CacheType      string      `json:"cacheType,omitempty"`
	CacheSize      uint32      `json:"cacheSize,omitempty"`
	TimeQuantum    TimeQuantum `json:"timeQuantum,omitempty"`
	Fields         []*Field    `json:"fields,omitempty"`
}

FrameOptions represents options to set when initializing a frame.

func (*FrameOptions) Encode 

func (o *FrameOptions) Encode() *internal.FrameMeta

Encode converts o into its internal representation.

type GCNotifier added in v0.9.0 

type GCNotifier interface {
	Close()
	AfterGC() <-chan struct{}
}

GCNotifier represents an interface for garbage collection notificationss. 

var NopGCNotifier GCNotifier

NopGCNotifier represents a GCNotifier that doesn't do anything.

type Gossiper added in v0.9.0 

type Gossiper interface {
	SendAsync(pb proto.Message) error
}

Gossiper is an interface for sharing messages via gossip. 

var NopGossiper Gossiper

NopBroadcaster represents a Broadcaster that doesn't do anything.

type Handler 

type Handler struct {
	Holder           *Holder
	Broadcaster      Broadcaster
	BroadcastHandler BroadcastHandler
	StatusHandler    StatusHandler

	// Local hostname & cluster configuration.
	Node         *Node
	Cluster      *Cluster
	RemoteClient *http.Client

	Router           *mux.Router
	NormalRouter     *mux.Router
	RestrictedRouter *mux.Router

	// The execution engine for running queries.
	Executor interface {
		Execute(context context.Context, index string, query *pql.Query, slices []uint64, opt *ExecOptions) ([]interface{}, error)
	}

	// The writer for any logging.
	LogOutput io.Writer
	// contains filtered or unexported fields
}

Handler represents an HTTP handler.

func NewHandler 

func NewHandler() *Handler

NewHandler returns a new instance of Handler with a default logger.

func (*Handler) InputJSONDataParser added in v0.5.0 

func (h *Handler) InputJSONDataParser(req map[string]interface{}, index *Index, name string) (map[string][]*Bit, error)

InputJSONDataParser validates input json file and executes SetBit.

func (*Handler) ServeHTTP 

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP handles an HTTP request.

func (*Handler) SetNormal 

func (h *Handler) SetNormal()

SetNormal is a method of the SecurityManager interface which provides normal URI routing.

func (*Handler) SetRestricted 

func (h *Handler) SetRestricted()

SetRestricted is a method of the SecurityManager interface which provides restricted URI routing.

type Hasher 

type Hasher interface {
	// Hashes the key into a number between [0,N).
	Hash(key uint64, n int) int
}

Hasher represents an interface to hash integers into buckets.

func NewHasher 

func NewHasher() Hasher

NewHasher returns a new instance of the default hasher.

type Holder 

type Holder struct {
	Broadcaster Broadcaster

	// Stats
	Stats StatsClient

	// Data directory path.
	Path string

	// The interval at which the cached row ids are persisted to disk.
	CacheFlushInterval time.Duration

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Holder represents a container for indexes.

func NewHolder 

func NewHolder() *Holder

NewHolder returns a new instance of Holder.

func (*Holder) ApplySchema added in v0.9.0 

func (h *Holder) ApplySchema(schema *internal.Schema) error

ApplySchema applies an internal Schema to Holder.

func (*Holder) Close 

func (h *Holder) Close() error

Close closes all open fragments.

func (*Holder) CreateIndex 

func (h *Holder) CreateIndex(name string, opt IndexOptions) (*Index, error)

CreateIndex creates an index. An error is returned if the index already exists.

func (*Holder) CreateIndexIfNotExists 

func (h *Holder) CreateIndexIfNotExists(name string, opt IndexOptions) (*Index, error)

CreateIndexIfNotExists returns an index by name. The index is created if it does not already exist.

func (*Holder) DeleteIndex 

func (h *Holder) DeleteIndex(name string) error

DeleteIndex removes an index from the holder.

func (*Holder) EncodeMaxSlices added in v0.9.0 

func (h *Holder) EncodeMaxSlices() *internal.MaxSlices

EncodeMaxSlices creates and internal representation of max slices.

func (*Holder) EncodeSchema added in v0.9.0 

func (h *Holder) EncodeSchema() *internal.Schema

EncodeSchema creates an internal representation of schema.

func (*Holder) Fragment 

func (h *Holder) Fragment(index, frame, view string, slice uint64) *Fragment

Fragment returns the fragment for an index, frame & slice.

func (*Holder) Frame 

func (h *Holder) Frame(index, name string) *Frame

Frame returns the frame for an index and name.

func (*Holder) HasData added in v0.9.0 

func (h *Holder) HasData() bool

HasData returns true if Holder contains at least one index. This is used to determine if the rebalancing of data is necessary when a node joins the cluster.

func (*Holder) Index 

func (h *Holder) Index(name string) *Index

Index returns the index by name.

func (*Holder) IndexPath 

func (h *Holder) IndexPath(name string) string

IndexPath returns the path where a given index is stored.

func (*Holder) Indexes 

func (h *Holder) Indexes() []*Index

Indexes returns a list of all indexes in the holder.

func (*Holder) MaxInverseSlices 

func (h *Holder) MaxInverseSlices() map[string]uint64

MaxInverseSlices returns MaxInverseSlice map for all indexes.

func (*Holder) MaxSlices 

func (h *Holder) MaxSlices() map[string]uint64

MaxSlices returns MaxSlice map for all indexes.

func (*Holder) Open 

func (h *Holder) Open() error

Open initializes the root data directory for the holder.

func (*Holder) Peek 

func (h *Holder) Peek() bool

Peek reads the root data directory for the holder without actually loading any data into memory. HasData is returned, and h.hasData is set.

func (*Holder) RecalculateCaches added in v0.8.0 

func (h *Holder) RecalculateCaches()

RecalculateCaches recalculates caches on every index in the holder. This is probably not practical to call in real-world workloads, but makes writing integration tests much eaiser, since one doesn't have to wait 10 seconds after setting bits to get expected response.

func (*Holder) Schema 

func (h *Holder) Schema() []*IndexInfo

Schema returns schema information for all indexes, frames, and views.

func (*Holder) View 

func (h *Holder) View(index, frame, name string) *View

View returns the view for an index, frame, and name.

type HolderCleaner added in v0.9.0 

type HolderCleaner struct {
	Node *Node

	Holder  *Holder
	Cluster *Cluster

	// Signals that the sync should stop.
	Closing <-chan struct{}
}

HolderCleaner removes fragments and data files that are no longer used.

func (*HolderCleaner) CleanHolder added in v0.9.0 

func (c *HolderCleaner) CleanHolder() error

CleanHolder compares the holder with the cluster state and removes any unnecessary fragments and files.

func (*HolderCleaner) IsClosing added in v0.9.0 

func (c *HolderCleaner) IsClosing() bool

IsClosing returns true if the cleaner has been marked to close.

type HolderSyncer 

type HolderSyncer struct {
	Holder *Holder

	Node         *Node
	Cluster      *Cluster
	RemoteClient *http.Client

	// Stats
	Stats StatsClient

	// Signals that the sync should stop.
	Closing <-chan struct{}
}

HolderSyncer is an active anti-entropy tool that compares the local holder with a remote holder based on block checksums and resolves differences.

func (*HolderSyncer) IsClosing 

func (s *HolderSyncer) IsClosing() bool

IsClosing returns true if the syncer has been marked to close.

func (*HolderSyncer) SyncHolder 

func (s *HolderSyncer) SyncHolder() error

SyncHolder compares the holder on host with the local holder and resolves differences.

type Index 

type Index struct {
	Stats StatsClient

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Index represents a container for frames.

func NewIndex 

func NewIndex(path, name string) (*Index, error)

NewIndex returns a new instance of Index.

func (*Index) Close 

func (i *Index) Close() error

Close closes the index and its frames.

func (*Index) ColumnAttrStore 

func (i *Index) ColumnAttrStore() *AttrStore

ColumnAttrStore returns the storage for column attributes.

func (*Index) ColumnLabel 

func (i *Index) ColumnLabel() string

ColumnLabel returns the column label.

func (*Index) CreateFrame 

func (i *Index) CreateFrame(name string, opt FrameOptions) (*Frame, error)

CreateFrame creates a frame.

func (*Index) CreateFrameIfNotExists 

func (i *Index) CreateFrameIfNotExists(name string, opt FrameOptions) (*Frame, error)

CreateFrameIfNotExists creates a frame with the given options if it doesn't exist.

func (*Index) CreateInputDefinition added in v0.5.0 

func (i *Index) CreateInputDefinition(pb *internal.InputDefinition) (*InputDefinition, error)

CreateInputDefinition creates a new input definition.

func (*Index) DeleteFrame 

func (i *Index) DeleteFrame(name string) error

DeleteFrame removes a frame from the index.

func (*Index) DeleteInputDefinition added in v0.5.0 

func (i *Index) DeleteInputDefinition(name string) error

DeleteInputDefinition removes an input definition from the index.

func (*Index) Frame 

func (i *Index) Frame(name string) *Frame

Frame returns a frame in the index by name.

func (*Index) FramePath 

func (i *Index) FramePath(name string) string

FramePath returns the path to a frame in the index.

func (*Index) Frames 

func (i *Index) Frames() []*Frame

Frames returns a list of all frames in the index.

func (*Index) InputBits added in v0.5.0 

func (i *Index) InputBits(frame string, bits []*Bit) error

InputBits Process the []Bit though the Frame import process

func (*Index) InputDefinition added in v0.5.0 

func (i *Index) InputDefinition(name string) (*InputDefinition, error)

InputDefinition returns an input definition in the index by name.

func (*Index) InputDefinitionPath added in v0.5.0 

func (i *Index) InputDefinitionPath() string

InputDefinitionPath returns the path to the input definition directory for the index.

func (*Index) InputDefinitions added in v0.9.0 

func (i *Index) InputDefinitions() []*InputDefinition

InputDefinitions returns a list of all inputDefinitions in the index.

func (*Index) MaxInverseSlice 

func (i *Index) MaxInverseSlice() uint64

MaxInverseSlice returns the max inverse slice in the index according to this node.

func (*Index) MaxSlice 

func (i *Index) MaxSlice() uint64

MaxSlice returns the max slice in the index according to this node.

func (*Index) Name 

func (i *Index) Name() string

Name returns name of the index.

func (*Index) Open 

func (i *Index) Open() error

Open opens and initializes the index.

func (*Index) Options added in v0.7.0 

func (i *Index) Options() IndexOptions

Options returns all options for this index.

func (*Index) Path 

func (i *Index) Path() string

Path returns the path the index was initialized with.

func (*Index) RecalculateCaches added in v0.8.0 

func (i *Index) RecalculateCaches()

RecalculateCaches recalculates caches on every frame in the index.

func (*Index) SetColumnLabel 

func (i *Index) SetColumnLabel(v string) error

SetColumnLabel sets the column label. Persists to meta file on update.

func (*Index) SetRemoteMaxInverseSlice 

func (i *Index) SetRemoteMaxInverseSlice(v uint64)

SetRemoteMaxInverseSlice sets the remote max inverse slice value received from another node.

func (*Index) SetRemoteMaxSlice 

func (i *Index) SetRemoteMaxSlice(newmax uint64)

SetRemoteMaxSlice sets the remote max slice value received from another node.

func (*Index) SetTimeQuantum 

func (i *Index) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the default time quantum for the index.

func (*Index) TimeQuantum 

func (i *Index) TimeQuantum() TimeQuantum

TimeQuantum returns the default time quantum for the index.

type IndexInfo 

type IndexInfo struct {
	Name   string       `json:"name"`
	Frames []*FrameInfo `json:"frames"`
}

IndexInfo represents schema information for an index.

func MergeSchemas 

func MergeSchemas(a, b []*IndexInfo) []*IndexInfo

MergeSchemas combines indexes and frames from a and b into one schema.

type IndexOptions 

type IndexOptions struct {
	ColumnLabel string      `json:"columnLabel,omitempty"`
	TimeQuantum TimeQuantum `json:"timeQuantum,omitempty"`
}

IndexOptions represents options to set when initializing an index.

func (*IndexOptions) Encode 

func (i *IndexOptions) Encode() *internal.IndexMeta

Encode converts i into its internal representation.

type InputDefinition added in v0.5.0 

type InputDefinition struct {
	// contains filtered or unexported fields
}

InputDefinition represents a container for the data input definition.

func NewInputDefinition added in v0.5.0 

func NewInputDefinition(path, index, name string) (*InputDefinition, error)

NewInputDefinition returns a new instance of InputDefinition.

func (*InputDefinition) AddFrame added in v0.5.0 

func (i *InputDefinition) AddFrame(frame InputFrame) error

AddFrame manually add frame to input definition.

func (*InputDefinition) Fields added in v0.5.0 

func (i *InputDefinition) Fields() []InputDefinitionField

Fields returns fields of the input definition was initialized with.

func (*InputDefinition) Frames added in v0.5.0 

func (i *InputDefinition) Frames() []InputFrame

Frames returns frames of the input definition was initialized with.

func (*InputDefinition) LoadDefinition added in v0.5.0 

func (i *InputDefinition) LoadDefinition(pb *internal.InputDefinition) error

LoadDefinition loads the protobuf format of a definition.

func (*InputDefinition) Open added in v0.5.0 

func (i *InputDefinition) Open() error

Open opens and initializes the InputDefinition from file.

type InputDefinitionField added in v0.5.0 

type InputDefinitionField struct {
	Name       string   `json:"name,omitempty"`
	PrimaryKey bool     `json:"primaryKey,omitempty"`
	Actions    []Action `json:"actions,omitempty"`
}

InputDefinitionField descripes a single field mapping in the InputDefinition.

func (*InputDefinitionField) Encode added in v0.5.0 

func (o *InputDefinitionField) Encode() *internal.InputDefinitionField

Encode converts InputDefinitionField into its internal representation.

type InputDefinitionInfo added in v0.5.0 

type InputDefinitionInfo struct {
	Frames []InputFrame           `json:"frames"`
	Fields []InputDefinitionField `json:"fields"`
}

InputDefinitionInfo represents the json message format needed to create an InputDefinition.

func (*InputDefinitionInfo) Encode added in v0.5.0 

func (i *InputDefinitionInfo) Encode() *internal.InputDefinition

Encode converts InputDefinitionInfo into its internal representation.

func (*InputDefinitionInfo) Validate added in v0.5.0 

func (i *InputDefinitionInfo) Validate() error

Validate the InputDefinitionInfo data.

type InputFrame added in v0.5.0 

type InputFrame struct {
	Name    string       `json:"name,omitempty"`
	Options FrameOptions `json:"options,omitempty"`
}

InputFrame defines the frame used in the input definition.

func (*InputFrame) Encode added in v0.5.0 

func (i *InputFrame) Encode() *internal.Frame

Encode converts InputFrame into its internal representation.

func (*InputFrame) Validate added in v0.5.0 

func (i *InputFrame) Validate() error

Validate the InputFrame data.

type InternalClient added in v0.8.0 

type InternalClient interface {
	MaxSliceByIndex(ctx context.Context) (map[string]uint64, error)
	MaxInverseSliceByIndex(ctx context.Context) (map[string]uint64, error)
	Schema(ctx context.Context) ([]*IndexInfo, error)
	CreateIndex(ctx context.Context, index string, opt IndexOptions) error
	FragmentNodes(ctx context.Context, index string, slice uint64) ([]*Node, error)
	ExecuteQuery(ctx context.Context, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)
	Import(ctx context.Context, index, frame string, slice uint64, bits []Bit) error
	ImportK(ctx context.Context, index, frame string, bits []Bit) error
	EnsureIndex(ctx context.Context, name string, options IndexOptions) error
	EnsureFrame(ctx context.Context, indexName string, frameName string, options FrameOptions) error
	ImportValue(ctx context.Context, index, frame, field string, slice uint64, vals []FieldValue) error
	ExportCSV(ctx context.Context, index, frame, view string, slice uint64, w io.Writer) error
	BackupTo(ctx context.Context, w io.Writer, index, frame, view string) error
	BackupSlice(ctx context.Context, index, frame, view string, slice uint64) (io.ReadCloser, error)
	RestoreFrom(ctx context.Context, r io.Reader, index, frame, view string) error
	CreateFrame(ctx context.Context, index, frame string, opt FrameOptions) error
	RestoreFrame(ctx context.Context, host, index, frame string) error
	FrameViews(ctx context.Context, index, frame string) ([]string, error)
	FragmentBlocks(ctx context.Context, index, frame, view string, slice uint64) ([]FragmentBlock, error)
	BlockData(ctx context.Context, index, frame, view string, slice uint64, block int) ([]uint64, []uint64, error)
	ColumnAttrDiff(ctx context.Context, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
	RowAttrDiff(ctx context.Context, index, frame string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
	SendMessage(ctx context.Context, pb proto.Message) error
	NodeID(uri *URI) (string, error)
}

InternalClient should be implemented by any struct that enables any transport between nodes TODO: Refactor Note from Travis: Typically an interface containing more than two or three methods is an indication that something hasn't been architected correctly. While I understand that putting the entire Client behind an interface might require this many methods, I don't want to let it go unquestioned.

type InternalHTTPClient added in v0.8.0 

type InternalHTTPClient struct {

	// The client to use for HTTP communication.
	HTTPClient *http.Client
	// contains filtered or unexported fields
}

InternalHTTPClient represents a client to the Pilosa cluster.

func NewInternalHTTPClient added in v0.8.0 

func NewInternalHTTPClient(host string, remoteClient *http.Client) (*InternalHTTPClient, error)

NewInternalHTTPClient returns a new instance of InternalHTTPClient to connect to host.

func NewInternalHTTPClientFromURI added in v0.8.0 

func NewInternalHTTPClientFromURI(defaultURI *URI, remoteClient *http.Client) *InternalHTTPClient

func (*InternalHTTPClient) BackupSlice added in v0.8.0 

func (c *InternalHTTPClient) BackupSlice(ctx context.Context, index, frame, view string, slice uint64) (io.ReadCloser, error)

BackupSlice retrieves a streaming backup from a single slice. This function tries slice owners until one succeeds.

func (*InternalHTTPClient) BackupTo added in v0.8.0 

func (c *InternalHTTPClient) BackupTo(ctx context.Context, w io.Writer, index, frame, view string) error

BackupTo backs up an entire frame from a cluster to w.

func (*InternalHTTPClient) BlockData added in v0.8.0 

func (c *InternalHTTPClient) BlockData(ctx context.Context, index, frame, view string, slice uint64, block int) ([]uint64, []uint64, error)

BlockData returns row/column id pairs for a block.

func (*InternalHTTPClient) ColumnAttrDiff added in v0.8.0 

func (c *InternalHTTPClient) ColumnAttrDiff(ctx context.Context, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

ColumnAttrDiff returns data from differing blocks on a remote host.

func (*InternalHTTPClient) CreateFrame added in v0.8.0 

func (c *InternalHTTPClient) CreateFrame(ctx context.Context, index, frame string, opt FrameOptions) error

CreateFrame creates a new frame on the server.

func (*InternalHTTPClient) CreateIndex added in v0.8.0 

func (c *InternalHTTPClient) CreateIndex(ctx context.Context, index string, opt IndexOptions) error

CreateIndex creates a new index on the server.

func (*InternalHTTPClient) EnsureFrame added in v0.8.0 

func (c *InternalHTTPClient) EnsureFrame(ctx context.Context, indexName string, frameName string, options FrameOptions) error

func (*InternalHTTPClient) EnsureIndex added in v0.8.0 

func (c *InternalHTTPClient) EnsureIndex(ctx context.Context, name string, options IndexOptions) error

func (*InternalHTTPClient) ExecuteQuery added in v0.8.0 

func (c *InternalHTTPClient) ExecuteQuery(ctx context.Context, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)

ExecuteQuery executes query against index on the server.

func (*InternalHTTPClient) ExportCSV added in v0.8.0 

func (c *InternalHTTPClient) ExportCSV(ctx context.Context, index, frame, view string, slice uint64, w io.Writer) error

ExportCSV bulk exports data for a single slice from a host to CSV format.

func (*InternalHTTPClient) FragmentBlocks added in v0.8.0 

func (c *InternalHTTPClient) FragmentBlocks(ctx context.Context, index, frame, view string, slice uint64) ([]FragmentBlock, error)

FragmentBlocks returns a list of block checksums for a fragment on a host. Only returns blocks which contain data.

func (*InternalHTTPClient) FragmentNodes added in v0.8.0 

func (c *InternalHTTPClient) FragmentNodes(ctx context.Context, index string, slice uint64) ([]*Node, error)

FragmentNodes returns a list of nodes that own a slice.

func (*InternalHTTPClient) FrameViews added in v0.8.0 

func (c *InternalHTTPClient) FrameViews(ctx context.Context, index, frame string) ([]string, error)

FrameViews returns a list of view names for a frame.

func (*InternalHTTPClient) Host added in v0.8.0 

func (c *InternalHTTPClient) Host() *URI

Host returns the host the client was initialized with.

func (*InternalHTTPClient) Import added in v0.8.0 

func (c *InternalHTTPClient) Import(ctx context.Context, index, frame string, slice uint64, bits []Bit) error

Import bulk imports bits for a single slice to a host.

func (*InternalHTTPClient) ImportK added in v0.9.0 

func (c *InternalHTTPClient) ImportK(ctx context.Context, index, frame string, bits []Bit) error

ImportK bulk imports bits to a host.

func (*InternalHTTPClient) ImportValue added in v0.8.0 

func (c *InternalHTTPClient) ImportValue(ctx context.Context, index, frame, field string, slice uint64, vals []FieldValue) error

ImportValue bulk imports field values for a single slice to a host.

func (*InternalHTTPClient) MaxInverseSliceByIndex added in v0.8.0 

func (c *InternalHTTPClient) MaxInverseSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxInverseSliceByIndex returns the number of inverse slices on a server by index.

func (*InternalHTTPClient) MaxSliceByIndex added in v0.8.0 

func (c *InternalHTTPClient) MaxSliceByIndex(ctx context.Context) (map[string]uint64, error)

MaxSliceByIndex returns the number of slices on a server by index.

func (*InternalHTTPClient) NodeID added in v0.9.0 

func (c *InternalHTTPClient) NodeID(uri *URI) (string, error)

func (*InternalHTTPClient) RestoreFrame added in v0.8.0 

func (c *InternalHTTPClient) RestoreFrame(ctx context.Context, host, index, frame string) error

RestoreFrame restores an entire frame from a host in another cluster.

func (*InternalHTTPClient) RestoreFrom added in v0.8.0 

func (c *InternalHTTPClient) RestoreFrom(ctx context.Context, r io.Reader, index, frame, view string) error

RestoreFrom restores a frame from a backup file to an entire cluster.

func (*InternalHTTPClient) RetrieveSliceFromURI added in v0.9.0 

func (c *InternalHTTPClient) RetrieveSliceFromURI(ctx context.Context, index, frame, view string, slice uint64, uri URI) (io.ReadCloser, error)

func (*InternalHTTPClient) RowAttrDiff added in v0.8.0 

func (c *InternalHTTPClient) RowAttrDiff(ctx context.Context, index, frame string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

RowAttrDiff returns data from differing blocks on a remote host.

func (*InternalHTTPClient) Schema added in v0.8.0 

func (c *InternalHTTPClient) Schema(ctx context.Context) ([]*IndexInfo, error)

Schema returns all index and frame schema information.

func (*InternalHTTPClient) SendMessage added in v0.9.0 

func (c *InternalHTTPClient) SendMessage(ctx context.Context, pb proto.Message) error

SendMessage posts a message synchronously.

type Iterator 

type Iterator interface {
	Seek(rowID, columnID uint64)
	Next() (rowID, columnID uint64, eof bool)
}

Iterator is an interface for looping over row/column pairs.

type LRUCache 

type LRUCache struct {
	// contains filtered or unexported fields
}

LRUCache represents a least recently used Cache implementation.

func NewLRUCache 

func NewLRUCache(maxEntries uint32) *LRUCache

NewLRUCache returns a new instance of LRUCache.

func (*LRUCache) Add 

func (c *LRUCache) Add(id, n uint64)

Add adds a count to the cache.

func (*LRUCache) BulkAdd 

func (c *LRUCache) BulkAdd(id, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*LRUCache) Get 

func (c *LRUCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*LRUCache) IDs 

func (c *LRUCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*LRUCache) Invalidate 

func (c *LRUCache) Invalidate()

Invalidate is a no-op.

func (*LRUCache) Len 

func (c *LRUCache) Len() int

Len returns the number of items in the cache.

func (*LRUCache) Recalculate 

func (c *LRUCache) Recalculate()

Recalculate is a no-op.

func (*LRUCache) SetStats added in v0.4.0 

func (c *LRUCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*LRUCache) Top 

func (c *LRUCache) Top() []BitmapPair

Top returns all counts in the cache.

type LimitIterator 

type LimitIterator struct {
	// contains filtered or unexported fields
}

LimitIterator wraps an Iterator and limits it to a max column/row pair.

func NewLimitIterator 

func NewLimitIterator(itr Iterator, maxRowID, maxColumnID uint64) *LimitIterator

NewLimitIterator returns a new LimitIterator.

func (*LimitIterator) Next 

func (itr *LimitIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair. If the underlying iterator returns a pair higher than the max then EOF is returned.

func (*LimitIterator) Seek 

func (itr *LimitIterator) Seek(rowID, columnID uint64)

Seek moves the underlying iterator to a column/row pair.

type MemberSet added in v0.9.0 

type MemberSet interface {
	// Open starts any network activity implemented by the MemberSet
	// Node is the local node, used for membership broadcasts.
	Open(n *Node) error
}

MemberSet represents an interface for Node membership and inter-node communication.

type MultiStatsClient 

type MultiStatsClient []StatsClient

MultiStatsClient joins multiple stats clients together.

func (MultiStatsClient) Close added in v0.8.0 

func (a MultiStatsClient) Close() error

Close shuts down the stats clients.

func (MultiStatsClient) Count 

func (a MultiStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs per second on all clients.

func (MultiStatsClient) CountWithCustomTags added in v0.4.0 

func (a MultiStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (MultiStatsClient) Gauge 

func (a MultiStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric on all clients.

func (MultiStatsClient) Histogram 

func (a MultiStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric on all clients.

func (MultiStatsClient) Open added in v0.8.0 

func (a MultiStatsClient) Open()

Open starts the stat service.

func (MultiStatsClient) Set 

func (a MultiStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements on all clients.

func (MultiStatsClient) SetLogger added in v0.4.0 

func (a MultiStatsClient) SetLogger(logger io.Writer)

SetLogger Sets the StatsD logger output type.

func (MultiStatsClient) Tags 

func (a MultiStatsClient) Tags() []string

Tags returns tags from the first client.

func (MultiStatsClient) Timing 

func (a MultiStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric on all clients.

func (MultiStatsClient) WithTags 

func (a MultiStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new set of clients with the additional tags.

type Node 

type Node struct {
	ID            string `json:"id"`
	URI           URI    `json:"uri"`
	IsCoordinator bool   `json:"isCoordinator"`
}

Node represents a node in the cluster.

func DecodeNode added in v0.9.0 

func DecodeNode(node *internal.Node) *Node

DecodeNode converts a proto message into a Node.

func DecodeNodes added in v0.9.0 

func DecodeNodes(a []*internal.Node) []*Node

DecodeNodes converts a proto message into a slice of Nodes.

func (Node) String added in v0.9.0 

func (n Node) String() string

type NodeEvent added in v0.9.0 

type NodeEvent struct {
	Event NodeEventType
	Node  *Node
}

NodeEvent is a single event related to node activity in the cluster.

func DecodeNodeEvent added in v0.9.0 

func DecodeNodeEvent(ne *internal.NodeEventMessage) *NodeEvent

type NodeEventType added in v0.9.0 

type NodeEventType int

NodeEventType are the types of events that can be sent from the ChannelEventDelegate. 

const (
	NodeJoin NodeEventType = iota
	NodeLeave
	NodeUpdate
)

type NodeIDs added in v0.9.0 

type NodeIDs []string

func (NodeIDs) ContainsID added in v0.9.0 

func (n NodeIDs) ContainsID(id string) bool

ContainsID returns true if idi matches one of the nodesets's IDs.

func (NodeIDs) Len added in v0.9.0 

func (n NodeIDs) Len() int

func (NodeIDs) Less added in v0.9.0 

func (n NodeIDs) Less(i, j int) bool

func (NodeIDs) Swap added in v0.9.0 

func (n NodeIDs) Swap(i, j int)

type Nodes 

type Nodes []*Node

Nodes represents a list of nodes.

func (Nodes) Clone 

func (a Nodes) Clone() []*Node

Clone returns a shallow copy of nodes.

func (Nodes) Contains 

func (a Nodes) Contains(n *Node) bool

Contains returns true if a node exists in the list.

func (Nodes) ContainsID added in v0.9.0 

func (a Nodes) ContainsID(id string) bool

ContainsID returns true if host matches one of the node's id.

func (Nodes) Filter 

func (a Nodes) Filter(n *Node) []*Node

Filter returns a new list of nodes with node removed.

func (Nodes) FilterID added in v0.9.0 

func (a Nodes) FilterID(id string) []*Node

FilterID returns a new list of nodes with ID removed.

func (Nodes) FilterURI added in v0.9.0 

func (a Nodes) FilterURI(uri URI) []*Node

FilterURI returns a new list of nodes with URI removed.

func (Nodes) IDs added in v0.9.0 

func (a Nodes) IDs() []string

IDs returns a list of all node IDs.

func (Nodes) URIs added in v0.9.0 

func (a Nodes) URIs() []URI

URIs returns a list of all uris.

type NopCache added in v0.5.0 

type NopCache struct {
	// contains filtered or unexported fields
}

NopCache represents a no-op Cache implementation.

func NewNopCache added in v0.5.0 

func NewNopCache() *NopCache

NewNopCache returns a new instance of NopCache.

func (*NopCache) Add added in v0.5.0 

func (c *NopCache) Add(id uint64, n uint64)

func (*NopCache) BulkAdd added in v0.5.0 

func (c *NopCache) BulkAdd(id uint64, n uint64)

func (*NopCache) Get added in v0.5.0 

func (c *NopCache) Get(id uint64) uint64

func (*NopCache) IDs added in v0.5.0 

func (c *NopCache) IDs() []uint64

func (*NopCache) Invalidate added in v0.5.0 

func (c *NopCache) Invalidate()

func (*NopCache) Len added in v0.5.0 

func (c *NopCache) Len() int

func (*NopCache) Recalculate added in v0.5.0 

func (c *NopCache) Recalculate()

func (*NopCache) SetStats added in v0.5.0 

func (c *NopCache) SetStats(s StatsClient)

func (*NopCache) Top added in v0.5.0 

func (c *NopCache) Top() []BitmapPair

type NopSecurityManager 

type NopSecurityManager struct {
}

NopSecurityManager provides a no-op implementation of the SecurityManager interface.

func (*NopSecurityManager) SetNormal 

func (sdm *NopSecurityManager) SetNormal()

SetNormal no-op.

func (*NopSecurityManager) SetRestricted 

func (sdm *NopSecurityManager) SetRestricted()

SetRestricted no-op.

type Pair 

type Pair struct {
	ID    uint64 `json:"id"`
	Key   string `json:"key,omitempty"`
	Count uint64 `json:"count"`
}

Pair holds an id/count pair.

type PairHeap 

type PairHeap struct {
	Pairs
}

PairHeap is a heap implementation over a group of Pairs.

func (PairHeap) Less 

func (p PairHeap) Less(i, j int) bool

Less implemets the Sort interface. reports whether the element with index i should sort before the element with index j.

type PairSet 

type PairSet struct {
	RowIDs    []uint64
	ColumnIDs []uint64
}

PairSet is a list of equal length row and column id lists.

type Pairs 

type Pairs []Pair

Pairs is a sortable slice of Pair objects.

func (Pairs) Add 

func (p Pairs) Add(other []Pair) []Pair

Add merges other into p and returns a new slice.

func (Pairs) Keys 

func (p Pairs) Keys() []uint64

Keys returns a slice of all keys in p.

func (Pairs) Len 

func (p Pairs) Len() int

func (Pairs) Less 

func (p Pairs) Less(i, j int) bool

func (*Pairs) Pop 

func (p *Pairs) Pop() interface{}

Pop removes the minimum element from the Pair slice.

func (*Pairs) Push 

func (p *Pairs) Push(x interface{})

Push appends the element onto the Pair slice.

func (Pairs) String 

func (p Pairs) String() string

func (Pairs) Swap 

func (p Pairs) Swap(i, j int)

type QueryRequest 

type QueryRequest struct {
	// Index to execute query against.
	Index string

	// The query string to parse and execute.
	Query string

	// The slices to include in the query execution.
	// If empty, all slices are included.
	Slices []uint64

	// Return column attributes, if true.
	ColumnAttrs bool

	// Do not return row attributes, if true.
	ExcludeAttrs bool

	// Do not return bits, if true.
	ExcludeBits bool

	// If true, indicates that query is part of a larger distributed query.
	// If false, this request is on the originating node.
	Remote bool
}

QueryRequest represent a request to process a query.

type QueryResponse 

type QueryResponse struct {
	// Result for each top-level query call.
	// Can be a Bitmap, Pairs, or uint64.
	Results []interface{}

	// Set of column attribute objects matching IDs returned in Result.
	ColumnAttrSets []*ColumnAttrSet

	// Error during parsing or execution.
	Err error
}

QueryResponse represent a response from a processed query.

func (*QueryResponse) MarshalJSON 

func (resp *QueryResponse) MarshalJSON() ([]byte, error)

MarshalJSON marshals QueryResponse into a JSON-encoded byte slice

type RankCache 

type RankCache struct {
	// contains filtered or unexported fields
}

RankCache represents a cache with sorted entries.

func NewRankCache 

func NewRankCache(maxEntries uint32) *RankCache

NewRankCache returns a new instance of RankCache.

func (*RankCache) Add 

func (c *RankCache) Add(id uint64, n uint64)

Add adds a count to the cache.

func (*RankCache) BulkAdd 

func (c *RankCache) BulkAdd(id uint64, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*RankCache) Get 

func (c *RankCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*RankCache) IDs 

func (c *RankCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*RankCache) Invalidate 

func (c *RankCache) Invalidate()

Invalidate recalculates the entries by rank.

func (*RankCache) Len 

func (c *RankCache) Len() int

Len returns the number of items in the cache.

func (*RankCache) ReadFrom 

func (c *RankCache) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom read from r into the cache.

func (*RankCache) Recalculate 

func (c *RankCache) Recalculate()

Recalculate rebuilds the cache.

func (*RankCache) SetStats added in v0.4.0 

func (c *RankCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*RankCache) Top 

func (c *RankCache) Top() []BitmapPair

Top returns an ordered list of pairs.

func (*RankCache) WriteTo 

func (c *RankCache) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the cache to w.

type ResizeJob added in v0.9.0 

type ResizeJob struct {
	ID           int64
	IDs          map[string]bool
	Instructions []*internal.ResizeInstruction
	Broadcaster  Broadcaster

	// The writer for any logging.
	LogOutput io.Writer
	// contains filtered or unexported fields
}

func NewResizeJob added in v0.9.0 

func NewResizeJob(existingNodes []*Node, node *Node, action string) *ResizeJob

NewResizeJob returns a new instance of ResizeJob.

func (*ResizeJob) Run added in v0.9.0 

func (j *ResizeJob) Run() error

Run distributes ResizeInstructions.

func (*ResizeJob) SetState added in v0.9.0 

func (j *ResizeJob) SetState(state string)

func (*ResizeJob) State added in v0.9.0 

func (j *ResizeJob) State() string

type RoaringIterator 

type RoaringIterator struct {
	// contains filtered or unexported fields
}

RoaringIterator converts a roaring.Iterator to output column/row pairs.

func NewRoaringIterator 

func NewRoaringIterator(itr *roaring.Iterator) *RoaringIterator

NewRoaringIterator returns a new iterator wrapping itr.

func (*RoaringIterator) Next 

func (itr *RoaringIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next column/row ID pair.

func (*RoaringIterator) Seek 

func (itr *RoaringIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a pair matching bseek/pseek. If the pair is not found then it moves to the next pair.

type SecurityManager 

type SecurityManager interface {
	SetRestricted()
	SetNormal()
}

SecurityManager provides the ability to limit access to restricted endpoints during cluster configuration.

type Server 

type Server struct {

	// Data storage and HTTP interface.
	Holder            *Holder
	Handler           *Handler
	Broadcaster       Broadcaster
	BroadcastReceiver BroadcastReceiver
	Gossiper          Gossiper
	RemoteClient      *http.Client

	// Cluster configuration.
	Network string
	NodeID  string
	URI     URI
	Cluster *Cluster

	GCNotifier GCNotifier

	// Background monitoring intervals.
	AntiEntropyInterval time.Duration
	MetricInterval      time.Duration
	DiagnosticInterval  time.Duration

	// TLS configuration
	TLS *tls.Config

	// Misc options.
	MaxWritesPerRequest int

	LogOutput io.Writer
	// contains filtered or unexported fields
}

Server represents a holder wrapped by a running HTTP server.

func NewServer 

func NewServer() *Server

NewServer returns a new instance of Server.

func (*Server) Addr 

func (s *Server) Addr() net.Addr

Addr returns the address of the listener.

func (*Server) Close 

func (s *Server) Close() error

Close closes the server and waits for it to shutdown.

func (*Server) ClusterStatus 

func (s *Server) ClusterStatus() (proto.Message, error)

ClusterStatus returns the ClusterState and NodeSet for the cluster.

func (*Server) HandleRemoteStatus 

func (s *Server) HandleRemoteStatus(pb proto.Message) error

HandleRemoteStatus receives incoming NodeStatus from remote nodes.

func (*Server) LoadNodeID added in v0.9.0 

func (s *Server) LoadNodeID() string

LoadNodeID gets NodeID from disk, or creates a new value. If server.NodeID is already set, a new ID is not created.

func (*Server) LocalStatus 

func (s *Server) LocalStatus() (proto.Message, error)

Server implements StatusHandler. LocalStatus is used to periodically sync information between nodes. Under normal conditions, nodes should remain in sync through Broadcast messages. For cases where a node fails to receive a Broadcast message, or when a new (empty) node needs to get in sync with the rest of the cluster, two things are shared via gossip: - MaxSlice/MaxInverseSlice by Index - Schema In a gossip implementation, memberlist.Delegate.LocalState() uses this.

func (*Server) Logger added in v0.5.0 

func (s *Server) Logger() *log.Logger

Logger returns a logger that writes to LogOutput

func (*Server) Open 

func (s *Server) Open() error

Open opens and initializes the server.

func (*Server) OpenListener 

func (s *Server) OpenListener() error

OpenListener opens a listener for the Server.

func (*Server) ReceiveMessage 

func (s *Server) ReceiveMessage(pb proto.Message) error

ReceiveMessage represents an implementation of BroadcastHandler.

func (*Server) SendAsync added in v0.9.0 

func (s *Server) SendAsync(pb proto.Message) error

SendAsync represents an implementation of Broadcaster.

func (*Server) SendSync added in v0.9.0 

func (s *Server) SendSync(pb proto.Message) error

SendSync represents an implementation of Broadcaster.

func (*Server) SendTo added in v0.9.0 

func (s *Server) SendTo(to *Node, pb proto.Message) error

SendTo represents an implementation of Broadcaster.

type SimpleCache 

type SimpleCache struct {
	// contains filtered or unexported fields
}

SimpleCache implements BitmapCache it is meant to be a short-lived cache for cases where writes are continuing to access the same bit within a short time frame (i.e. good for write-heavy loads) A read-heavy use case would cause the cache to get bigger, potentially causing the node to run out of memory.

func (*SimpleCache) Add 

func (s *SimpleCache) Add(id uint64, b *Bitmap)

Add adds the bitmap to the cache, keyed on the id.

func (*SimpleCache) Fetch 

func (s *SimpleCache) Fetch(id uint64) (*Bitmap, bool)

Fetch retrieves the bitmap at the id in the cache.

type SliceIterator 

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator iterates over a pair of row/column ID slices.

func NewSliceIterator 

func NewSliceIterator(rowIDs, columnIDs []uint64) *SliceIterator

NewSliceIterator returns an iterator to iterate over a set of row/column ID pairs. Both slices MUST have an equal length. Otherwise the function will panic.

func (*SliceIterator) Next 

func (itr *SliceIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair.

func (*SliceIterator) Seek 

func (itr *SliceIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a given pair. If the pair is not found, the iterator seeks to the next pair.

type StaticMemberSet added in v0.9.0 

type StaticMemberSet struct {
	// contains filtered or unexported fields
}

StaticMemberSet represents a basic MemberSet for testing.

func NewStaticMemberSet added in v0.9.0 

func NewStaticMemberSet(nodes []*Node) *StaticMemberSet

NewStaticMemberSet creates a statically defined MemberSet.

func (*StaticMemberSet) Open added in v0.9.0 

func (s *StaticMemberSet) Open(n *Node) error

Open implements the MemberSet interface to start network activity, but for a static MemberSet it does nothing.

type StatsClient 

type StatsClient interface {
	// Returns a sorted list of tags on the client.
	Tags() []string

	// Returns a new client with additional tags appended.
	WithTags(tags ...string) StatsClient

	// Tracks the number of times something occurs per second.
	Count(name string, value int64, rate float64)

	// Tracks the number of times something occurs per second with custom tags
	CountWithCustomTags(name string, value int64, rate float64, tags []string)

	// Sets the value of a metric.
	Gauge(name string, value float64, rate float64)

	// Tracks statistical distribution of a metric.
	Histogram(name string, value float64, rate float64)

	// Tracks number of unique elements.
	Set(name string, value string, rate float64)

	// Tracks timing information for a metric.
	Timing(name string, value time.Duration, rate float64)

	// SetLogger Set the logger output type
	SetLogger(logger io.Writer)

	// Starts the service
	Open()

	// Closes the client
	Close() error
}

StatsClient represents a client to a stats server. 

var NopStatsClient StatsClient

NopStatsClient represents a client that doesn't do anything.

type StatusHandler 

type StatusHandler interface {
	LocalStatus() (proto.Message, error)
	ClusterStatus() (proto.Message, error)
	HandleRemoteStatus(proto.Message) error
}

StatusHandler specifies the methods which an object must implement to share state in the cluster. These are used by the GossipMemberSet to implement the LocalState and MergeRemoteState methods of memberlist.Delegate

type SumCount added in v0.7.0 

type SumCount struct {
	Sum   int64 `json:"sum"`
	Count int64 `json:"count"`
}

SumCount represents a grouping of sum & count for Sum() and Average() calls.

func (*SumCount) Add added in v0.7.0 

func (sc *SumCount) Add(other SumCount) SumCount

type TLSConfig added in v0.8.0 

type TLSConfig struct {
	// CertificatePath contains the path to the certificate (.crt or .pem file)
	CertificatePath string `toml:"certificate-path"`
	// CertificateKeyPath contains the path to the certificate key (.key file)
	CertificateKeyPath string `toml:"certificate-key-path"`
	// SkipVerify disables verification for self-signed certificates
	SkipVerify bool `toml:"skip-verify"`
}

TLSConfig contains TLS configuration

type TimeQuantum 

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func ParseTimeQuantum 

func ParseTimeQuantum(v string) (TimeQuantum, error)

ParseTimeQuantum parses v into a time quantum.

func (TimeQuantum) HasDay 

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour 

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth 

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear 

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (*TimeQuantum) Set added in v0.7.0 

func (q *TimeQuantum) Set(value string) error

Set sets the time quantum value.

func (TimeQuantum) String added in v0.7.0 

func (q TimeQuantum) String() string

func (TimeQuantum) Type added in v0.7.0 

func (q TimeQuantum) Type() string

Type returns the type of a time quantum value.

func (TimeQuantum) Valid 

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type TopOptions 

type TopOptions struct {
	// Number of rows to return.
	N int

	// Bitmap to intersect with.
	Src *Bitmap

	// Specific rows to filter against.
	RowIDs       []uint64
	MinThreshold uint64

	// Filter field name & values.
	FilterField       string
	FilterValues      []interface{}
	TanimotoThreshold uint64
}

TopOptions represents options passed into the Top() function.

type Topology added in v0.9.0 

type Topology struct {
	NodeIDs []string

	ClusterID string
	// contains filtered or unexported fields
}

Topology represents the list of hosts in the cluster.

func NewTopology added in v0.9.0 

func NewTopology() *Topology

func (*Topology) AddID added in v0.9.0 

func (t *Topology) AddID(nodeID string) bool

AddID adds the node ID to the topology and returns true if added.

func (*Topology) ContainsID added in v0.9.0 

func (t *Topology) ContainsID(id string) bool

ContainsID returns true if id matches one of the topology's IDs.

func (*Topology) Encode added in v0.9.0 

func (t *Topology) Encode() *internal.Topology

Encode converts t into its internal representation.

func (*Topology) RemoveID added in v0.9.0 

func (t *Topology) RemoveID(nodeID string) bool

RemoveID removes the node ID from the topology and returns true if removed.

type URI added in v0.8.0 

type URI struct {
	// contains filtered or unexported fields
}

URI represents a Pilosa URI. A Pilosa URI consists of three parts: 1) Scheme: Protocol of the URI. Default: http. 2) Host: Hostname or IP URI. Default: localhost. IPv6 addresses should be written in brackets, e.g., `[fd42:4201:f86b:7e09:216:3eff:fefa:ed80]`. 3) Port: Port of the URI. Default: 10101.

All parts of the URI are optional. The following are equivalent: 

http://localhost:10101
http://localhost
http://:10101
localhost:10101
localhost
:10101

func AddressWithDefaults added in v0.6.0 

func AddressWithDefaults(addr string) (*URI, error)

AddressWithDefaults converts addr into a valid address, using defaults when necessary.

func DecodeURI added in v0.9.0 

func DecodeURI(i *internal.URI) URI

func DefaultURI added in v0.8.0 

func DefaultURI() *URI

DefaultURI creates and returns the default URI.

func NewURIFromAddress added in v0.8.0 

func NewURIFromAddress(address string) (*URI, error)

NewURIFromAddress parses the passed address and returns a URI.

func NewURIFromHostPort added in v0.8.0 

func NewURIFromHostPort(host string, port uint16) (*URI, error)

NewURIFromHostPort returns a URI with specified host and port.

func (URI) Encode added in v0.9.0 

func (u URI) Encode() *internal.URI

Encode converts o into its internal representation.

func (URI) Equals added in v0.8.0 

func (u URI) Equals(other *URI) bool

Equals returns true if the checked URI is equivalent to this URI.

func (*URI) Host added in v0.8.0 

func (u *URI) Host() string

Host returns the host of this URI.

func (*URI) HostPort added in v0.8.0 

func (u *URI) HostPort() string

HostPort returns `Host:Port`

func (*URI) MarshalJSON added in v0.9.0 

func (u *URI) MarshalJSON() ([]byte, error)

MarshalJSON marshals URI into a JSON-encoded byte slice.

func (*URI) Normalize added in v0.8.0 

func (u *URI) Normalize() string

Normalize returns the address in a form usable by a HTTP client.

func (*URI) Path added in v0.8.0 

func (u *URI) Path(path string) string

Path returns URI with path

func (*URI) Port added in v0.8.0 

func (u *URI) Port() uint16

Port returns the port of this URI.

func (*URI) Scheme added in v0.8.0 

func (u *URI) Scheme() string

Scheme returns the scheme of this URI.

func (*URI) Set added in v0.8.0 

func (u *URI) Set(value string) error

Set sets the time quantum value.

func (*URI) SetHost added in v0.8.0 

func (u *URI) SetHost(host string) error

SetHost sets the host of this URI.

func (*URI) SetPort added in v0.8.0 

func (u *URI) SetPort(port uint16)

SetPort sets the port of this URI.

func (*URI) SetScheme added in v0.8.0 

func (u *URI) SetScheme(scheme string) error

SetScheme sets the scheme of this URI.

func (URI) String added in v0.8.0 

func (u URI) String() string

String returns the address as a string.

func (URI) Type added in v0.8.0 

func (u URI) Type() string

Type returns the type of a time quantum value.

func (*URI) UnmarshalJSON added in v0.9.0 

func (u *URI) UnmarshalJSON(b []byte) error

type URIs added in v0.9.0 

type URIs []URI

func (URIs) HostPortStrings added in v0.9.0 

func (u URIs) HostPortStrings() []string

type View 

type View struct {
	RowAttrStore *AttrStore
	LogOutput    io.Writer
	// contains filtered or unexported fields
}

View represents a container for frame data.

func NewView 

func NewView(path, index, frame, name string, cacheSize uint32) *View

NewView returns a new instance of View.

func (*View) ClearBit 

func (v *View) ClearBit(rowID, columnID uint64) (changed bool, err error)

ClearBit clears a bit within the view.

func (*View) Close 

func (v *View) Close() error

Close closes the view and its fragments.

func (*View) CreateFragmentIfNotExists 

func (v *View) CreateFragmentIfNotExists(slice uint64) (*Fragment, error)

CreateFragmentIfNotExists returns a fragment in the view by slice.

func (*View) DeleteFragment added in v0.9.0 

func (v *View) DeleteFragment(slice uint64) error

DeleteFragment removes the fragment from the view.

func (*View) FieldRange added in v0.6.0 

func (v *View) FieldRange(op pql.Token, bitDepth uint, predicate uint64) (*Bitmap, error)

FieldRange returns bitmaps with a field value encoding matching the predicate.

func (*View) FieldRangeBetween added in v0.7.0 

func (v *View) FieldRangeBetween(bitDepth uint, predicateMin, predicateMax uint64) (*Bitmap, error)

FieldRangeBetween returns bitmaps with a field value encoding matching any value between predicateMin and predicateMax.

func (*View) FieldSum added in v0.7.0 

func (v *View) FieldSum(filter *Bitmap, bitDepth uint) (sum, count uint64, err error)

FieldSum returns the sum & count of a field.

func (*View) FieldValue added in v0.5.0 

func (v *View) FieldValue(columnID uint64, bitDepth uint) (value uint64, exists bool, err error)

FieldValue uses a column of bits to read a multi-bit value.

func (*View) Fragment 

func (v *View) Fragment(slice uint64) *Fragment

Fragment returns a fragment in the view by slice.

func (*View) FragmentPath 

func (v *View) FragmentPath(slice uint64) string

FragmentPath returns the path to a fragment in the view.

func (*View) Fragments 

func (v *View) Fragments() []*Fragment

Fragments returns a list of all fragments in the view.

func (*View) Frame 

func (v *View) Frame() string

Frame returns the frame name the view was initialized with.

func (*View) Index 

func (v *View) Index() string

Index returns the index name the view was initialized with.

func (*View) MaxSlice 

func (v *View) MaxSlice() uint64

MaxSlice returns the max slice in the view.

func (*View) Name 

func (v *View) Name() string

Name returns the name the view was initialized with.

func (*View) Open 

func (v *View) Open() error

Open opens and initializes the view.

func (*View) Path 

func (v *View) Path() string

Path returns the path the view was initialized with.

func (*View) RecalculateCaches added in v0.8.0 

func (v *View) RecalculateCaches()

RecalculateCaches recalculates the cache on every fragment in the view.

func (*View) SetBit 

func (v *View) SetBit(rowID, columnID uint64) (changed bool, err error)

SetBit sets a bit within the view.

func (*View) SetFieldValue added in v0.5.0 

func (v *View) SetFieldValue(columnID uint64, bitDepth uint, value uint64) (changed bool, err error)

SetFieldValue uses a column of bits to set a multi-bit value.

type ViewInfo 

type ViewInfo struct {
	Name string `json:"name"`
}

ViewInfo represents schema information for a view.

Source Files

View all Source files

Directories

Path Synopsis
cmd
Package cmd contains all the pilosa subcommand definitions (1 per file).
 Package cmd contains all the pilosa subcommand definitions (1 per file).
pilosa command
This is the entrypoint for the Pilosa binary.
 This is the entrypoint for the Pilosa binary.
package ctl contains all pilosa subcommands other than 'server'.
 package ctl contains all pilosa subcommands other than 'server'.
Package internal is a generated protocol buffer package.
 Package internal is a generated protocol buffer package.
Package pql defines the Pilosa Query Language.
 Package pql defines the Pilosa Query Language.
Package roaring implements roaring bitmaps with support for incremental changes.
 Package roaring implements roaring bitmaps with support for incremental changes.
Package statik contains static assets for the Web UI.
 Package statik contains static assets for the Web UI.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.