pilosa

package module
v0.10.1-0...-34e24af Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2018 License: Apache-2.0 Imports: 42 Imported by: 109

README

Build Status GoDoc Go Report Card license CLA Assistant GitHub release

An open source, distributed bitmap index.

Want to contribute? One of the easiest ways is to tell us how you're using (or want to use) Pilosa. We learn from every discussion!

Docs

See our Documentation for information about installing and working with Pilosa.

Getting Started

  1. Install Pilosa.

  2. Start Pilosa with the default configuration:

    pilosa server

    and verify that it's running:

    curl localhost:10101/nodes

  3. Follow along with the Sample Project to get a better understanding of Pilosa's capabilities.

Data Model

Check out how the Pilosa Data Model works.

Query Language

You can interact with Pilosa directly in the console using the Pilosa Query Language (PQL).

Client Libraries

There are supported libraries for the following languages:

Licenses

The core Pilosa code base and all default builds (referred to as Pilosa Community Edition) are licensed completely under the Apache License, Version 2.0. If you build Pilosa with the enterprise build tag (Pilosa Enterprise Edition), then that build will include features licensed under the GNU Affero General Public License (AGPL). Enterprise code is located entirely in the github.com/pilosa/pilosa/enterprise directory. See github.com/pilosa/pilosa/NOTICE and github.com/pilosa/pilosa/LICENSE for more information about Pilosa licenses.

Get Support

There are several channels available for you to reach out to us for support.

Contributing

Pilosa is an open source project. Please see our Contributing Guide for information about how to get involved.

Documentation

Overview

Package pilosa implements the core of the Pilosa distributed bitmap index. It contains all the domain objects, interfaces, and logic that defines pilosa.

Index

Constants

View Source 
const (
	// DefaultPartitionN is the default number of partitions in a cluster.
	DefaultPartitionN = 256

	// ClusterState represents the state returned in the /status endpoint.
	ClusterStateStarting = "STARTING"
	ClusterStateNormal   = "NORMAL"
	ClusterStateResizing = "RESIZING"

	// NodeState represents the state of a node during startup.
	NodeStateLoading = "LOADING"
	NodeStateReady   = "READY"
)
View Source 
const (
	DefaultFieldType = FieldTypeSet

	DefaultCacheType = CacheTypeRanked

	// Default ranked field cache
	DefaultCacheSize = 50000
)

Default field settings.

View Source 
const (
	FieldTypeSet  = "set"
	FieldTypeInt  = "int"
	FieldTypeTime = "time"
)

Field types.

View Source 
const (
	CacheTypeLRU    = "lru"
	CacheTypeRanked = "ranked"
	CacheTypeNone   = "none"
)

Cache types.

View Source 
const (
	// ShardWidth is the number of column IDs in a shard.
	ShardWidth = 1048576

	// HashBlockSize is the number of rows in a merkle hash block.
	HashBlockSize = 100
)
View Source 
const (
	LogEntryTypeInsertColumn = 1
	LogEntryTypeInsertRow    = 2
)
View Source 
const (
	DefaultDiagnosticServer = "https://diagnostics.pilosa.com/v0/diagnostics"
)

Default server settings.

View Source 
const DefaultMapSize = 10 * (1 << 30) // 10GB

DefaultMapSize is the default size of mapped memory for the translate store. It is passed as an int to syscall.Mmap and so can only be larger than 2^31 on 64bit systems.

View Source 
const (
	DefaultReplicationRetryInterval = 1 * time.Second
)
View Source 
const (

	// FileLimit is the maximum open file limit (ulimit -n) to automatically set.
	FileLimit = 262144 // (512^2)
)
View Source 
const (
	ReplicationBufferSize = 65536
)
View Source 
const TimeFormat = "2006-01-02T15:04"

TimeFormat is the go-style time format used to parse string dates.

View Source 
const TranslateStoreBufferSize = 65536

TranslateStoreBufferSize is the buffer size used for streaming data.

View Source 
const (
	ViewStandard = "standard"
)

View layout modes.

Variables

View Source 
var (
	ErrHostRequired = errors.New("host required")

	ErrIndexRequired = errors.New("index required")
	ErrIndexExists   = errors.New("index already exists")
	ErrIndexNotFound = errors.New("index not found")

	// ErrFieldRequired is returned when no field is specified.
	ErrFieldRequired = errors.New("field required")
	ErrFieldExists   = errors.New("field already exists")
	ErrFieldNotFound = errors.New("field not found")

	ErrBSIGroupNotFound         = errors.New("bsigroup not found")
	ErrBSIGroupExists           = errors.New("bsigroup already exists")
	ErrBSIGroupNameRequired     = errors.New("bsigroup name required")
	ErrInvalidBSIGroupType      = errors.New("invalid bsigroup type")
	ErrInvalidBSIGroupRange     = errors.New("invalid bsigroup range")
	ErrInvalidBSIGroupValueType = errors.New("invalid bsigroup value type")
	ErrBSIGroupValueTooLow      = errors.New("bsigroup value too low")
	ErrBSIGroupValueTooHigh     = errors.New("bsigroup value too high")
	ErrInvalidRangeOperation    = errors.New("invalid range operation")
	ErrInvalidBetweenValue      = errors.New("invalid value for between operation")

	ErrInvalidView      = errors.New("invalid view")
	ErrInvalidCacheType = errors.New("invalid cache type")

	ErrName  = errors.New("invalid index or field name, must match [a-z0-9_-]")
	ErrLabel = errors.New("invalid row or column label, must match [A-Za-z0-9_-]")

	// ErrFragmentNotFound is returned when a fragment does not exist.
	ErrFragmentNotFound = errors.New("fragment not found")
	ErrQueryRequired    = errors.New("query required")
	ErrTooManyWrites    = errors.New("too many write commands")

	ErrClusterDoesNotOwnShard = errors.New("cluster does not own shard")

	ErrNodeIDNotExists    = errors.New("node with provided ID does not exist")
	ErrNodeNotCoordinator = errors.New("node is not the coordinator")
	ErrResizeNotRunning   = errors.New("no resize job currently running")

	ErrNotImplemented = errors.New("not implemented")
)

System errors.

View Source 
var (
	ErrTranslateStoreClosed       = errors.New("pilosa: translate store closed")
	ErrTranslateStoreReaderClosed = errors.New("pilosa: translate store reader closed")
	ErrReplicationNotSupported    = errors.New("pilosa: replication not supported")
	ErrTranslateStoreReadOnly     = errors.New("pilosa: operation not supported, translate store read only")
)
View Source 
var BuildTime = "not recorded"
View Source 
var Enterprise = "0"
View Source 
var EnterpriseEnabled = false
View Source 
var ErrInvalidTimeQuantum = errors.New("invalid time quantum")

ErrInvalidTimeQuantum is returned when parsing a time quantum.

View Source 
var Expvar = expvar.NewMap("index")

Expvar global expvar map.

View Source 
var Version = "v0.0.0"

Functions

func DecodeAttrs added in v0.9.0 

func DecodeAttrs(v []byte) (map[string]interface{}, error)

DecodeAttrs decodes a byte slice into an attribute map.

func DecodeNodeEvent added in v0.9.0 

func DecodeNodeEvent(ne *internal.NodeEventMessage) *nodeEvent

func EncodeAttrs added in v0.9.0 

func EncodeAttrs(attr map[string]interface{}) ([]byte, error)

EncodeAttrs encodes an attribute map into a byte slice.

func EncodeColumnAttrSet 

func EncodeColumnAttrSet(set *ColumnAttrSet) *internal.ColumnAttrSet

EncodeColumnAttrSet converts set into its internal representation.

func EncodeColumnAttrSets 

func EncodeColumnAttrSets(a []*ColumnAttrSet) []*internal.ColumnAttrSet

EncodeColumnAttrSets converts a into its internal representation.

func EncodeIndexes added in v0.4.0 

func EncodeIndexes(a []*Index) []*internal.Index

EncodeIndexes converts a into its internal representation.

func EncodeNode added in v0.9.0 

func EncodeNode(n *Node) *internal.Node

EncodeNode converts a Node into its internal representation.

func EncodeNodes added in v0.9.0 

func EncodeNodes(a []*Node) []*internal.Node

EncodeNodes converts a slice of Nodes into its internal representation.

func EncodePairs 

func EncodePairs(a Pairs) []*internal.Pair

func EncodeRow 

func EncodeRow(r *Row) *internal.Row

EncodeRow converts r into its internal representation.

func EncodeValCount 

func EncodeValCount(vc ValCount) *internal.ValCount

func MarshalMessage 

func MarshalMessage(m proto.Message) ([]byte, error)

MarshalMessage encodes the protobuf message into a byte slice.

func NewCluster 

func NewCluster() *cluster

NewCluster returns a new instance of Cluster with defaults.

func UnmarshalMessage 

func UnmarshalMessage(buf []byte) (proto.Message, error)

UnmarshalMessage decodes the byte slice into a protobuf message.

func UvarintSize 

func UvarintSize(x uint64) (i int)

Copied & modified from encoding/binary.

func ValidLogEntriesLen 

func ValidLogEntriesLen(p []byte) (n int)

ValidLogEntriesLen returns the maximum length of p that contains valid entries.

Types

type API added in v0.9.0 

type API struct {
	// contains filtered or unexported fields
}

API provides the top level programmatic interface to Pilosa. It is usually wrapped by a handler which provides an external interface (e.g. HTTP).

func NewAPI added in v0.9.0 

func NewAPI(opts ...APIOption) (*API, error)

NewAPI returns a new API instance.

func (*API) ClusterMessage added in v0.9.0 

func (api *API) ClusterMessage(ctx context.Context, reqBody io.Reader) error

PostClusterMessage is for internal use. It decodes a protobuf message out of the body and forwards it to the BroadcastHandler.

func (*API) CreateField added in v0.9.0 

func (api *API) CreateField(ctx context.Context, indexName string, fieldName string, opts FieldOption) (*Field, error)

CreateField makes the named field in the named index with the given options. This method currently only takes a single functional option, but that may be changed in the future to support multiple options.

func (*API) CreateIndex added in v0.9.0 

func (api *API) CreateIndex(ctx context.Context, indexName string, options IndexOptions) (*Index, error)

CreateIndex makes a new Pilosa index.

func (*API) DeleteField added in v0.9.0 

func (api *API) DeleteField(ctx context.Context, indexName string, fieldName string) error

DeleteField removes the named field from the named index. If the index is not found, an error is returned. If the field is not found, it is ignored and no action is taken.

func (*API) DeleteIndex added in v0.9.0 

func (api *API) DeleteIndex(ctx context.Context, indexName string) error

DeleteIndex removes the named index. If the index is not found it does nothing and returns no error.

func (*API) DeleteView added in v0.9.0 

func (api *API) DeleteView(ctx context.Context, indexName string, fieldName string, viewName string) error

DeleteView removes the given view.

func (*API) ExportCSV added in v0.9.0 

func (api *API) ExportCSV(ctx context.Context, indexName string, fieldName string, shard uint64, w io.Writer) error

ExportCSV encodes the fragment designated by the index,field,shard as CSV of the form <row>,<col>

func (*API) FieldAttrDiff added in v1.0.0 

func (api *API) FieldAttrDiff(ctx context.Context, indexName string, fieldName string, blocks []AttrBlock) (map[uint64]map[string]interface{}, error)

func (*API) FragmentBlockData added in v0.9.0 

func (api *API) FragmentBlockData(ctx context.Context, body io.Reader) ([]byte, error)

FragmentBlockData is an endpoint for internal usage. It is not guaranteed to return anything useful. Currently it returns protobuf encoded row and column ids from a "block" which is a subdivision of a fragment.

func (*API) FragmentBlocks added in v0.9.0 

func (api *API) FragmentBlocks(ctx context.Context, indexName string, fieldName string, shard uint64) ([]FragmentBlock, error)

FragmentBlocks returns the checksums and block ids for all blocks in the specified fragment.

func (*API) GetTranslateData added in v1.0.0 

func (api *API) GetTranslateData(ctx context.Context, w io.WriteCloser, offset int64) error

func (*API) Hosts added in v0.9.0 

func (api *API) Hosts(ctx context.Context) []*Node

Hosts returns a list of the hosts in the cluster including their ID, URL, and which is the coordinator.

func (*API) Import added in v0.9.0 

func (api *API) Import(ctx context.Context, req internal.ImportRequest) error

Import bulk imports data into a particular index,field,shard.

func (*API) ImportValue added in v0.9.0 

func (api *API) ImportValue(ctx context.Context, req internal.ImportValueRequest) error

ImportValue bulk imports values into a particular field.

func (*API) Index added in v0.9.0 

func (api *API) Index(ctx context.Context, indexName string) (*Index, error)

Index retrieves the named index.

func (*API) IndexAttrDiff added in v0.9.0 

func (api *API) IndexAttrDiff(ctx context.Context, indexName string, blocks []AttrBlock) (map[uint64]map[string]interface{}, error)

IndexAttrDiff

func (*API) Info added in v0.10.0 

func (api *API) Info() serverInfo

Info returns information about this server instance

func (*API) LocalID added in v0.9.0 

func (api *API) LocalID() string

LocalID returns the current node's ID.

func (*API) LongQueryTime added in v0.9.0 

func (api *API) LongQueryTime() time.Duration

LongQueryTime returns the configured threshold for logging/statting long running queries.

func (*API) MarshalFragment added in v0.9.0 

func (api *API) MarshalFragment(ctx context.Context, indexName string, fieldName string, shard uint64) (io.WriterTo, error)

MarshalFragment returns an object which can write the specified fragment's data to an io.Writer. The serialized data can be read back into a fragment with the UnmarshalFragment API call.

func (*API) MaxShards added in v1.0.0 

func (api *API) MaxShards(ctx context.Context) map[string]uint64

MaxShards returns the maximum shard number for each index in a map.

func (*API) Query added in v0.9.0 

func (api *API) Query(ctx context.Context, req *QueryRequest) (QueryResponse, error)

Query parses a PQL query out of the request and executes it.

func (*API) RecalculateCaches added in v0.9.0 

func (api *API) RecalculateCaches(ctx context.Context) error

RecalculateCaches forces all TopN caches to be updated. Used mainly for integration tests.

func (*API) RemoveNode added in v0.9.0 

func (api *API) RemoveNode(id string) (*Node, error)

RemoveNode puts the cluster into the "RESIZING" state and begins the job of removing the given node.

func (*API) ResizeAbort added in v0.9.0 

func (api *API) ResizeAbort() error

ResizeAbort stops the current resize job.

func (*API) Schema added in v0.9.0 

func (api *API) Schema(ctx context.Context) []*IndexInfo

Schema returns information about each index in Pilosa including which fields and views they contain.

func (*API) SetCoordinator added in v0.9.0 

func (api *API) SetCoordinator(ctx context.Context, id string) (oldNode, newNode *Node, err error)

SetCoordinator makes a new Node the cluster coordinator.

func (*API) ShardNodes added in v1.0.0 

func (api *API) ShardNodes(ctx context.Context, indexName string, shard uint64) ([]*Node, error)

ShardNodes returns the node and all replicas which should contain a shard's data.

func (*API) State added in v0.9.0 

func (api *API) State() string

State returns the cluster state which is usually "NORMAL", but could be "STARTING", "RESIZING", or potentially others. See cluster.go for more details.

func (*API) StatsWithTags added in v0.9.0 

func (api *API) StatsWithTags(tags []string) StatsClient

StatsWithTags returns an instance of whatever implementation of StatsClient pilosa is using with the given tags.

func (*API) UnmarshalFragment added in v0.9.0 

func (api *API) UnmarshalFragment(ctx context.Context, indexName string, fieldName string, shard uint64, reader io.ReadCloser) error

UnmarshalFragment creates a new fragment (if necessary) and reads data from a Reader which was previously written by MarshalFragment to populate the fragment's data.

func (*API) Version added in v0.9.0 

func (api *API) Version() string

Version returns the Pilosa version.

func (*API) Views added in v0.9.0 

func (api *API) Views(ctx context.Context, indexName string, fieldName string) ([]*View, error)

Views returns the views in the given field.

type APIOption 

type APIOption func(*API) error

APIOption is a functional option type for pilosa.API

func OptAPIServer added in v1.0.0 

func OptAPIServer(s *Server) APIOption

type ApiMethodNotAllowedError added in v0.9.0 

type ApiMethodNotAllowedError struct {
	// contains filtered or unexported fields
}

ApiMethodNotAllowedError wraps an error value indicating that a particular API method is not allowed in the current cluster state.

type AttrBlock 

type AttrBlock struct {
	ID       uint64 `json:"id"`
	Checksum []byte `json:"checksum"`
}

AttrBlock represents a checksummed block of the attribute store.

type AttrBlocks 

type AttrBlocks []AttrBlock

AttrBlocks represents a list of blocks.

func (AttrBlocks) Diff 

func (a AttrBlocks) Diff(other []AttrBlock) []uint64

Diff returns a list of block ids that are different or are new in other. Block lists must be in sorted order.

type AttrStore 

type AttrStore interface {
	Path() string
	Open() error
	Close() error
	Attrs(id uint64) (m map[string]interface{}, err error)
	SetAttrs(id uint64, m map[string]interface{}) error
	SetBulkAttrs(m map[uint64]map[string]interface{}) error
	Blocks() ([]AttrBlock, error)
	BlockData(i uint64) (map[uint64]map[string]interface{}, error)
}

AttrStore represents an interface for handling row/column attributes. 

var NopAttrStore AttrStore

NopAttrStore represents an AttrStore that doesn't do anything.

func NewNopAttrStore added in v0.9.0 

func NewNopAttrStore(string) AttrStore

type BadRequestError added in v0.9.0 

type BadRequestError struct {
	// contains filtered or unexported fields
}

BadRequestError wraps an error value to signify that a request could not be read, decoded, or parsed such that in an HTTP scenario, http.StatusBadRequest would be returned.

type Bit 

type Bit struct {
	RowID     uint64
	ColumnID  uint64
	RowKey    string
	ColumnKey string
	Timestamp int64
}

Bit represents the intersection of a row and a column. It can be specifed by integer ids or string keys.

type BitmapCache 

type BitmapCache interface {
	Fetch(id uint64) (*Row, bool)
	Add(id uint64, b *Row)
}

BitmapCache provides an interface for caching full bitmaps.

type BitmapPair 

type BitmapPair struct {
	ID    uint64
	Count uint64
}

BitmapPair represents a id/count pair with an associated identifier.

type BitmapPairs 

type BitmapPairs []BitmapPair

BitmapPairs is a sortable list of BitmapPair objects.

func (BitmapPairs) Len 

func (p BitmapPairs) Len() int

func (BitmapPairs) Less 

func (p BitmapPairs) Less(i, j int) bool

func (BitmapPairs) Swap 

func (p BitmapPairs) Swap(i, j int)

type Broadcaster 

type Broadcaster interface {
	SendSync(pb proto.Message) error
	SendAsync(pb proto.Message) error
	SendTo(to *Node, pb proto.Message) error
}

Broadcaster is an interface for broadcasting messages. 

var NopBroadcaster Broadcaster

NopBroadcaster represents a Broadcaster that doesn't do anything.

type BufIterator 

type BufIterator struct {
	// contains filtered or unexported fields
}

BufIterator wraps an iterator to provide the ability to unread values.

func NewBufIterator 

func NewBufIterator(itr Iterator) *BufIterator

NewBufIterator returns a buffered iterator that wraps itr.

func (*BufIterator) Next 

func (itr *BufIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next pair in the row. If a value has been buffered then it is returned and the buffer is cleared.

func (*BufIterator) Peek 

func (itr *BufIterator) Peek() (rowID, columnID uint64, eof bool)

Peek reads the next value but leaves it on the buffer.

func (*BufIterator) Seek 

func (itr *BufIterator) Seek(rowID, columnID uint64)

Seek moves to the first pair equal to or greater than pseek/bseek.

func (*BufIterator) Unread 

func (itr *BufIterator) Unread()

Unread pushes previous pair on to the buffer. Panics if the buffer is already full.

type Cache 

type Cache interface {
	Add(id uint64, n uint64)
	BulkAdd(id uint64, n uint64)
	Get(id uint64) uint64
	Len() int

	// Returns a list of all IDs.
	IDs() []uint64

	// Updates the cache, if necessary.
	Invalidate()

	// Rebuilds the cache
	Recalculate()

	// Returns an ordered list of the top ranked bitmaps.
	Top() []BitmapPair

	// SetStats defines the stats client used in the cache.
	SetStats(s StatsClient)
}

Cache represents a cache of counts.

type CmdIO 

type CmdIO struct {
	Stdin  io.Reader
	Stdout io.Writer
	Stderr io.Writer
}

CmdIO holds standard unix inputs and outputs.

func NewCmdIO 

func NewCmdIO(stdin io.Reader, stdout, stderr io.Writer) *CmdIO

NewCmdIO returns a new instance of CmdIO with inputs and outputs set to the arguments.

type ColumnAttrSet 

type ColumnAttrSet struct {
	ID    uint64                 `json:"id"`
	Key   string                 `json:"key,omitempty"`
	Attrs map[string]interface{} `json:"attrs,omitempty"`
}

ColumnAttrSet represents a set of attributes for a vertical column in an index. Can have a set of attributes attached to it.

type DiagnosticsCollector added in v0.9.0 

type DiagnosticsCollector struct {
	VersionURL string

	Logger Logger
	// contains filtered or unexported fields
}

DiagnosticsCollector represents a collector/sender of diagnostics data.

func NewDiagnosticsCollector added in v0.9.0 

func NewDiagnosticsCollector(host string) *DiagnosticsCollector

NewDiagnosticsCollector returns a new DiagnosticsCollector given an addr in the format "hostname:port".

func (*DiagnosticsCollector) CheckVersion added in v0.9.0 

func (d *DiagnosticsCollector) CheckVersion() error

CheckVersion of the local build against Pilosa master.

func (*DiagnosticsCollector) EnrichWithMemoryInfo added in v0.9.0 

func (d *DiagnosticsCollector) EnrichWithMemoryInfo()

EnrichWithMemoryInfo adds memory information to the diagnostics payload.

func (*DiagnosticsCollector) EnrichWithOSInfo added in v0.9.0 

func (d *DiagnosticsCollector) EnrichWithOSInfo()

EnrichWithOSInfo adds OS information to the diagnostics payload.

func (*DiagnosticsCollector) EnrichWithSchemaProperties added in v0.9.0 

func (d *DiagnosticsCollector) EnrichWithSchemaProperties()

EnrichWithSchemaProperties adds schema info to the diagnostics payload.

func (*DiagnosticsCollector) Flush added in v0.9.0 

func (d *DiagnosticsCollector) Flush() error

Flush sends the current metrics.

func (*DiagnosticsCollector) Set added in v0.9.0 

func (d *DiagnosticsCollector) Set(name string, value interface{})

Set adds a key value metric.

func (*DiagnosticsCollector) SetVersion added in v0.9.0 

func (d *DiagnosticsCollector) SetVersion(v string)

SetVersion of locally running Pilosa Cluster to check against master.

type ExecOptions 

type ExecOptions struct {
	Remote          bool
	ExcludeRowAttrs bool
	ExcludeColumns  bool
}

ExecOptions represents an execution context for a single Execute() call.

type ExpvarStatsClient 

type ExpvarStatsClient struct {
	// contains filtered or unexported fields
}

ExpvarStatsClient writes stats out to expvars.

func NewExpvarStatsClient 

func NewExpvarStatsClient() *ExpvarStatsClient

NewExpvarStatsClient returns a new instance of ExpvarStatsClient. This client points at the root of the expvar index map.

func (*ExpvarStatsClient) Close added in v0.8.0 

func (c *ExpvarStatsClient) Close() error

Close no-op.

func (*ExpvarStatsClient) Count 

func (c *ExpvarStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs.

func (*ExpvarStatsClient) CountWithCustomTags added in v0.4.0 

func (c *ExpvarStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (*ExpvarStatsClient) Gauge 

func (c *ExpvarStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric.

func (*ExpvarStatsClient) Histogram 

func (c *ExpvarStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric. This works the same as gauge for this client.

func (*ExpvarStatsClient) Open added in v0.8.0 

func (c *ExpvarStatsClient) Open()

Open no-op.

func (*ExpvarStatsClient) Set 

func (c *ExpvarStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements.

func (*ExpvarStatsClient) SetLogger added in v0.4.0 

func (c *ExpvarStatsClient) SetLogger(logger Logger)

SetLogger has no logger.

func (*ExpvarStatsClient) Tags 

func (c *ExpvarStatsClient) Tags() []string

Tags returns a sorted list of tags on the client.

func (*ExpvarStatsClient) Timing 

func (c *ExpvarStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric.

func (*ExpvarStatsClient) WithTags 

func (c *ExpvarStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new client with additional tags appended.

type Field added in v0.5.0 

type Field struct {
	Stats StatsClient

	Logger Logger
	// contains filtered or unexported fields
}

Field represents a container for views.

func NewField added in v1.0.0 

func NewField(path, index, name string, options FieldOptions) (*Field, error)

NewField returns a new instance of field.

func (*Field) CacheSize added in v1.0.0 

func (f *Field) CacheSize() uint32

CacheSize returns the ranked field cache size.

func (*Field) CacheType 

func (f *Field) CacheType() string

CacheType returns the caching mode for the field.

func (*Field) ClearBit added in v1.0.0 

func (f *Field) ClearBit(rowID, colID uint64) (changed bool, err error)

ClearBit clears a bit within the field.

func (*Field) Close added in v1.0.0 

func (f *Field) Close() error

Close closes the field and its views.

func (*Field) CreateViewIfNotExists 

func (f *Field) CreateViewIfNotExists(name string) (*View, error)

CreateViewIfNotExists returns the named view, creating it if necessary. Additionally, a CreateViewMessage is sent to the cluster.

func (*Field) DeleteView 

func (f *Field) DeleteView(name string) error

DeleteView removes the view from the field.

func (*Field) Import added in v1.0.0 

func (f *Field) Import(rowIDs, columnIDs []uint64, timestamps []*time.Time) error

Import bulk imports data.

func (*Field) ImportValue 

func (f *Field) ImportValue(columnIDs []uint64, values []int64) error

ImportValue bulk imports range-encoded value data.

func (*Field) Index added in v1.0.0 

func (f *Field) Index() string

Index returns the index name the field was initialized with.

func (*Field) Keys 

func (f *Field) Keys() bool

Keys returns true if the field uses string keys.

func (*Field) Max added in v0.5.0 

func (f *Field) Max(filter *Row, name string) (max, count int64, err error)

Max returns the max for a field. An optional filtering row can be provided.

func (*Field) MaxShard 

func (f *Field) MaxShard() uint64

MaxShard returns the max shard in the field.

func (*Field) Min added in v0.5.0 

func (f *Field) Min(filter *Row, name string) (min, count int64, err error)

Min returns the min for a field. An optional filtering row can be provided.

func (*Field) Name added in v0.5.0 

func (f *Field) Name() string

Name returns the name the field was initialized with.

func (*Field) Open added in v1.0.0 

func (f *Field) Open() error

Open opens and initializes the field.

func (*Field) Options added in v1.0.0 

func (f *Field) Options() FieldOptions

Options returns all options for this field.

func (*Field) Path added in v1.0.0 

func (f *Field) Path() string

Path returns the path the field was initialized with.

func (*Field) Range added in v1.0.0 

func (f *Field) Range(name string, op pql.Token, predicate int64) (*Row, error)

func (*Field) RangeBetween 

func (f *Field) RangeBetween(name string, predicateMin, predicateMax int64) (*Row, error)

func (*Field) RecalculateCaches 

func (f *Field) RecalculateCaches()

RecalculateCaches recalculates caches on every view in the field.

func (*Field) Row added in v1.0.0 

func (f *Field) Row(rowID uint64) (*Row, error)

Row returns a row of the standard view.

func (*Field) RowAttrStore added in v1.0.0 

func (f *Field) RowAttrStore() AttrStore

RowAttrStore returns the attribute storage.

func (*Field) SetBit added in v1.0.0 

func (f *Field) SetBit(rowID, colID uint64, t *time.Time) (changed bool, err error)

SetBit sets a bit on a view within the field.

func (*Field) SetCacheSize added in v1.0.0 

func (f *Field) SetCacheSize(v uint32) error

SetCacheSize sets the cache size for ranked fames. Persists to meta file on update. defaults to DefaultCacheSize 50000

func (*Field) SetTimeQuantum 

func (f *Field) SetTimeQuantum(q TimeQuantum) error

SetTimeQuantum sets the time quantum for the field.

func (*Field) SetValue added in v1.0.0 

func (f *Field) SetValue(columnID uint64, value int64) (changed bool, err error)

SetValue sets a field value for a column.

func (*Field) Sum added in v1.0.0 

func (f *Field) Sum(filter *Row, name string) (sum, count int64, err error)

Sum returns the sum and count for a field. An optional filtering row can be provided.

func (*Field) TimeQuantum added in v1.0.0 

func (f *Field) TimeQuantum() TimeQuantum

TimeQuantum returns the time quantum for the field.

func (*Field) Type added in v0.5.0 

func (f *Field) Type() string

Type returns the field type.

func (*Field) Value added in v1.0.0 

func (f *Field) Value(columnID uint64) (value int64, exists bool, err error)

Value reads a field value for a column.

func (*Field) View 

func (f *Field) View(name string) *View

View returns a view in the field by name.

func (*Field) ViewPath 

func (f *Field) ViewPath(name string) string

ViewPath returns the path to a view in the field.

func (*Field) ViewRow 

func (f *Field) ViewRow(viewName string, rowID uint64) (*Row, error)

ViewRow returns a row for a view and shard. TODO: unexport this with views (it's only used in tests).

func (*Field) Views 

func (f *Field) Views() []*View

Views returns a list of all views in the field.

type FieldInfo added in v1.0.0 

type FieldInfo struct {
	Name    string       `json:"name"`
	Options FieldOptions `json:"options"`
	Views   []*ViewInfo  `json:"views,omitempty"`
}

FieldInfo represents schema information for a field.

type FieldOption added in v1.0.0 

type FieldOption func(fo *FieldOptions) error

FieldOption is a functional option type for pilosa.FieldOptions.

func OptFieldTypeInt added in v1.0.0 

func OptFieldTypeInt(min, max int64) FieldOption

func OptFieldTypeSet added in v1.0.0 

func OptFieldTypeSet(cacheType string, cacheSize uint32) FieldOption

func OptFieldTypeTime added in v1.0.0 

func OptFieldTypeTime(timeQuantum TimeQuantum) FieldOption

type FieldOptions added in v1.0.0 

type FieldOptions struct {
	Type        string      `json:"type,omitempty"`
	CacheType   string      `json:"cacheType,omitempty"`
	CacheSize   uint32      `json:"cacheSize,omitempty"`
	Min         int64       `json:"min,omitempty"`
	Max         int64       `json:"max,omitempty"`
	TimeQuantum TimeQuantum `json:"timeQuantum,omitempty"`
	Keys        bool        `json:"keys,omitempty"`
}

FieldOptions represents options to set when initializing a field.

func (*FieldOptions) Encode 

func (o *FieldOptions) Encode() *internal.FieldOptions

Encode converts o into its internal representation.

func (*FieldOptions) MarshalJSON added in v1.0.0 

func (o *FieldOptions) MarshalJSON() ([]byte, error)

type FieldValue added in v0.7.0 

type FieldValue struct {
	ColumnID uint64
	Value    int64
}

FieldValue represents the value for a column within a range-encoded field.

type Fragment 

type Fragment struct {

	// Cache for row counts.
	CacheType string // passed in by field

	CacheSize uint32

	// Number of operations performed before performing a snapshot.
	// This limits the size of fragments on the heap and flushes them to disk
	// so that they can be mmapped and heap utilization can be kept low.
	MaxOpN int

	// Logger used for out-of-band log entries.
	Logger Logger

	// Row attribute storage.
	// This is set by the parent field unless overridden for testing.
	RowAttrStore AttrStore
	// contains filtered or unexported fields
}

Fragment represents the intersection of a field and shard in an index.

func NewFragment 

func NewFragment(path, index, field, view string, shard uint64) *Fragment

NewFragment returns a new instance of Fragment.

func (*Fragment) Blocks 

func (f *Fragment) Blocks() []FragmentBlock

Blocks returns info for all blocks containing data.

func (*Fragment) Checksum 

func (f *Fragment) Checksum() []byte

Checksum returns a checksum for the entire fragment. If two fragments have the same checksum then they have the same data.

func (*Fragment) Close 

func (f *Fragment) Close() error

Close flushes the underlying storage, closes the file and unlocks it.

func (*Fragment) FlushCache 

func (f *Fragment) FlushCache() error

FlushCache writes the cache data to disk.

func (*Fragment) InvalidateChecksums 

func (f *Fragment) InvalidateChecksums()

InvalidateChecksums clears all cached block checksums.

func (*Fragment) Open 

func (f *Fragment) Open() error

Open opens the underlying storage.

func (*Fragment) ReadFrom 

func (f *Fragment) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom reads a data file from r and loads it into the fragment.

func (*Fragment) RecalculateCache 

func (f *Fragment) RecalculateCache()

RecalculateCache rebuilds the cache regardless of invalidate time delay.

func (*Fragment) Snapshot 

func (f *Fragment) Snapshot() error

Snapshot writes the storage bitmap to disk and reopens it.

func (*Fragment) WriteTo 

func (f *Fragment) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the fragment's data to w.

type FragmentBlock 

type FragmentBlock struct {
	ID       int    `json:"id"`
	Checksum []byte `json:"checksum"`
}

FragmentBlock represents info about a subsection of the rows in a block. This is used for comparing data in remote blocks for active anti-entropy.

type FragmentSyncer 

type FragmentSyncer struct {
	Fragment *Fragment

	Node    *Node
	Cluster *cluster

	Closing <-chan struct{}
}

FragmentSyncer syncs a local fragment to one on a remote host.

type GCNotifier added in v0.9.0 

type GCNotifier interface {
	Close()
	AfterGC() <-chan struct{}
}

GCNotifier represents an interface for garbage collection notificationss. 

var NopGCNotifier GCNotifier

NopGCNotifier represents a GCNotifier that doesn't do anything.

type Handler 

type Handler interface {
	Serve() error
	Close() error
}
var NopHandler Handler = nopHandler{}

type Hasher 

type Hasher interface {
	// Hashes the key into a number between [0,N).
	Hash(key uint64, n int) int
}

Hasher represents an interface to hash integers into buckets.

func NewHasher 

func NewHasher() Hasher

NewHasher returns a new instance of the default hasher.

type Holder 

type Holder struct {
	Broadcaster Broadcaster

	NewAttrStore func(string) AttrStore

	// Stats
	Stats StatsClient

	// Data directory path.
	Path string

	// The interval at which the cached row ids are persisted to disk.
	CacheFlushInterval time.Duration

	Logger Logger
	// contains filtered or unexported fields
}

Holder represents a container for indexes.

func NewHolder 

func NewHolder() *Holder

NewHolder returns a new instance of Holder.

func (*Holder) ApplySchema added in v0.9.0 

func (h *Holder) ApplySchema(schema *internal.Schema) error

ApplySchema applies an internal Schema to Holder.

func (*Holder) Close 

func (h *Holder) Close() error

Close closes all open fragments.

func (*Holder) CreateIndex 

func (h *Holder) CreateIndex(name string, opt IndexOptions) (*Index, error)

CreateIndex creates an index. An error is returned if the index already exists.

func (*Holder) CreateIndexIfNotExists 

func (h *Holder) CreateIndexIfNotExists(name string, opt IndexOptions) (*Index, error)

CreateIndexIfNotExists returns an index by name. The index is created if it does not already exist.

func (*Holder) DeleteIndex 

func (h *Holder) DeleteIndex(name string) error

DeleteIndex removes an index from the holder.

func (*Holder) EncodeMaxShards 

func (h *Holder) EncodeMaxShards() *internal.MaxShards

EncodeMaxShards creates and internal representation of max shards.

func (*Holder) EncodeSchema added in v0.9.0 

func (h *Holder) EncodeSchema() *internal.Schema

EncodeSchema creates an internal representation of schema.

func (*Holder) Field added in v1.0.0 

func (h *Holder) Field(index, name string) *Field

Field returns the field for an index and name.

func (*Holder) Fragment 

func (h *Holder) Fragment(index, field, view string, shard uint64) *Fragment

Fragment returns the fragment for an index, field & shard.

func (*Holder) HasData added in v0.9.0 

func (h *Holder) HasData() (bool, error)

HasData returns true if Holder contains at least one index. This is used to determine if the rebalancing of data is necessary when a node joins the cluster.

func (*Holder) Index 

func (h *Holder) Index(name string) *Index

Index returns the index by name.

func (*Holder) IndexPath 

func (h *Holder) IndexPath(name string) string

IndexPath returns the path where a given index is stored.

func (*Holder) Indexes 

func (h *Holder) Indexes() []*Index

Indexes returns a list of all indexes in the holder.

func (*Holder) MaxShards 

func (h *Holder) MaxShards() map[string]uint64

MaxShards returns MaxShard map for all indexes.

func (*Holder) Open 

func (h *Holder) Open() error

Open initializes the root data directory for the holder.

func (*Holder) RecalculateCaches added in v0.8.0 

func (h *Holder) RecalculateCaches()

RecalculateCaches recalculates caches on every index in the holder. This is probably not practical to call in real-world workloads, but makes writing integration tests much eaiser, since one doesn't have to wait 10 seconds after setting bits to get expected response.

func (*Holder) Schema 

func (h *Holder) Schema() []*IndexInfo

Schema returns schema information for all indexes, fields, and views.

func (*Holder) View 

func (h *Holder) View(index, field, name string) *View

View returns the view for an index, field, and name.

type HolderCleaner added in v0.9.0 

type HolderCleaner struct {
	Node *Node

	Holder  *Holder
	Cluster *cluster

	// Signals that the sync should stop.
	Closing <-chan struct{}
}

HolderCleaner removes fragments and data files that are no longer used.

func (*HolderCleaner) CleanHolder added in v0.9.0 

func (c *HolderCleaner) CleanHolder() error

CleanHolder compares the holder with the cluster state and removes any unnecessary fragments and files.

func (*HolderCleaner) IsClosing added in v0.9.0 

func (c *HolderCleaner) IsClosing() bool

IsClosing returns true if the cleaner has been marked to close.

type HolderSyncer 

type HolderSyncer struct {
	Holder *Holder

	Node    *Node
	Cluster *cluster

	// Stats
	Stats StatsClient

	// Signals that the sync should stop.
	Closing <-chan struct{}
	// contains filtered or unexported fields
}

HolderSyncer is an active anti-entropy tool that compares the local holder with a remote holder based on block checksums and resolves differences.

func (*HolderSyncer) IsClosing 

func (s *HolderSyncer) IsClosing() bool

IsClosing returns true if the syncer has been marked to close.

func (*HolderSyncer) SyncHolder 

func (s *HolderSyncer) SyncHolder() error

SyncHolder compares the holder on host with the local holder and resolves differences.

type Index 

type Index struct {
	NewAttrStore func(string) AttrStore

	Stats StatsClient

	Logger Logger
	// contains filtered or unexported fields
}

Index represents a container for fields.

func NewIndex 

func NewIndex(path, name string) (*Index, error)

NewIndex returns a new instance of Index.

func (*Index) Close 

func (i *Index) Close() error

Close closes the index and its fields.

func (*Index) ColumnAttrStore 

func (i *Index) ColumnAttrStore() AttrStore

ColumnAttrStore returns the storage for column attributes.

func (*Index) CreateField added in v1.0.0 

func (i *Index) CreateField(name string, opt FieldOptions) (*Field, error)

CreateField creates a field.

func (*Index) CreateFieldIfNotExists added in v1.0.0 

func (i *Index) CreateFieldIfNotExists(name string, opt FieldOptions) (*Field, error)

CreateFieldIfNotExists creates a field with the given options if it doesn't exist.

func (*Index) DeleteField added in v1.0.0 

func (i *Index) DeleteField(name string) error

DeleteField removes a field from the index.

func (*Index) Field added in v1.0.0 

func (i *Index) Field(name string) *Field

Field returns a field in the index by name.

func (*Index) FieldPath 

func (i *Index) FieldPath(name string) string

FieldPath returns the path to a field in the index.

func (*Index) Fields added in v1.0.0 

func (i *Index) Fields() []*Field

Fields returns a list of all fields in the index.

func (*Index) Keys added in v1.0.0 

func (i *Index) Keys() bool

Keys returns true if the index uses string keys.

func (*Index) MaxShard 

func (i *Index) MaxShard() uint64

MaxShard returns the max shard in the index according to this node.

func (*Index) Name 

func (i *Index) Name() string

Name returns name of the index.

func (*Index) Open 

func (i *Index) Open() error

Open opens and initializes the index.

func (*Index) Options added in v0.7.0 

func (i *Index) Options() IndexOptions

Options returns all options for this index.

func (*Index) Path 

func (i *Index) Path() string

Path returns the path the index was initialized with.

func (*Index) RecalculateCaches added in v0.8.0 

func (i *Index) RecalculateCaches()

RecalculateCaches recalculates caches on every field in the index.

func (*Index) SetRemoteMaxShard 

func (i *Index) SetRemoteMaxShard(newmax uint64)

SetRemoteMaxShard sets the remote max shard value received from another node.

type IndexInfo 

type IndexInfo struct {
	Name   string       `json:"name"`
	Fields []*FieldInfo `json:"fields"`
}

IndexInfo represents schema information for an index.

type IndexOptions 

type IndexOptions struct {
	Keys bool `json:"keys"`
}

IndexOptions represents options to set when initializing an index.

func (*IndexOptions) Encode 

func (i *IndexOptions) Encode() *internal.IndexMeta

Encode converts i into its internal representation.

type InternalClient added in v0.8.0 

type InternalClient interface {
	MaxShardByIndex(ctx context.Context) (map[string]uint64, error)
	Schema(ctx context.Context) ([]*IndexInfo, error)
	CreateIndex(ctx context.Context, index string, opt IndexOptions) error
	FragmentNodes(ctx context.Context, index string, shard uint64) ([]*Node, error)
	Query(ctx context.Context, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)
	QueryNode(ctx context.Context, uri *URI, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)
	Import(ctx context.Context, index, field string, shard uint64, bits []Bit) error
	ImportK(ctx context.Context, index, field string, bits []Bit) error
	EnsureIndex(ctx context.Context, name string, options IndexOptions) error
	EnsureField(ctx context.Context, indexName string, fieldName string) error
	ImportValue(ctx context.Context, index, field string, shard uint64, vals []FieldValue) error
	ExportCSV(ctx context.Context, index, field string, shard uint64, w io.Writer) error
	CreateField(ctx context.Context, index, field string) error
	FragmentBlocks(ctx context.Context, uri *URI, index, field string, shard uint64) ([]FragmentBlock, error)
	BlockData(ctx context.Context, uri *URI, index, field string, shard uint64, block int) ([]uint64, []uint64, error)
	ColumnAttrDiff(ctx context.Context, uri *URI, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
	RowAttrDiff(ctx context.Context, uri *URI, index, field string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
	SendMessage(ctx context.Context, uri *URI, pb proto.Message) error
	RetrieveShardFromURI(ctx context.Context, index, field string, shard uint64, uri URI) (io.ReadCloser, error)
}

InternalClient should be implemented by any struct that enables any transport between nodes TODO: Refactor Note from Travis: Typically an interface containing more than two or three methods is an indication that something hasn't been architected correctly. While I understand that putting the entire Client behind an interface might require this many methods, I don't want to let it go unquestioned.

type InternalQueryClient added in v1.0.0 

type InternalQueryClient interface {
	QueryNode(ctx context.Context, uri *URI, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)
}

type Iterator 

type Iterator interface {
	Seek(rowID, columnID uint64)
	Next() (rowID, columnID uint64, eof bool)
}

Iterator is an interface for looping over row/column pairs.

type LRUCache 

type LRUCache struct {
	// contains filtered or unexported fields
}

LRUCache represents a least recently used Cache implementation.

func NewLRUCache 

func NewLRUCache(maxEntries uint32) *LRUCache

NewLRUCache returns a new instance of LRUCache.

func (*LRUCache) Add 

func (c *LRUCache) Add(id, n uint64)

Add adds a count to the cache.

func (*LRUCache) BulkAdd 

func (c *LRUCache) BulkAdd(id, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*LRUCache) Get 

func (c *LRUCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*LRUCache) IDs 

func (c *LRUCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*LRUCache) Invalidate 

func (c *LRUCache) Invalidate()

Invalidate is a no-op.

func (*LRUCache) Len 

func (c *LRUCache) Len() int

Len returns the number of items in the cache.

func (*LRUCache) Recalculate 

func (c *LRUCache) Recalculate()

Recalculate is a no-op.

func (*LRUCache) SetStats added in v0.4.0 

func (c *LRUCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*LRUCache) Top 

func (c *LRUCache) Top() []BitmapPair

Top returns all counts in the cache.

type LimitIterator 

type LimitIterator struct {
	// contains filtered or unexported fields
}

LimitIterator wraps an Iterator and limits it to a max column/row pair.

func NewLimitIterator 

func NewLimitIterator(itr Iterator, maxRowID, maxColumnID uint64) *LimitIterator

NewLimitIterator returns a new LimitIterator.

func (*LimitIterator) Next 

func (itr *LimitIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair. If the underlying iterator returns a pair higher than the max then EOF is returned.

func (*LimitIterator) Seek 

func (itr *LimitIterator) Seek(rowID, columnID uint64)

Seek moves the underlying iterator to a column/row pair.

type LogEntry added in v1.0.0 

type LogEntry struct {
	Type  uint8
	Index []byte
	Frame []byte

	IDs  []uint64
	Keys [][]byte

	// Length of the entry, in bytes.
	// This is only populated after ReadFrom() or WriteTo().
	Length uint64
}

func (*LogEntry) HeaderSize 

func (e *LogEntry) HeaderSize() int64

HeaderSize returns the number of bytes required for size, type, index, frame, & pair count.

func (*LogEntry) ReadFrom added in v1.0.0 

func (e *LogEntry) ReadFrom(r io.Reader) (_ int64, err error)

ReadFrom deserializes a LogEntry from r. r must be a ByteReader.

func (*LogEntry) WriteTo added in v1.0.0 

func (e *LogEntry) WriteTo(w io.Writer) (_ int64, err error)

WriteTo serializes a LogEntry to w.

type Logger added in v0.9.0 

type Logger interface {
	Printf(format string, v ...interface{})
	Debugf(format string, v ...interface{})
}

Logger represents an interface for a shared logger. 

var NopLogger Logger

NopLogger represents a Logger that doesn't do anything.

type MemberServer 

type MemberServer interface {
	ReceiveMessage(proto.Message) error
	LocalStatus() (proto.Message, error)
	HandleRemoteStatus(proto.Message) error
	Node() *Node
}

type MultiStatsClient 

type MultiStatsClient []StatsClient

MultiStatsClient joins multiple stats clients together.

func (MultiStatsClient) Close added in v0.8.0 

func (a MultiStatsClient) Close() error

Close shuts down the stats clients.

func (MultiStatsClient) Count 

func (a MultiStatsClient) Count(name string, value int64, rate float64)

Count tracks the number of times something occurs per second on all clients.

func (MultiStatsClient) CountWithCustomTags added in v0.4.0 

func (a MultiStatsClient) CountWithCustomTags(name string, value int64, rate float64, tags []string)

CountWithCustomTags Tracks the number of times something occurs per second with custom tags

func (MultiStatsClient) Gauge 

func (a MultiStatsClient) Gauge(name string, value float64, rate float64)

Gauge sets the value of a metric on all clients.

func (MultiStatsClient) Histogram 

func (a MultiStatsClient) Histogram(name string, value float64, rate float64)

Histogram tracks statistical distribution of a metric on all clients.

func (MultiStatsClient) Open added in v0.8.0 

func (a MultiStatsClient) Open()

Open starts the stat service.

func (MultiStatsClient) Set 

func (a MultiStatsClient) Set(name string, value string, rate float64)

Set tracks number of unique elements on all clients.

func (MultiStatsClient) SetLogger added in v0.4.0 

func (a MultiStatsClient) SetLogger(logger Logger)

SetLogger Sets the StatsD logger output type.

func (MultiStatsClient) Tags 

func (a MultiStatsClient) Tags() []string

Tags returns tags from the first client.

func (MultiStatsClient) Timing 

func (a MultiStatsClient) Timing(name string, value time.Duration, rate float64)

Timing tracks timing information for a metric on all clients.

func (MultiStatsClient) WithTags 

func (a MultiStatsClient) WithTags(tags ...string) StatsClient

WithTags returns a new set of clients with the additional tags.

type Node 

type Node struct {
	ID            string `json:"id"`
	URI           URI    `json:"uri"`
	IsCoordinator bool   `json:"isCoordinator"`
}

Node represents a node in the cluster.

func DecodeNode added in v0.9.0 

func DecodeNode(node *internal.Node) *Node

DecodeNode converts a proto message into a Node.

func DecodeNodes added in v0.9.0 

func DecodeNodes(a []*internal.Node) []*Node

DecodeNodes converts a proto message into a slice of Nodes.

func (Node) String added in v0.9.0 

func (n Node) String() string

type NodeEventType added in v0.9.0 

type NodeEventType int

NodeEventType are the types of node events. 

const (
	NodeJoin NodeEventType = iota
	NodeLeave
	NodeUpdate
)

type NodeIDs added in v0.9.0 

type NodeIDs []string

func (NodeIDs) ContainsID added in v0.9.0 

func (n NodeIDs) ContainsID(id string) bool

ContainsID returns true if idi matches one of the nodesets's IDs.

func (NodeIDs) Len added in v0.9.0 

func (n NodeIDs) Len() int

func (NodeIDs) Less added in v0.9.0 

func (n NodeIDs) Less(i, j int) bool

func (NodeIDs) Swap added in v0.9.0 

func (n NodeIDs) Swap(i, j int)

type Nodes 

type Nodes []*Node

Nodes represents a list of nodes.

func (Nodes) Clone 

func (a Nodes) Clone() []*Node

Clone returns a shallow copy of nodes.

func (Nodes) Contains 

func (a Nodes) Contains(n *Node) bool

Contains returns true if a node exists in the list.

func (Nodes) ContainsID added in v0.9.0 

func (a Nodes) ContainsID(id string) bool

ContainsID returns true if host matches one of the node's id.

func (Nodes) Filter 

func (a Nodes) Filter(n *Node) []*Node

Filter returns a new list of nodes with node removed.

func (Nodes) FilterID added in v0.9.0 

func (a Nodes) FilterID(id string) []*Node

FilterID returns a new list of nodes with ID removed.

func (Nodes) FilterURI added in v0.9.0 

func (a Nodes) FilterURI(uri URI) []*Node

FilterURI returns a new list of nodes with URI removed.

func (Nodes) IDs added in v0.9.0 

func (a Nodes) IDs() []string

IDs returns a list of all node IDs.

func (Nodes) URIs added in v0.9.0 

func (a Nodes) URIs() []URI

URIs returns a list of all uris.

type NopCache added in v0.5.0 

type NopCache struct {
	// contains filtered or unexported fields
}

NopCache represents a no-op Cache implementation.

func NewNopCache added in v0.5.0 

func NewNopCache() *NopCache

NewNopCache returns a new instance of NopCache.

func (*NopCache) Add added in v0.5.0 

func (c *NopCache) Add(id uint64, n uint64)

func (*NopCache) BulkAdd added in v0.5.0 

func (c *NopCache) BulkAdd(id uint64, n uint64)

func (*NopCache) Get added in v0.5.0 

func (c *NopCache) Get(id uint64) uint64

func (*NopCache) IDs added in v0.5.0 

func (c *NopCache) IDs() []uint64

func (*NopCache) Invalidate added in v0.5.0 

func (c *NopCache) Invalidate()

func (*NopCache) Len added in v0.5.0 

func (c *NopCache) Len() int

func (*NopCache) Recalculate added in v0.5.0 

func (c *NopCache) Recalculate()

func (*NopCache) SetStats added in v0.5.0 

func (c *NopCache) SetStats(s StatsClient)

func (*NopCache) Top added in v0.5.0 

func (c *NopCache) Top() []BitmapPair

type NopInternalClient 

type NopInternalClient struct{}

func NewNopInternalClient 

func NewNopInternalClient() *NopInternalClient

func (*NopInternalClient) BlockData 

func (n *NopInternalClient) BlockData(ctx context.Context, uri *URI, index, field string, shard uint64, block int) ([]uint64, []uint64, error)

func (*NopInternalClient) ColumnAttrDiff 

func (n *NopInternalClient) ColumnAttrDiff(ctx context.Context, uri *URI, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

func (*NopInternalClient) CreateField 

func (n *NopInternalClient) CreateField(ctx context.Context, index, field string) error

func (*NopInternalClient) CreateIndex 

func (n *NopInternalClient) CreateIndex(ctx context.Context, index string, opt IndexOptions) error

func (*NopInternalClient) EnsureField 

func (n *NopInternalClient) EnsureField(ctx context.Context, indexName string, fieldName string) error

func (*NopInternalClient) EnsureIndex 

func (n *NopInternalClient) EnsureIndex(ctx context.Context, name string, options IndexOptions) error

func (*NopInternalClient) ExportCSV 

func (n *NopInternalClient) ExportCSV(ctx context.Context, index, field string, shard uint64, w io.Writer) error

func (*NopInternalClient) FragmentBlocks 

func (n *NopInternalClient) FragmentBlocks(ctx context.Context, uri *URI, index, field string, shard uint64) ([]FragmentBlock, error)

func (*NopInternalClient) FragmentNodes 

func (n *NopInternalClient) FragmentNodes(ctx context.Context, index string, shard uint64) ([]*Node, error)

func (*NopInternalClient) Import 

func (n *NopInternalClient) Import(ctx context.Context, index, field string, shard uint64, bits []Bit) error

func (*NopInternalClient) ImportK 

func (n *NopInternalClient) ImportK(ctx context.Context, index, field string, bits []Bit) error

func (*NopInternalClient) ImportValue 

func (n *NopInternalClient) ImportValue(ctx context.Context, index, field string, shard uint64, vals []FieldValue) error

func (*NopInternalClient) MaxShardByIndex 

func (n *NopInternalClient) MaxShardByIndex(ctx context.Context) (map[string]uint64, error)

func (*NopInternalClient) Query 

func (n *NopInternalClient) Query(ctx context.Context, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)

func (*NopInternalClient) QueryNode 

func (n *NopInternalClient) QueryNode(ctx context.Context, uri *URI, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)

func (*NopInternalClient) RetrieveShardFromURI 

func (n *NopInternalClient) RetrieveShardFromURI(ctx context.Context, index, field string, shard uint64, uri URI) (io.ReadCloser, error)

func (*NopInternalClient) RowAttrDiff 

func (n *NopInternalClient) RowAttrDiff(ctx context.Context, uri *URI, index, field string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)

func (*NopInternalClient) Schema 

func (n *NopInternalClient) Schema(ctx context.Context) ([]*IndexInfo, error)

func (*NopInternalClient) SendMessage 

func (n *NopInternalClient) SendMessage(ctx context.Context, uri *URI, pb proto.Message) error

type NopInternalQueryClient 

type NopInternalQueryClient struct{}

func NewNopInternalQueryClient 

func NewNopInternalQueryClient() *NopInternalQueryClient

func (*NopInternalQueryClient) QueryNode 

func (n *NopInternalQueryClient) QueryNode(ctx context.Context, uri *URI, index string, queryRequest *internal.QueryRequest) (*internal.QueryResponse, error)

type NopSystemInfo added in v0.9.0 

type NopSystemInfo struct {
}

NopSystemInfo is a no-op implementation of SystemInfo.

func NewNopSystemInfo added in v0.9.0 

func NewNopSystemInfo() *NopSystemInfo

NewNopSystemInfo creates a no-op implementation of SystemInfo.

func (*NopSystemInfo) Family added in v0.9.0 

func (n *NopSystemInfo) Family() (string, error)

Family is a no-op implementation of SystemInfo.Family.

func (*NopSystemInfo) KernelVersion added in v0.9.0 

func (n *NopSystemInfo) KernelVersion() (string, error)

KernelVersion is a no-op implementation of SystemInfo.KernelVersion.

func (*NopSystemInfo) MemFree added in v0.9.0 

func (n *NopSystemInfo) MemFree() (uint64, error)

MemFree is a no-op implementation of SystemInfo.MemFree.

func (*NopSystemInfo) MemTotal added in v0.9.0 

func (n *NopSystemInfo) MemTotal() (uint64, error)

MemTotal is a no-op implementation of SystemInfo.MemTotal.

func (*NopSystemInfo) MemUsed added in v0.9.0 

func (n *NopSystemInfo) MemUsed() (uint64, error)

MemUsed is a no-op implementation of SystemInfo.MemUsed.

func (*NopSystemInfo) OSVersion added in v0.9.0 

func (n *NopSystemInfo) OSVersion() (string, error)

OSVersion is a no-op implementation of SystemInfo.OSVersion.

func (*NopSystemInfo) Platform added in v0.9.0 

func (n *NopSystemInfo) Platform() (string, error)

Platform is a no-op implementation of SystemInfo.Platform.

func (*NopSystemInfo) Uptime added in v0.9.0 

func (n *NopSystemInfo) Uptime() (uint64, error)

Uptime is a no-op implementation of SystemInfo.Uptime.

type Pair 

type Pair struct {
	ID    uint64 `json:"id"`
	Key   string `json:"key,omitempty"`
	Count uint64 `json:"count"`
}

Pair holds an id/count pair.

type PairHeap 

type PairHeap struct {
	Pairs
}

PairHeap is a heap implementation over a group of Pairs.

func (PairHeap) Less 

func (p PairHeap) Less(i, j int) bool

Less implemets the Sort interface. reports whether the element with index i should sort before the element with index j.

type Pairs 

type Pairs []Pair

Pairs is a sortable slice of Pair objects.

func (Pairs) Add 

func (p Pairs) Add(other []Pair) []Pair

Add merges other into p and returns a new slice.

func (Pairs) Keys 

func (p Pairs) Keys() []uint64

Keys returns a slice of all keys in p.

func (Pairs) Len 

func (p Pairs) Len() int

func (Pairs) Less 

func (p Pairs) Less(i, j int) bool

func (*Pairs) Pop 

func (p *Pairs) Pop() interface{}

Pop removes the minimum element from the Pair slice.

func (*Pairs) Push 

func (p *Pairs) Push(x interface{})

Push appends the element onto the Pair slice.

func (Pairs) String 

func (p Pairs) String() string

func (Pairs) Swap 

func (p Pairs) Swap(i, j int)

type QueryRequest 

type QueryRequest struct {
	// Index to execute query against.
	Index string

	// The query string to parse and execute.
	Query string

	// The shards to include in the query execution.
	// If empty, all shards are included.
	Shards []uint64

	// Return column attributes, if true.
	ColumnAttrs bool

	// Do not return row attributes, if true.
	ExcludeRowAttrs bool

	// Do not return columns, if true.
	ExcludeColumns bool

	// If true, indicates that query is part of a larger distributed query.
	// If false, this request is on the originating node.
	Remote bool
}

QueryRequest represent a request to process a query.

type QueryResponse 

type QueryResponse struct {
	// Result for each top-level query call.
	// Can be a Bitmap, Pairs, or uint64.
	Results []interface{}

	// Set of column attribute objects matching IDs returned in Result.
	ColumnAttrSets []*ColumnAttrSet

	// Error during parsing or execution.
	Err error
}

QueryResponse represent a response from a processed query.

func (*QueryResponse) MarshalJSON 

func (resp *QueryResponse) MarshalJSON() ([]byte, error)

MarshalJSON marshals QueryResponse into a JSON-encoded byte slice

type RankCache 

type RankCache struct {
	// contains filtered or unexported fields
}

RankCache represents a cache with sorted entries.

func NewRankCache 

func NewRankCache(maxEntries uint32) *RankCache

NewRankCache returns a new instance of RankCache.

func (*RankCache) Add 

func (c *RankCache) Add(id uint64, n uint64)

Add adds a count to the cache.

func (*RankCache) BulkAdd 

func (c *RankCache) BulkAdd(id uint64, n uint64)

BulkAdd adds a count to the cache unsorted. You should Invalidate after completion.

func (*RankCache) Get 

func (c *RankCache) Get(id uint64) uint64

Get returns a count for a given id.

func (*RankCache) IDs 

func (c *RankCache) IDs() []uint64

IDs returns a list of all IDs in the cache.

func (*RankCache) Invalidate 

func (c *RankCache) Invalidate()

Invalidate recalculates the entries by rank.

func (*RankCache) Len 

func (c *RankCache) Len() int

Len returns the number of items in the cache.

func (*RankCache) ReadFrom 

func (c *RankCache) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom read from r into the cache.

func (*RankCache) Recalculate 

func (c *RankCache) Recalculate()

Recalculate rebuilds the cache.

func (*RankCache) SetStats added in v0.4.0 

func (c *RankCache) SetStats(s StatsClient)

SetStats defines the stats client used in the cache.

func (*RankCache) Top 

func (c *RankCache) Top() []BitmapPair

Top returns an ordered list of pairs.

func (*RankCache) WriteTo 

func (c *RankCache) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the cache to w.

type RoaringIterator 

type RoaringIterator struct {
	// contains filtered or unexported fields
}

RoaringIterator converts a roaring.Iterator to output column/row pairs.

func NewRoaringIterator 

func NewRoaringIterator(itr *roaring.Iterator) *RoaringIterator

NewRoaringIterator returns a new iterator wrapping itr.

func (*RoaringIterator) Next 

func (itr *RoaringIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next column/row ID pair.

func (*RoaringIterator) Seek 

func (itr *RoaringIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a pair matching bseek/pseek. If the pair is not found then it moves to the next pair.

type Row added in v1.0.0 

type Row struct {

	// String keys translated to/from segment columns.
	Keys []string

	// Attributes associated with the row.
	Attrs map[string]interface{}
	// contains filtered or unexported fields
}

Row is a set of integers (the associated columns), and attributes which are arbitrary key/value pairs storing metadata about what the row represents.

func DecodeRow 

func DecodeRow(pr *internal.Row) *Row

DecodeRow converts r from its internal representation.

func NewRow added in v1.0.0 

func NewRow(columns ...uint64) *Row

NewRow returns a new instance of Row.

func Union 

func Union(rows []*Row) *Row

Union performs a union on a slice of rows.

func (*Row) ClearBit 

func (r *Row) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th column of the row.

func (*Row) Columns added in v1.0.0 

func (r *Row) Columns() []uint64

Columns returns the columns in r as a slice of ints.

func (*Row) Count added in v1.0.0 

func (r *Row) Count() uint64

Count returns the number of columns in the row.

func (*Row) DecrementCount 

func (r *Row) DecrementCount(i uint64)

DecrementCount decrements the row cached counter.

func (*Row) Difference added in v1.0.0 

func (r *Row) Difference(other *Row) *Row

Difference returns the diff of r and other.

func (*Row) IncrementCount 

func (r *Row) IncrementCount(i uint64)

IncrementCount increments the row cached counter, note this is an optimization that assumes that the caller is aware the size increased.

func (*Row) Intersect added in v1.0.2 

func (r *Row) Intersect(other *Row) *Row

Intersect returns the itersection of r and other.

func (*Row) IntersectionCount 

func (r *Row) IntersectionCount(other *Row) uint64

IntersectionCount returns the number of intersections between r and other.

func (*Row) InvalidateCount 

func (r *Row) InvalidateCount()

InvalidateCount updates the cached count in the row.

func (*Row) MarshalJSON added in v1.0.0 

func (r *Row) MarshalJSON() ([]byte, error)

MarshalJSON returns a JSON-encoded byte slice of r.

func (*Row) Merge added in v1.0.0 

func (r *Row) Merge(other *Row)

Merge merges data from other into r.

func (*Row) Segments added in v1.0.0 

func (r *Row) Segments() []RowSegment

Segments returns a list of all segments in the row.

func (*Row) SetBit added in v1.0.0 

func (r *Row) SetBit(i uint64) (changed bool)

SetBit sets the i-th column of the row.

func (*Row) Union added in v1.0.0 

func (r *Row) Union(other *Row) *Row

Union returns the bitwise union of r and other.

func (*Row) Xor added in v1.0.0 

func (r *Row) Xor(other *Row) *Row

Xor returns the xor of r and other.

type RowSegment 

type RowSegment struct {
	// contains filtered or unexported fields
}

RowSegment holds a subset of a row. This could point to a mmapped roaring bitmap or an in-memory bitmap. The width of the segment will always match the shard width.

func (*RowSegment) ClearBit 

func (s *RowSegment) ClearBit(i uint64) (changed bool)

ClearBit clears the i-th column of the row.

func (*RowSegment) Columns 

func (s *RowSegment) Columns() []uint64

Columns returns a list of all columns set in the segment.

func (*RowSegment) Count 

func (s *RowSegment) Count() uint64

Count returns the number of set columns in the row.

func (*RowSegment) Difference 

func (s *RowSegment) Difference(other *RowSegment) *RowSegment

Difference returns the diff of s and other.

func (*RowSegment) Intersect 

func (s *RowSegment) Intersect(other *RowSegment) *RowSegment

Intersect returns the itersection of s and other.

func (*RowSegment) IntersectionCount 

func (s *RowSegment) IntersectionCount(other *RowSegment) uint64

IntersectionCount returns the number of intersections between s and other.

func (*RowSegment) InvalidateCount 

func (s *RowSegment) InvalidateCount()

InvalidateCount updates the cached count in the row.

func (*RowSegment) Merge 

func (s *RowSegment) Merge(other *RowSegment)

Merge adds chunks from other to s. Chunks in s are overwritten if they exist in other.

func (*RowSegment) SetBit 

func (s *RowSegment) SetBit(i uint64) (changed bool)

SetBit sets the i-th column of the row.

func (*RowSegment) Union 

func (s *RowSegment) Union(other *RowSegment) *RowSegment

Union returns the bitwise union of s and other.

func (*RowSegment) Xor 

func (s *RowSegment) Xor(other *RowSegment) *RowSegment

Xor returns the xor of s and other.

type Server 

type Server struct {
	URI URI
	// contains filtered or unexported fields
}

Server represents a holder wrapped by a running HTTP server.

func NewServer 

func NewServer(opts ...ServerOption) (*Server, error)

NewServer returns a new instance of Server.

func (*Server) Close 

func (s *Server) Close() error

Close closes the server and waits for it to shutdown.

func (*Server) HandleRemoteStatus 

func (s *Server) HandleRemoteStatus(pb proto.Message) error

HandleRemoteStatus receives incoming NodeStatus from remote nodes.

func (*Server) Holder 

func (s *Server) Holder() *Holder

TODO: have this return an interface for Holder instead of concrete object?

func (*Server) LocalStatus 

func (s *Server) LocalStatus() (proto.Message, error)

Server implements StatusHandler. LocalStatus is used to periodically sync information between nodes. Under normal conditions, nodes should remain in sync through Broadcast messages. For cases where a node fails to receive a Broadcast message, or when a new (empty) node needs to get in sync with the rest of the cluster, two things are shared via gossip: - MaxShard by Index - Schema In a gossip implementation, memberlist.Delegate.LocalState() uses this.

func (*Server) Node 

func (s *Server) Node() *Node

Node returns the pilosa.Node object. It is used by membership protocols to get this node's name(ID), location(URI), and coordinator status.

func (*Server) Open 

func (s *Server) Open() error

Open opens and initializes the server.

func (*Server) ReceiveMessage 

func (s *Server) ReceiveMessage(pb proto.Message) error

ReceiveMessage represents an implementation of BroadcastHandler.

func (*Server) SendAsync added in v0.9.0 

func (s *Server) SendAsync(pb proto.Message) error

SendAsync represents an implementation of Broadcaster.

func (*Server) SendSync added in v0.9.0 

func (s *Server) SendSync(pb proto.Message) error

SendSync represents an implementation of Broadcaster.

func (*Server) SendTo added in v0.9.0 

func (s *Server) SendTo(to *Node, pb proto.Message) error

SendTo represents an implementation of Broadcaster.

func (*Server) SyncData added in v1.0.0 

func (s *Server) SyncData() error

SyncData manually invokes the anti entropy process which makes sure that this node has the data from all replicas across the cluster.

type ServerOption added in v0.9.0 

type ServerOption func(s *Server) error

ServerOption is a functional option type for pilosa.Server

func OptServerAntiEntropyInterval added in v0.9.0 

func OptServerAntiEntropyInterval(interval time.Duration) ServerOption

func OptServerAttrStoreFunc added in v0.9.0 

func OptServerAttrStoreFunc(af func(string) AttrStore) ServerOption

func OptServerClusterDisabled added in v1.0.0 

func OptServerClusterDisabled(disabled bool, hosts []string) ServerOption

OptClusterDisabled tells the server whether to use a static cluster with the defined hosts. Mostly used for testing.

func OptServerClusterHasher added in v1.0.0 

func OptServerClusterHasher(h Hasher) ServerOption

func OptServerDataDir added in v0.9.0 

func OptServerDataDir(dir string) ServerOption

func OptServerDiagnosticsInterval added in v0.9.0 

func OptServerDiagnosticsInterval(dur time.Duration) ServerOption

func OptServerGCNotifier added in v0.9.0 

func OptServerGCNotifier(gcn GCNotifier) ServerOption

func OptServerInternalClient added in v1.0.0 

func OptServerInternalClient(c InternalClient) ServerOption

func OptServerIsCoordinator added in v1.0.0 

func OptServerIsCoordinator(is bool) ServerOption

func OptServerLogger added in v0.9.0 

func OptServerLogger(l Logger) ServerOption

func OptServerLongQueryTime added in v0.9.0 

func OptServerLongQueryTime(dur time.Duration) ServerOption

func OptServerMaxWritesPerRequest added in v0.9.0 

func OptServerMaxWritesPerRequest(n int) ServerOption

func OptServerMetricInterval added in v0.9.0 

func OptServerMetricInterval(dur time.Duration) ServerOption

func OptServerNodeID added in v1.0.0 

func OptServerNodeID(nodeID string) ServerOption

func OptServerPrimaryTranslateStore added in v1.0.0 

func OptServerPrimaryTranslateStore(store TranslateStore) ServerOption

func OptServerReplicaN added in v0.9.0 

func OptServerReplicaN(n int) ServerOption

func OptServerStatsClient added in v0.9.0 

func OptServerStatsClient(sc StatsClient) ServerOption

func OptServerSystemInfo added in v0.9.0 

func OptServerSystemInfo(si SystemInfo) ServerOption

func OptServerURI added in v0.9.0 

func OptServerURI(uri *URI) ServerOption

type SimpleCache 

type SimpleCache struct {
	// contains filtered or unexported fields
}

SimpleCache implements BitmapCache it is meant to be a short-lived cache for cases where writes are continuing to access the same row within a short time frame (i.e. good for write-heavy loads) A read-heavy use case would cause the cache to get bigger, potentially causing the node to run out of memory.

func (*SimpleCache) Add 

func (s *SimpleCache) Add(id uint64, b *Row)

Add adds the bitmap to the cache, keyed on the id.

func (*SimpleCache) Fetch 

func (s *SimpleCache) Fetch(id uint64) (*Row, bool)

Fetch retrieves the bitmap at the id in the cache.

type SliceIterator 

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator iterates over a pair of row/column ID slices.

func NewSliceIterator 

func NewSliceIterator(rowIDs, columnIDs []uint64) *SliceIterator

NewSliceIterator returns an iterator to iterate over a set of row/column ID pairs. Both slices MUST have an equal length. Otherwise the function will panic.

func (*SliceIterator) Next 

func (itr *SliceIterator) Next() (rowID, columnID uint64, eof bool)

Next returns the next row/column ID pair.

func (*SliceIterator) Seek 

func (itr *SliceIterator) Seek(bseek, pseek uint64)

Seek moves the cursor to a given pair. If the pair is not found, the iterator seeks to the next pair.

type StandardLogger added in v0.9.0 

type StandardLogger struct {
	// contains filtered or unexported fields
}

StandardLogger is a basic implementation of pilosa.Logger based on log.Logger.

func NewStandardLogger added in v0.9.0 

func NewStandardLogger(w io.Writer) *StandardLogger

func (*StandardLogger) Debugf added in v0.9.0 

func (s *StandardLogger) Debugf(format string, v ...interface{})

func (*StandardLogger) Logger added in v0.9.0 

func (s *StandardLogger) Logger() *log.Logger

func (*StandardLogger) Printf added in v0.9.0 

func (s *StandardLogger) Printf(format string, v ...interface{})

type StatsClient 

type StatsClient interface {
	// Returns a sorted list of tags on the client.
	Tags() []string

	// Returns a new client with additional tags appended.
	WithTags(tags ...string) StatsClient

	// Tracks the number of times something occurs per second.
	Count(name string, value int64, rate float64)

	// Tracks the number of times something occurs per second with custom tags
	CountWithCustomTags(name string, value int64, rate float64, tags []string)

	// Sets the value of a metric.
	Gauge(name string, value float64, rate float64)

	// Tracks statistical distribution of a metric.
	Histogram(name string, value float64, rate float64)

	// Tracks number of unique elements.
	Set(name string, value string, rate float64)

	// Tracks timing information for a metric.
	Timing(name string, value time.Duration, rate float64)

	// SetLogger Set the logger output type
	SetLogger(logger Logger)

	// Starts the service
	Open()

	// Closes the client
	Close() error
}

StatsClient represents a client to a stats server. 

var NopStatsClient StatsClient

NopStatsClient represents a client that doesn't do anything.

type SystemInfo added in v0.9.0 

type SystemInfo interface {
	Uptime() (uint64, error)
	Platform() (string, error)
	Family() (string, error)
	OSVersion() (string, error)
	KernelVersion() (string, error)
	MemFree() (uint64, error)
	MemTotal() (uint64, error)
	MemUsed() (uint64, error)
}

SystemInfo collects information about the host OS.

type TimeQuantum 

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func ParseTimeQuantum 

func ParseTimeQuantum(v string) (TimeQuantum, error)

ParseTimeQuantum parses v into a time quantum.

func (TimeQuantum) HasDay 

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour 

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth 

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear 

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (*TimeQuantum) Set added in v0.7.0 

func (q *TimeQuantum) Set(value string) error

Set sets the time quantum value.

func (TimeQuantum) String added in v0.7.0 

func (q TimeQuantum) String() string

func (TimeQuantum) Type added in v0.7.0 

func (q TimeQuantum) Type() string

Type returns the type of a time quantum value.

func (TimeQuantum) Valid 

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type TopOptions 

type TopOptions struct {
	// Number of rows to return.
	N int

	// Bitmap to intersect with.
	Src *Row

	// Specific rows to filter against.
	RowIDs       []uint64
	MinThreshold uint64

	// Filter field name & values.
	FilterName        string
	FilterValues      []interface{}
	TanimotoThreshold uint64
}

TopOptions represents options passed into the Top() function.

type Topology added in v0.9.0 

type Topology struct {
	NodeIDs []string

	ClusterID string
	// contains filtered or unexported fields
}

Topology represents the list of hosts in the cluster.

func NewTopology added in v0.9.0 

func NewTopology() *Topology

func (*Topology) AddID added in v0.9.0 

func (t *Topology) AddID(nodeID string) bool

AddID adds the node ID to the topology and returns true if added.

func (*Topology) ContainsID added in v0.9.0 

func (t *Topology) ContainsID(id string) bool

ContainsID returns true if id matches one of the topology's IDs.

func (*Topology) Encode added in v0.9.0 

func (t *Topology) Encode() *internal.Topology

Encode converts t into its internal representation.

func (*Topology) RemoveID added in v0.9.0 

func (t *Topology) RemoveID(nodeID string) bool

RemoveID removes the node ID from the topology and returns true if removed.

type TranslateFile added in v1.0.0 

type TranslateFile struct {
	Path    string
	MapSize int

	// If non-nil, data is streamed from a primary and this is a read-only store.
	PrimaryTranslateStore TranslateStore

	// Delay after attempting to connect to a primary that the store will retry.
	ReplicationRetryInterval time.Duration
	// contains filtered or unexported fields
}

TranslateFile is an on-disk storage engine for translating string-to-uint64 values.

func NewTranslateFile added in v1.0.0 

func NewTranslateFile() *TranslateFile

NewTranslateFile returns a new instance of TranslateFile.

func (*TranslateFile) Close added in v1.0.0 

func (s *TranslateFile) Close() (err error)

func (*TranslateFile) Closing added in v1.0.0 

func (s *TranslateFile) Closing() <-chan struct{}

Closing returns a channel that is closed when the store is closed.

func (*TranslateFile) IsReadOnly 

func (s *TranslateFile) IsReadOnly() bool

IsReadOnly returns true if this store is being replicated from a primary store.

func (*TranslateFile) Open added in v1.0.0 

func (s *TranslateFile) Open() (err error)

func (*TranslateFile) Reader added in v1.0.0 

func (s *TranslateFile) Reader(ctx context.Context, offset int64) (io.ReadCloser, error)

Reader returns a reader that streams the underlying data file.

func (*TranslateFile) Size 

func (s *TranslateFile) Size() int64

Size returns the number of bytes in use in the data file.

func (*TranslateFile) TranslateColumnToString added in v1.0.0 

func (s *TranslateFile) TranslateColumnToString(index string, value uint64) (string, error)

TranslateColumnToString converts a uint64 id to its associated string value. If the id is not associated with a string value then a blank string is returned.

func (*TranslateFile) TranslateColumnsToUint64 added in v1.0.0 

func (s *TranslateFile) TranslateColumnsToUint64(index string, values []string) ([]uint64, error)

TranslateColumnsToUint64 converts values to a uint64 id. If value does not have an associated id then one is created.

func (*TranslateFile) TranslateRowToString added in v1.0.0 

func (s *TranslateFile) TranslateRowToString(index, frame string, id uint64) (string, error)

func (*TranslateFile) TranslateRowsToUint64 added in v1.0.0 

func (s *TranslateFile) TranslateRowsToUint64(index, frame string, values []string) ([]uint64, error)

func (*TranslateFile) WriteNotify added in v1.0.0 

func (s *TranslateFile) WriteNotify() <-chan struct{}

WriteNotify returns a channel that is closed when a new entry is written.

type TranslateFileReader 

type TranslateFileReader struct {
	// contains filtered or unexported fields
}

TranslateFileReader implements a reader that continuously streams data from a store.

func NewTranslateFileReader 

func NewTranslateFileReader(ctx context.Context, store *TranslateFile, offset int64) *TranslateFileReader

NewTranslateFileReader returns a new instance of TranslateFileReader.

func (*TranslateFileReader) Close 

func (r *TranslateFileReader) Close() error

Close closes the underlying file reader.

func (*TranslateFileReader) Open 

func (r *TranslateFileReader) Open() (err error)

Open initializes the reader.

func (*TranslateFileReader) Read 

func (r *TranslateFileReader) Read(p []byte) (n int, err error)

Read reads the next section of the available data to p. This should always read from the start of an entry and read n bytes to the end of another entry.

type TranslateStore added in v1.0.0 

type TranslateStore interface {
	TranslateColumnsToUint64(index string, values []string) ([]uint64, error)
	TranslateColumnToString(index string, values uint64) (string, error)

	TranslateRowsToUint64(index, frame string, values []string) ([]uint64, error)
	TranslateRowToString(index, frame string, values uint64) (string, error)

	// Returns a reader from the given offset of the raw data file.
	// The returned reader must be closed by the caller when done.
	Reader(ctx context.Context, off int64) (io.ReadCloser, error)
}

TranslateStore is the storage for translation string-to-uint64 values.

type URI added in v0.8.0 

type URI struct {
	// contains filtered or unexported fields
}

URI represents a Pilosa URI. A Pilosa URI consists of three parts: 1) Scheme: Protocol of the URI. Default: http. 2) Host: Hostname or IP URI. Default: localhost. IPv6 addresses should be written in brackets, e.g., `[fd42:4201:f86b:7e09:216:3eff:fefa:ed80]`. 3) Port: Port of the URI. Default: 10101.

All parts of the URI are optional. The following are equivalent: 

http://localhost:10101
http://localhost
http://:10101
localhost:10101
localhost
:10101

func AddressWithDefaults added in v0.6.0 

func AddressWithDefaults(addr string) (*URI, error)

AddressWithDefaults converts addr into a valid address, using defaults when necessary.

func DecodeURI added in v0.9.0 

func DecodeURI(i *internal.URI) URI

func DefaultURI added in v0.8.0 

func DefaultURI() *URI

DefaultURI creates and returns the default URI.

func NewURIFromAddress added in v0.8.0 

func NewURIFromAddress(address string) (*URI, error)

NewURIFromAddress parses the passed address and returns a URI.

func NewURIFromHostPort added in v0.8.0 

func NewURIFromHostPort(host string, port uint16) (*URI, error)

NewURIFromHostPort returns a URI with specified host and port.

func (URI) Encode added in v0.9.0 

func (u URI) Encode() *internal.URI

Encode converts o into its internal representation.

func (URI) Equals added in v0.8.0 

func (u URI) Equals(other *URI) bool

Equals returns true if the checked URI is equivalent to this URI.

func (*URI) Host added in v0.8.0 

func (u *URI) Host() string

Host returns the host of this URI.

func (*URI) HostPort added in v0.8.0 

func (u *URI) HostPort() string

HostPort returns `Host:Port`

func (*URI) MarshalJSON added in v0.9.0 

func (u *URI) MarshalJSON() ([]byte, error)

MarshalJSON marshals URI into a JSON-encoded byte slice.

func (*URI) Normalize added in v0.8.0 

func (u *URI) Normalize() string

Normalize returns the address in a form usable by a HTTP client.

func (*URI) Path added in v0.8.0 

func (u *URI) Path(path string) string

Path returns URI with path

func (*URI) Port added in v0.8.0 

func (u *URI) Port() uint16

Port returns the port of this URI.

func (*URI) Scheme added in v0.8.0 

func (u *URI) Scheme() string

Scheme returns the scheme of this URI.

func (*URI) Set added in v0.8.0 

func (u *URI) Set(value string) error

Set sets the time quantum value.

func (*URI) SetHost added in v0.8.0 

func (u *URI) SetHost(host string) error

SetHost sets the host of this URI.

func (*URI) SetPort added in v0.8.0 

func (u *URI) SetPort(port uint16)

SetPort sets the port of this URI.

func (*URI) SetScheme added in v0.8.0 

func (u *URI) SetScheme(scheme string) error

SetScheme sets the scheme of this URI.

func (URI) String added in v0.8.0 

func (u URI) String() string

String returns the address as a string.

func (URI) Type added in v0.8.0 

func (u URI) Type() string

Type returns the type of a time quantum value.

func (*URI) UnmarshalJSON added in v0.9.0 

func (u *URI) UnmarshalJSON(b []byte) error

type URIs added in v0.9.0 

type URIs []URI

func (URIs) HostPortStrings added in v0.9.0 

func (u URIs) HostPortStrings() []string

type ValCount added in v0.9.0 

type ValCount struct {
	Val   int64 `json:"value"`
	Count int64 `json:"count"`
}

ValCount represents a grouping of sum & count for Sum() and Average() calls.

func (*ValCount) Add added in v0.9.0 

func (vc *ValCount) Add(other ValCount) ValCount

func (*ValCount) Larger added in v0.9.0 

func (vc *ValCount) Larger(other ValCount) ValCount

Larger returns the larger of the two ValCounts.

func (*ValCount) Smaller added in v0.9.0 

func (vc *ValCount) Smaller(other ValCount) ValCount

Smaller returns the smaller of the two ValCounts.

type VerboseLogger added in v0.9.0 

type VerboseLogger struct {
	// contains filtered or unexported fields
}

VerboseLogger is an implementation of pilosa.Logger which includes debug messages.

func NewVerboseLogger added in v0.9.0 

func NewVerboseLogger(w io.Writer) *VerboseLogger

func (*VerboseLogger) Debugf added in v0.9.0 

func (vb *VerboseLogger) Debugf(format string, v ...interface{})

func (*VerboseLogger) Logger added in v0.9.0 

func (vb *VerboseLogger) Logger() *log.Logger

func (*VerboseLogger) Printf added in v0.9.0 

func (vb *VerboseLogger) Printf(format string, v ...interface{})

type View 

type View struct {
	RowAttrStore AttrStore
	Logger       Logger
	// contains filtered or unexported fields
}

View represents a container for field data.

func NewView 

func NewView(path, index, field, name string, cacheSize uint32) *View

NewView returns a new instance of View.

func (*View) CreateFragmentIfNotExists 

func (v *View) CreateFragmentIfNotExists(shard uint64) (*Fragment, error)

CreateFragmentIfNotExists returns a fragment in the view by shard.

func (*View) Fragment 

func (v *View) Fragment(shard uint64) *Fragment

Fragment returns a fragment in the view by shard.

type ViewInfo 

type ViewInfo struct {
	Name string `json:"name"`
}

ViewInfo represents schema information for a view.

Source Files

View all Source files

Directories

Path Synopsis
cmd
Package cmd contains all the pilosa subcommand definitions (1 per file).
 Package cmd contains all the pilosa subcommand definitions (1 per file).
pilosa command
This is the entrypoint for the Pilosa binary.
 This is the entrypoint for the Pilosa binary.
package ctl contains all pilosa subcommands other than 'server'.
 package ctl contains all pilosa subcommands other than 'server'.
Package enterprise injects enterprise implementations of various Pilosa features when Pilosa is built with "ENTERPRISE=1 make install".
 Package enterprise injects enterprise implementations of various Pilosa features when Pilosa is built with "ENTERPRISE=1 make install".
b
Package internal is a generated protocol buffer package.
 Package internal is a generated protocol buffer package.
test
Package lru implements an LRU cache.
 Package lru implements an LRU cache.
Package pql defines the Pilosa Query Language.
 Package pql defines the Pilosa Query Language.
Package roaring implements roaring bitmaps with support for incremental changes.
 Package roaring implements roaring bitmaps with support for incremental changes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.