btts

Documentation

Overview

See BIGTABLE.md for tiles and traces are stored in BigTable.

Index

• Constants
• type BigTableTraceStore
  • func NewBigTableTraceStoreFromConfig(ctx context.Context, cfg *config.PerfBigTableConfig, ts oauth2.TokenSource, ...) (*BigTableTraceStore, error)
  • func (b *BigTableTraceStore) GetLatestTile() (TileKey, error)
  • func (b *BigTableTraceStore) GetOrderedParamSet(tileKey TileKey) (*paramtools.OrderedParamSet, error)
  • func (b *BigTableTraceStore) GetSource(index int32, traceId string) (string, error)
  • func (b *BigTableTraceStore) OffsetFromIndex(index int32) int32
  • func (b *BigTableTraceStore) QueryCount(tileKey TileKey, q *regexp.Regexp) (int64, error)
  • func (b *BigTableTraceStore) QueryTraces(tileKey TileKey, q *regexp.Regexp) (map[string][]float32, error)
  • func (b *BigTableTraceStore) ReadTraces(tileKey TileKey, keys []string) (map[string][]float32, error)
  • func (b *BigTableTraceStore) TileKey(index int32) TileKey
  • func (b *BigTableTraceStore) UpdateOrderedParamSet(tileKey TileKey, p paramtools.ParamSet) (*paramtools.OrderedParamSet, error)
  • func (b *BigTableTraceStore) WriteTraces(index int32, values map[string]float32, source string, timestamp time.Time) error
• type OpsCacheEntry
  • func NewOpsCacheEntry() (*OpsCacheEntry, error)
  • func NewOpsCacheEntryFromRow(row bigtable.Row) (*OpsCacheEntry, error)
• type TileKey
  • func TileKeyFromOffset(tileOffset int32) TileKey
  • func TileKeyFromOpsRowName(s string) (TileKey, error)
  • func (t TileKey) Offset() int32
  • func (t TileKey) OpsRowName() string
  • func (t TileKey) PrevTile() TileKey
  • func (t TileKey) TraceRowName(traceId string, shards int32) string
  • func (t TileKey) TraceRowPrefix(shard int32) string

Constants

const (
	// BigTable column families.
	VALUES_FAMILY  = "V"
	SOURCES_FAMILY = "S"
	HASHES_FAMILY  = "H"
	OPS_FAMILY     = "D"

	// Columns in the "H" column family.
	HASHES_SOURCE_COLUMN        = "S" // Column
	HASHES_SOURCE_FULL_COL_NAME = HASHES_FAMILY + ":" + HASHES_SOURCE_COLUMN

	// Columns in the "D" column family.
	OPS_HASH_COLUMN    = "H"   // Column
	OPS_OPS_COLUMN     = "OPS" // Column
	HASH_FULL_COL_NAME = OPS_FAMILY + ":" + OPS_HASH_COLUMN
	OPS_FULL_COL_NAME  = OPS_FAMILY + ":" + OPS_OPS_COLUMN

	// MAX_MUTATIONS is the max number of mutations we can send in a single ApplyBulk call. Can be up to 100,000 according to BigTable docs.
	MAX_MUTATIONS = 100000

	TIMEOUT       = 4 * time.Minute
	WRITE_TIMEOUT = 10 * time.Minute
)

const BadTileKey = TileKey(-1)

BadTileKey is returned in error conditions.

Types

type BigTableTraceStore

type BigTableTraceStore struct {
	// contains filtered or unexported fields
}

func NewBigTableTraceStoreFromConfig

func NewBigTableTraceStoreFromConfig(ctx context.Context, cfg *config.PerfBigTableConfig, ts oauth2.TokenSource, cacheOps bool) (*BigTableTraceStore, error)

func (*BigTableTraceStore) GetLatestTile

func (b *BigTableTraceStore) GetLatestTile() (TileKey, error)

GetLatestTile returns the latest, i.e. the newest tile.

func (*BigTableTraceStore) GetOrderedParamSet

func (b *BigTableTraceStore) GetOrderedParamSet(tileKey TileKey) (*paramtools.OrderedParamSet, error)

GetOrderedParamSet returns the OPS for the given tile.

func (*BigTableTraceStore) GetSource

func (b *BigTableTraceStore) GetSource(index int32, traceId string) (string, error)

GetSource returns the full GCS URL of the file that contained the point at 'index' of trace 'traceId'.

The traceId must be an OPS encoded key.

func (*BigTableTraceStore) OffsetFromIndex

func (b *BigTableTraceStore) OffsetFromIndex(index int32) int32

Returns the offset within a tile for the given index.

func (*BigTableTraceStore) QueryCount

func (b *BigTableTraceStore) QueryCount(tileKey TileKey, q *regexp.Regexp) (int64, error)

QueryCount does the same work as QueryTraces but only returns the number of traces that would be returned.

func (*BigTableTraceStore) QueryTraces

func (b *BigTableTraceStore) QueryTraces(tileKey TileKey, q *regexp.Regexp) (map[string][]float32, error)

QueryTraces returns a map of encoded keys to a slice of floats for all traces that match the given query.

func (*BigTableTraceStore) ReadTraces

func (b *BigTableTraceStore) ReadTraces(tileKey TileKey, keys []string) (map[string][]float32, error)

ReadTraces loads the traces for the given keys.

Note that the keys are the structured keys, ReadTraces will convert them into OPS encoded keys.

func (*BigTableTraceStore) TileKey

func (b *BigTableTraceStore) TileKey(index int32) TileKey

Given the index return the TileKey of the tile that would contain that column.

func (*BigTableTraceStore) UpdateOrderedParamSet

func (b *BigTableTraceStore) UpdateOrderedParamSet(tileKey TileKey, p paramtools.ParamSet) (*paramtools.OrderedParamSet, error)

UpdateOrderedParamSet will add all params from 'p' to the OrderedParamSet for 'tileKey' and write it back to BigTable.

func (*BigTableTraceStore) WriteTraces

func (b *BigTableTraceStore) WriteTraces(index int32, values map[string]float32, source string, timestamp time.Time) error

WriteTraces writes the given values into the store.

The keys of 'values' must be the OPS encoded Params of the trace, i.e. at this point we know the OPS has been updated.

type OpsCacheEntry

type OpsCacheEntry struct {
	// contains filtered or unexported fields
}

When ingesting we keep a cache of the OrderedParamSets we have seen per-tile.

func NewOpsCacheEntry

func NewOpsCacheEntry() (*OpsCacheEntry, error)

func NewOpsCacheEntryFromRow

func NewOpsCacheEntryFromRow(row bigtable.Row) (*OpsCacheEntry, error)

type TileKey

type TileKey int32

TileKey is the identifier for each tile held in BigTable.

Note that tile keys are in the opposite order of tile offset, that is, the first tile for a repo would be 0, and then 1, etc. Those are the offsets, and the most recent tile has the largest offset. To make it easy to find the most recent tile we calculate tilekey as math.MaxInt32 - tileoffset, so that more recent tiles come first in sort order.

func TileKeyFromOffset

func TileKeyFromOffset(tileOffset int32) TileKey

TileKeyFromOffset returns a TileKey from the tile offset.

func TileKeyFromOpsRowName

func TileKeyFromOpsRowName(s string) (TileKey, error)

func (TileKey) Offset

func (t TileKey) Offset() int32

Offset returns the tile offset, i.e. the not-reversed number.

func (TileKey) OpsRowName

func (t TileKey) OpsRowName() string

OpsRowName returns the name of the BigTable row that the OrderedParamSet for this tile is stored at.

func (TileKey) PrevTile

func (t TileKey) PrevTile() TileKey

func (TileKey) TraceRowName

func (t TileKey) TraceRowName(traceId string, shards int32) string

TraceRowPrefix returns the BigTable row name for the given trace, for the given number of shards. TraceRowName(",0=1,", 3) -> 2:2147483647:,0=1,

func (TileKey) TraceRowPrefix

func (t TileKey) TraceRowPrefix(shard int32) string

TraceRowPrefix returns the prefix of a BigTable row name for any Trace in this tile.