bigtable

Documentation

Overview

Package bigtable provides an implementation of the Storage interface backed by Google Cloud Platform's BigTable.

Intermediate Log Table

The Intermediate Log Table stores LogEntry protobufs that have been ingested, but haven't yet been archived. It is a tall table whose rows are keyed off of the log's (Path,Stream-Index) in that order.

Each entry in the table will contain the following schema:

• Column Family "log"
• Column "data": the LogEntry raw protobuf data. Soft size limit of ~1MB.

The log path is the composite of the log's (Prefix, Name) properties. Logs belonging to the same stream will share the same path, so they will be clustered together and suitable for efficient iteration. Immediately following the path will be the log's stream index.

[            20 bytes          ]     ~    [       1-5 bytes      ]
 B64(SHA256(Path(Prefix, Name)))  + '~' + HEX(cmpbin(StreamIndex))

As there is no (technical) size constraint to either the Prefix or Name values, they will both be hashed using SHA256 to produce a unique key representing that specific log stream.

This allows a key to be generated representing "immediately after the row" by appending two '~' characters to the base hash. Since the second '~' character is always greater than any HEX(cmpbin(*)) value, this will effectively upper-bound the row.

"cmpbin" (github.com/luci/luci-go/common/cmpbin) will be used to format the stream index. It is a variable-width number encoding scheme that offers the guarantee that byte-sorted encoded numbers will maintain the same order as the numbers themselves.

Index

• Constants
• Variables
• func Initialize(ctx context.Context, o Options) error
• func New(ctx context.Context, o Options) (storage.Storage, error)
• type Options
• type Testing
  • func NewMemoryInstance(c context.Context, opts Options) Testing

Constants

const DefaultMaxLogAge = time.Duration(7 * 24 * time.Hour)

DefaultMaxLogAge is the maximum age of a log (7 days).

Variables

var (
	// StorageScopes is the set of OAuth scopes needed to use the storage
	// functionality.
	StorageScopes = []string{
		bigtable.Scope,
	}

	// StorageReadOnlyScopes is the set of OAuth scopes needed to use the storage
	// functionality.
	StorageReadOnlyScopes = []string{
		bigtable.ReadonlyScope,
	}
)

var InitializeScopes = []string{
	bigtable.AdminScope,
}

InitializeScopes is the set of OAuth scopes needed to use the Initialize functionality.

Functions

func Initialize

func Initialize(ctx context.Context, o Options) error

Initialize sets up a Storage schema in BigTable. If the schema is already set up properly, no action will be taken.

If, however, the table or table's schema doesn't exist, Initialize will create and configure it.

If nil is returned, the table is ready for use as a Storage via New.

func New

func New(ctx context.Context, o Options) (storage.Storage, error)

New instantiates a new Storage instance connected to a BigTable instance.

The returned Storage instance will close the Client when its Close() method is called.

Types

type Options

type Options struct {
	// Project is the name of the project to connect to.
	Project string
	// Instance is the name of the instance to connect to.
	Instance string
	// ClientOptions are additional client options to use when instantiating the
	// client instance.
	ClientOptions []option.ClientOption

	// Table is the name of the BigTable table to use for logs.
	LogTable string

	// Cache, if not nil, will be used to cache data.
	Cache caching.Cache
}

Options is a set of configuration options for BigTable storage.

type Testing

type Testing interface {
	storage.Storage

	DataMap() map[string][]byte
	SetMaxRowSize(int)
	SetErr(error)
	MaxLogAge() time.Duration
}

Testing is an extension of storage.Storage with additional testing capabilities.

func NewMemoryInstance

func NewMemoryInstance(c context.Context, opts Options) Testing

NewMemoryInstance returns an in-memory BigTable Storage implementation. This can be supplied in the Raw field in Options to simulate a BigTable connection.

Close should be called on the resulting value after the user is finished in order to free resources.