bundlestore

README

Bundlestore

Overview

Stores are the way we persist and retrieve bundles. They deal with the moving of bundle data, not its generation in any way. The storage may be local or remote, and for simplicity we don't expose that detail. Functions may perform special processing and caching that differ from store to store.

There are a few different store types that handle bundles differently. They can use bundles in memory, bundles on disk, and bundles located behind an http server. Further, we can wrap these stores in a parent store to add additional business logic to our handling.

Bazel Content Addressable Store

Bundlestore servers now also support the CAS API over gRPC, backed by the same underlying Store as the HTTP API.

Bundle name conventions

• HTTP Bundlestore server - For now names look like 'bs-.bundle'
• GRPC Bazel CAS server - based on Digest - 'blob-.blob'. See bazel/README.md Note that the above means that although the Store backing a server can support HTTP and GRPC APIs, the HTTP API naming restriction prevents accessing GRPC artifacts.

Server

Server makes a store accessible via http and doesn't do much else at this time. Future work includes ex: bundle validation, and generating better bundles with a different basis.

The use case for server is motivated by snapshot/git/gitdb. which needs to upload/download bundles from persistent storage and does so by contacting this [off-box] server via httpStore. Note that the server in turn may use httpStore internally to talk to, for instance, a SAN.

Server API

GET

Example:

curl -X GET -o local-output.bundle http://localhost:9094/bundle/bs-0000000000000000000000000000000000000000.bundle

POST

Example:

curl -X POST --data-binary "@/abspath/local-input.bundle" http://localhost:9094/bundle/bs-0000000000000000000000000000000000000000.bundle

Documentation

Index

• Constants
• func DefaultStore(store *store.FileStore) store.Store
• func Defaults() *ice.MagicBag
• func MakeFileStoreInEnvOrTemp(tmp *temp.TempDir) (*store.FileStore, error)
• func MakeHTTPServer(cfg *store.StoreConfig) *httpServer
• func Module() ice.Module
• func RunServer(bag *ice.MagicBag, schema jsonconfig.Schema, config []byte)
• type Server
  • func MakeServer(s store.Store, ttl *store.TTLConfig, stat stats.StatsReceiver, ...) *Server
  • func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request)

Constants

const (
	// Store-related environment variables
	BundlestoreDirEnvVar = "BUNDLESTORE_STORE_DIR"

	// Maximum concurrent connections allowed to access underlying Store resources
	MaxConnections = 100
)

Functions

func DefaultStore

func DefaultStore(store *store.FileStore) store.Store

func Defaults

func Defaults() *ice.MagicBag

Creates a MagicBag for a default bundlestore server and returns it

func MakeFileStoreInEnvOrTemp

func MakeFileStoreInEnvOrTemp(tmp *temp.TempDir) (*store.FileStore, error)

Make a File Store based on the environment, or in temp if unset

func MakeHTTPServer

func MakeHTTPServer(cfg *store.StoreConfig) *httpServer

func Module

func Module() ice.Module

Module returns a module that supports serving Bundlestore

func RunServer

func RunServer(bag *ice.MagicBag, schema jsonconfig.Schema, config []byte)

Types

type Server

type Server struct {
	// contains filtered or unexported fields
}

Represents a Bundlestore Server that serves HTTP and gRPC Bazel CAS APIs over an underlying Store with shared configuration and stats. StoreConfig is in part as a simplified solution to 2 problems golang-related problems we have with our current architecture: - values of Store and StatsReceiver interfaces don't pass down cleanly - encapsulating store, config and stats here prevented from being reused in cas because of cyclic dependency rules

func MakeServer

func MakeServer(s store.Store, ttl *store.TTLConfig, stat stats.StatsReceiver, gc *bazel.GRPCConfig) *Server

Make a new server that delegates to an underlying store. TTL may be nil, in which case defaults are applied downstream. TTL duration may be overriden by request headers, but we always pass this TTLKey to the store.

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request)

Implements http.Handler interface