gitdb

README

GitDB

GitDB is an implementation of ScootDB (snapshot.DB) that stores local values in a git ODB.

Code Structure Overview

• db.go structure definition, top-level entry point, concurrency control
• backends.go ID definition and parsing; backend definition
• create.go create Snapshots (locally)
• checkout.go run git commands to checkout
• local_data.go Snapshots stored locally
• stream.go get Snapshots from an upstream git repo

Backends

GitDB uses different Backends to identify, upload and download Snapshots.

Snapshot ID format

GitDB uses Snapshot IDs that encode the backend, kind and per-backend data, separated by '-':

• -(-)+

E.g.:

• local-gc-3cf4f8cf84976621c3ca9f19dff114842b4e5db7
  • backend: local
  • kind: GitCommitSnapshot *sha: 3cf4f8cf84976621c3ca9f19dff114842b4e5db7
• stream-gc-sm-3cf4f8cf84976621c3ca9f19dff114842b4e5db7
  • backend: stream
  • kind: GitCommitSnapshot
  • stream name: source_master *sha: 3cf4f8cf84976621c3ca9f19dff114842b4e5db7
• bs-fs-sm-530c6daad567a0765c10064e1c7fc4fa486a2638-d1f58ef31066244fc8590bfb6940b4060b1baab1
  • backend: Bundlestore
  • kind: FSSnapshot
  • stream name: source_master
  • bundle name: 530c6daad567a0765c10064e1c7fc4fa486a2638
  • sha: d1f58ef31066244fc8590bfb6940b4060b1baab1

Documentation

Index

• func CreateBundlestoreSnapshot(sha string, kind SnapshotKind, bundleKey, streamName string) snapshot
• func Module() ice.Module
• type AutoUploadDest
• type BundlestoreConfig
• type DB
  • func MakeDBFromRepo(dataRepo *repo.Repository, updater RepoUpdater, tmp *temp.TempDir, ...) *DB
  • func MakeDBNewRepo(initer RepoIniter, updater RepoUpdater, tmp *temp.TempDir, ...) *DB
  • func (db *DB) Checkout(id snap.ID) (path string, err error)
  • func (db *DB) Close()
  • func (db *DB) ExportGitCommit(id snap.ID, exportRepo *repo.Repository) (string, error)
  • func (db *DB) IDForStreamCommitSHA(streamName string, sha string) snap.ID
  • func (db *DB) IngestDir(dir string) (snap.ID, error)
  • func (db *DB) IngestGitCommit(ingestRepo *repo.Repository, commitish string) (snap.ID, error)
  • func (db *DB) IngestGitWorkingDir(ingestRepo *repo.Repository) (snap.ID, error)
  • func (db *DB) ReadFileAll(id snap.ID, path string) ([]byte, error)
  • func (db *DB) ReleaseCheckout(path string) error
  • func (db *DB) StreamName() string
  • func (db *DB) Update() error
  • func (db *DB) UpdateInterval() time.Duration
  • func (db *DB) UploadFile(filePath string, ttl *store.TTLValue) (string, error)
• type NoopRepoUpdater
  • func (u *NoopRepoUpdater) Update(*repo.Repository) error
  • func (u *NoopRepoUpdater) UpdateInterval() time.Duration
• type RepoIniter
• type RepoUpdater
• type SnapshotKind
• type StreamConfig
• type TagsConfig
• type TmpRepoIniter
  • func (i *TmpRepoIniter) Init() (*repo.Repository, error)

Functions

func CreateBundlestoreSnapshot

func CreateBundlestoreSnapshot(sha string, kind SnapshotKind, bundleKey, streamName string) snapshot

func Module

func Module() ice.Module

Module returns a module that supports typical GitDB usage

Types

type AutoUploadDest

type AutoUploadDest int

const (
	AutoUploadNone AutoUploadDest = iota
	AutoUploadTags
	AutoUploadBundlestore
)

type BundlestoreConfig

type BundlestoreConfig struct {
	Store store.Store
}

BundlestoreConfig defines how to talk to Bundlestore

type DB

type DB struct {
	InitDoneCh chan error
	// contains filtered or unexported fields
}

DB stores its data in a Git Repo

func MakeDBFromRepo

func MakeDBFromRepo(
	dataRepo *repo.Repository,
	updater RepoUpdater,
	tmp *temp.TempDir,
	stream *StreamConfig,
	tags *TagsConfig,
	bundles *BundlestoreConfig,
	autoUploadDest AutoUploadDest,
	stat stats.StatsReceiver) *DB

MakeDBFromRepo makes a gitdb.DB that uses dataRepo for data and tmp for temporary directories

func MakeDBNewRepo

func MakeDBNewRepo(
	initer RepoIniter,
	updater RepoUpdater,
	tmp *temp.TempDir,
	stream *StreamConfig,
	tags *TagsConfig,
	bundles *BundlestoreConfig,
	autoUploadDest AutoUploadDest,
	stat stats.StatsReceiver) *DB

MakeDBNewRepo makes a gitDB that uses a new DB, populated by initer

func (*DB) Checkout

func (db *DB) Checkout(id snap.ID) (path string, err error)

Checkout puts the snapshot identified by id in the local filesystem, returning the path where it lives or an error.

func (*DB) Close

func (db *DB) Close()

Close stops the DB

func (*DB) ExportGitCommit

func (db *DB) ExportGitCommit(id snap.ID, exportRepo *repo.Repository) (string, error)

ExportGitCommit applies a snapshot to a repository, returning the sha of the exported commit or an error.

func (*DB) IDForStreamCommitSHA

func (db *DB) IDForStreamCommitSHA(streamName string, sha string) snap.ID

IDForStreamCommitSHA gets a SnapshotID from a string name and commit sha

func (*DB) IngestDir

func (db *DB) IngestDir(dir string) (snap.ID, error)

IngestDir ingests a directory directly.

func (*DB) IngestGitCommit

func (db *DB) IngestGitCommit(ingestRepo *repo.Repository, commitish string) (snap.ID, error)

IngestGitCommit ingests the commit identified by commitish from ingestRepo

func (*DB) IngestGitWorkingDir

func (db *DB) IngestGitWorkingDir(ingestRepo *repo.Repository) (snap.ID, error)

IngestGitWorkingDir ingests HEAD + working dir modifications from the ingestRepo.

func (*DB) ReadFileAll

func (db *DB) ReadFileAll(id snap.ID, path string) ([]byte, error)

ReadFileAll reads the contents of the file path in FSSnapshot ID, or errors

func (*DB) ReleaseCheckout

func (db *DB) ReleaseCheckout(path string) error

ReleaseCheckout releases a path from a previous Checkout. This allows Scoot to reuse the path. Scoot will not touch path after Checkout until ReleaseCheckout.

func (*DB) StreamName

func (db *DB) StreamName() string

func (*DB) Update

func (db *DB) Update() error

Update is used to trigger an underlying RepoUpdater to Update() the Repository. If db.updater is nil, has no effect.

func (*DB) UpdateInterval

func (db *DB) UpdateInterval() time.Duration

Returns RepoUpdater's interval if an updater exists, or a zero duration

func (*DB) UploadFile

func (db *DB) UploadFile(filePath string, ttl *store.TTLValue) (string, error)

Manual write of a file (like an existing bundle) to the underlying store Intended for HTTP-backed stores that implement bundlestore's TTL fields Base of the filePath will be used as bundle name Returns location of stored file

type NoopRepoUpdater

type NoopRepoUpdater struct{}

func (*NoopRepoUpdater) Update

func (u *NoopRepoUpdater) Update(*repo.Repository) error

func (*NoopRepoUpdater) UpdateInterval

func (u *NoopRepoUpdater) UpdateInterval() time.Duration

type RepoIniter

type RepoIniter interface {
	Init() (*repo.Repository, error)
}

Interface for defining initialization that results in a valid repo.Repository. Distinct from gitfiler.RepoIniter

type RepoUpdater

type RepoUpdater interface {
	Update(r *repo.Repository) error
	UpdateInterval() time.Duration
}

Interface for defining update behavior for an underlying repo.Repository. Provides an update mechanism and an interval definition

type SnapshotKind

type SnapshotKind string

SnapshotKind describes the kind of a Snapshot: is it an FSSnapshot or a GitCommitSnapshot kind instead of type because type is a keyword

const (
	KindFSSnapshot        SnapshotKind = "fs"
	KindGitCommitSnapshot SnapshotKind = "gc"
)

type StreamConfig

type StreamConfig struct {
	// Name (used in IDs (so it should be short)
	// e.g. sm for a Stream following Source (repo)'s Master (branch)
	//      name may include a ref string that'll override the refspec configured for 'sm'
	//      the full id might look like "stream-gc-sm:<branch>-<sha>" for a given branch and sha
	Name string

	// Remote to fetch from (e.g. https://github.com/twitter/scoot)
	Remote string

	// Name of ref to follow in data repo (e.g. refs/remotes/upstream/master)
	RefSpec string
}

A Stream is a sequence of GitCommitSnapshots that updates. The backend is a git refspec that can be fetched from a Git remote.

type TagsConfig

type TagsConfig struct {
	// Name (used in IDs (so it should be short)
	// e.g. sss for Source (repo)'s Scoot Snapshots
	Name string

	// Remote to fetch from and push to (e.g. https://github.com/twitter/scoot)
	Remote string

	// Prefix for tag names (e.g. reserved_scoot)
	Prefix string
}

TagsConfig specifies how GitDB should store Snapshots as tags in another git repo

type TmpRepoIniter

type TmpRepoIniter struct {
	// contains filtered or unexported fields
}

TmpRepoIniter creates a new Repo in a temp dir

func (*TmpRepoIniter) Init

func (i *TmpRepoIniter) Init() (*repo.Repository, error)

Init creates a new temp dir and a repo in it