gitcollector

README

gitcollector

gitcollector collects and stores git repositories.

gitcollector is the source{d} tool to download and update git repositories at large scale. To that end, it uses a custom repository storage file format called siva optimized for saving storage space and keeping repositories up-to-date.

Status

The project is currently in beta stage.

Storing repositories using rooted repositories

A rooted repository is a bare Git repository that stores all objects from all repositories that share a common history, that is, they have the same initial commit. It is stored using the Siva file format.

Rooted repositories have a few particularities that you should know to work with them effectively:

• They have no HEAD reference.
• All references are of the following form: {REFERENCE_NAME}/{REMOTE_NAME}. For example, the reference refs/heads/master of the remote foo would be /refs/heads/master/foo.
• Each remote represents a repository that shares the common history of the rooted repository. A remote can have multiple endpoints.
• A rooted repository is simply a repository with all the objects from all the repositories which share the same root commit.
• The root commit for a repository is obtained following the first parent of each commit from HEAD.

Getting started

Plain command

gitcollector entry point usage is done through the subcommand download (at this time is the only subcommand):

Usage:
  gitcollector [OPTIONS] download [download-OPTIONS]

Help Options:
  -h, --help                                     Show this help message

[download command options]
          --library=                             path where download to [$GITCOLLECTOR_LIBRARY]
          --bucket=                              library bucketization level (default: 2) [$GITCOLLECTOR_LIBRARY_BUCKET]
          --tmp=                                 directory to place generated temporal files (default: /tmp) [$GITCOLLECTOR_TMP]
          --workers=                             number of workers, default to GOMAXPROCS [$GITCOLLECTOR_WORKERS]
          --no-updates                           don't allow updates on already downloaded repositories [$GITCOLLECTOR_NO_UPDATES]
          --orgs=                                list of github organization names separated by comma [$GITHUB_ORGANIZATIONS]
          --token=                               github token [$GITHUB_TOKEN]
          --metrics-db=                          uri to a database where metrics will be sent [$GITCOLLECTOR_METRICS_DB_URI]
          --metrics-db-table=                    table name where the metrics will be added (default: gitcollector_metrics) [$GITCOLLECTOR_METRICS_DB_TABLE]
          --metrics-sync-timeout=                timeout in seconds to send metrics (default: 30) [$GITCOLLECTOR_METRICS_SYNC]

    Log Options:
          --log-level=[info|debug|warning|error] Logging level (default: info) [$LOG_LEVEL]
          --log-format=[text|json]               log format, defaults to text on a terminal and json otherwise [$LOG_FORMAT]
          --log-fields=                          default fields for the logger, specified in json [$LOG_FIELDS]
          --log-force-format                     ignore if it is running on a terminal or not [$LOG_FORCE_FORMAT]

Usage example, --library and --orgs are always required:

gitcollector download --library=/path/to/repos/directoy --orgs=src-d

To collect repositories from several github organizations:

gitcollector download --library=/path/to/repos/directoy --orgs=src-d,bblfsh

Note that all the download command options are also configurable with environment variables.

Docker

gitcollector upload a new docker image to docker hub on each new release. To use it:

docker run --rm --name gitcollector_1 \
-e "GITHUB_ORGANIZATIONS=src-d,bblfsh" \
-e "GITHUB_TOKEN=foo" \
-v /path/to/repos/directory:/library \
gitcollector:latest

Note that you must mount a local directory into the specific container path shown in -v /path/to/repos/directory:/library. This directory is where the repositories will be downloaded into rooted repositories in siva files format.

License

GPL v3.0, see LICENSE

Documentation

Index

• Variables
• type Job
• type JobScheduler
  • func NewJobScheduler(schedule ScheduleFn, opts *JobSchedulerOpts) *JobScheduler
  • func (s *JobScheduler) Finish()
  • func (s *JobScheduler) Jobs() chan Job
  • func (s *JobScheduler) Schedule()
• type JobSchedulerOpts
• type MetricsCollector
• type Provider
• type ScheduleFn
• type Worker
  • func NewWorker(jobs chan Job, metrics MetricsCollector) *Worker
  • func (w *Worker) Start()
  • func (w *Worker) Stop(immediate bool)
• type WorkerPool
  • func NewWorkerPool(scheduler *JobScheduler, metrics MetricsCollector) *WorkerPool
  • func (wp *WorkerPool) Close()
  • func (wp *WorkerPool) Run()
  • func (wp *WorkerPool) SetWorkers(n int)
  • func (wp *WorkerPool) Size() int
  • func (wp *WorkerPool) Stop()
  • func (wp *WorkerPool) Wait()

Variables

var (
	// ErrProviderStopped is returned when a provider has been stopped.
	ErrProviderStopped = errors.NewKind("provider stopped")

	// ErrProviderStop is returned when a provider fails on Stop.
	ErrProviderStop = errors.NewKind("provider failed on stop")
)

var (
	// ErrNewJobsNotFound is returned if there's no more jobs to schedule.
	ErrNewJobsNotFound = errors.NewKind(
		"couldn't find new jobs to schedule")

	// ErrClosedChannel is returned if the jobs source is closed.
	ErrClosedChannel = errors.NewKind("channel is closed")
)

Types

type Job

type Job interface {
	// Process perform the necessary work on the job.
	Process(context.Context) error
}

Job represents a gitcollector task.

type JobScheduler

type JobScheduler struct {
	// contains filtered or unexported fields
}

JobScheduler schedules the Jobs to be processed.

func NewJobScheduler

func NewJobScheduler(
	schedule ScheduleFn,
	opts *JobSchedulerOpts,
) *JobScheduler

NewJobScheduler builds a new JobScheduler.

func (*JobScheduler) Finish

func (s *JobScheduler) Finish()

Finish finishes to schedule Jobs.

func (*JobScheduler) Jobs

func (s *JobScheduler) Jobs() chan Job

Jobs returns the channel where the JobScheduler will schedule the Jobs.

func (*JobScheduler) Schedule

func (s *JobScheduler) Schedule()

Schedule schedules Jobs.

type JobSchedulerOpts

type JobSchedulerOpts struct {
	Capacity       int
	NotWaitNewJobs bool
	NewJobTimeout  time.Duration
	JobTimeout     time.Duration
}

JobSchedulerOpts are configuration options for a JobScheduler.

type MetricsCollector

type MetricsCollector interface {
	// Start starts collecting metrics.
	Start()
	// Stop stops collectingMetrincs.
	Stop(immediate bool)
	// Success registers metrics about successfully processed Job.
	Success(Job)
	// Faile register metrics about a failed processed Job.
	Fail(Job)
	// Discover register metrics about a discovered Job.
	Discover(Job)
}

MetricsCollector represents a component in charge to collect jobs metrics.

type Provider

type Provider interface {
	Start() error
	Stop() error
}

Provider interface represents a service to generate new Jobs.

type ScheduleFn

type ScheduleFn func(*JobSchedulerOpts) (Job, error)

ScheduleFn is a function to schedule the next Job.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker is in charge of process gitcollector.Jobs.

func NewWorker

func NewWorker(jobs chan Job, metrics MetricsCollector) *Worker

NewWorker builds a new Worker.

func (*Worker) Start

func (w *Worker) Start()

Start starts the Worker. It shouldn't be restarted after a call to Stop.

func (*Worker) Stop

func (w *Worker) Stop(immediate bool)

Stop stops the Worker.

type WorkerPool

type WorkerPool struct {
	// contains filtered or unexported fields
}

WorkerPool holds a pool of workers to process Jobs.

func NewWorkerPool

func NewWorkerPool(
	scheduler *JobScheduler,
	metrics MetricsCollector,
) *WorkerPool

NewWorkerPool builds a new WorkerPool.

func (*WorkerPool) Close

func (wp *WorkerPool) Close()

Close stops all the workers in the pool waiting for the jobs to finish.

func (*WorkerPool) Run

func (wp *WorkerPool) Run()

Run notify workers to start.

func (*WorkerPool) SetWorkers

func (wp *WorkerPool) SetWorkers(n int)

SetWorkers set the number of Workers in the pool to n.

func (*WorkerPool) Size

func (wp *WorkerPool) Size() int

Size returns the current number of workers in the pool.

func (*WorkerPool) Stop

func (wp *WorkerPool) Stop()

Stop stops all the workers in the pool immediately.

func (*WorkerPool) Wait

func (wp *WorkerPool) Wait()

Wait waits for the workers to finish. A worker will finish when the queue to retrieve jobs from is closed.