git

README

git-prep-directory

Build tools friendly way of repeatedly cloning a git repository using a submodule cache and keeping timestamps to commit times.

Concept

Executing the tool/library repeatedly on the same repository with different commit refs results in two optimizations:

• Faster clone times due to less network traffic as most submodules change so infrequently that they are likely already exist in the submodule cache.
• Faster build times for tools like Make and Docker as unchanged files keep their timestamp this are cached by the build tools.

For each execution of a given commit ref the tool/library does the follow steps:

1. Clone the given repository
2. Checkout the given revision
3. Find all submodules and check if they are already cached

• If cached, assert the required commit is checked out and initialize them
• If not cached, checkout the submodule and store it in the cache.

1. Set the timestamp of all files and directories to their respective commit time.

This results in the following path hierarchy:

.
└── src
    ├── HEAD
    ├── c                       <-- repository commits
    │   ├── 39529a7ed3
    │   ├── 5e90e55e6b
    │   ├── 607f0489ec
    │   └── :
    ├── config
    ├── index
    ├── modules                 <-- submodules cache
    │   ├── path/to/submodule/1
    │   ├── path/to/submodule/2
    │   ├── path/to/submodule/3
    │   └── :
    ├── objects
    ├── packed-refs
    └── refs

Command line tool

Installation

$ go get github.com/sensiblecodeio/git-prep-directory/cmd/git-prep-directory

Usage

$ git-prep-directory --help
NAME:
   git-prep-directory - Build tools friendly way of repeatedly cloning a git
   repository using a submodule cache and setting file timestamps to commit times.

USAGE:
   git-prep-directory [global options] command [command options] [arguments...]

VERSION:
   1.0.0

COMMANDS:
   help, h      Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --url, -u                    URL to clone
   --ref, -r                    ref to checkout
   --destination, -d "./src"    destination dir
   --help, -h                   show help
   --version, -v                print the version

Go Library

import "github.com/sensiblecodeio/git-prep-directory"

buildDirectory, err := git.PrepBuildDirectory(<OUT_PATH>, <REPO_URL>, <GIT_REF>)

Documentation

Index

• Variables
• func AlreadyHaveRef(gitDir, sha string) bool
• func Checkout(gitDir, checkoutDir, ref string) error
• func Chtimes(path string, atime, mtime time.Time) error
• func Clone(ctx context.Context, url, gitDir string, messages io.Writer) error
• func Command(workdir, command string, args ...string) *exec.Cmd
• func CommitTimes(gitDir, revision string) (map[string]time.Time, error)
• func ContextRun(ctx context.Context, cmd *exec.Cmd) error
• func Describe(gitDir, ref string) (desc string, err error)
• func Fetch(ctx context.Context, gitDir, url string, messages io.Writer) error
• func GetSubmoduleRev(gitDir, submodulePath, mainRev string) (string, error)
• func GetSubmoduleRevs(gitDir, mainRev string, submodules []Submodule) error
• func HaveFile(gitDir, ref, path string) (ok bool, err error)
• func LocalMirror(url, gitDir, ref string, timeout time.Duration, messages io.Writer) error
• func MultipleErrors(errs <-chan error) error
• func PrepSubmodules(gitDir, checkoutDir, mainRev string, timeout time.Duration, messages io.Writer) error
• func RecursiveCheckout(gitDir, checkoutPath, rev string, timeout time.Duration, messages io.Writer) error
• func RevParse(gitDir, ref string) (sha string, err error)
• func SafeCleanup(path string) error
• func SetMTimes(gitDir, checkoutDir, ref string) error
• type BuildDirectory
  • func PrepBuildDirectory(gitDir, remote, ref string, timeout time.Duration, messages io.Writer) (*BuildDirectory, error)
• type ErrMultiple
  • func (em *ErrMultiple) Error() string
• type Submodule
  • func ParseSubmodules(filename string) ([]Submodule, error)

Variables

var ShaLike = regexp.MustCompile("[0-9a-zA-Z]{40}")

ShaLike specifies a valid git hash.

Functions

func AlreadyHaveRef

func AlreadyHaveRef(gitDir, sha string) bool

AlreadyHaveRef returns true if ref is sha-like and is in the object database. The "sha-like" condition ensures that refs like `master` are always freshened.

func Checkout

func Checkout(gitDir, checkoutDir, ref string) error

Checkout switches branches or restores working tree files.

func Chtimes

func Chtimes(path string, atime, mtime time.Time) error

Chtimes changes the modification and access time of a given file.

func Clone

func Clone(ctx context.Context, url, gitDir string, messages io.Writer) error

Clone clones a git repository as mirror.

func Command

func Command(workdir, command string, args ...string) *exec.Cmd

Command invokes a `command` in `workdir` with `args`, connecting Stdout and Stderr to Stderr.

func CommitTimes

func CommitTimes(gitDir, revision string) (map[string]time.Time, error)

CommitTimes returns the most recent committed timestamp of each file in the whole of history. It's faster than invoking 'git log -1' on each file.

func ContextRun

func ContextRun(ctx context.Context, cmd *exec.Cmd) error

ContextRun runs cmd within a net Context. If the context is cancelled or times out, the process is killed.

func Describe

func Describe(gitDir, ref string) (desc string, err error)

Describe describes a commit given a reference using the most recent tag reachable from it.

func Fetch

func Fetch(ctx context.Context, gitDir, url string, messages io.Writer) error

Fetch fetches all branches from a given remote.

func GetSubmoduleRev

func GetSubmoduleRev(gitDir, submodulePath, mainRev string) (string, error)

GetSubmoduleRev returns the revisions of all files in a given submodule.

func GetSubmoduleRevs

func GetSubmoduleRevs(gitDir, mainRev string, submodules []Submodule) error

GetSubmoduleRevs returns the revisions of all files in a given list of submodules.

func HaveFile

func HaveFile(gitDir, ref, path string) (ok bool, err error)

HaveFile checks if a git directory has files checked out.

func LocalMirror

func LocalMirror(url, gitDir, ref string, timeout time.Duration, messages io.Writer) error

LocalMirror creates or updates a mirror of `url` at `gitDir` using `git clone --mirror`.

func MultipleErrors

func MultipleErrors(errs <-chan error) error

MultipleErrors reads errors out of a channel, counting only the non-nil ones. If there are zero non-nil errs, nil is returned.

func PrepSubmodules

func PrepSubmodules(gitDir, checkoutDir, mainRev string, timeout time.Duration, messages io.Writer) error

PrepSubmodules in parallel initializes all submodules and additionally stores them in a local cache.

func RecursiveCheckout

func RecursiveCheckout(gitDir, checkoutPath, rev string, timeout time.Duration, messages io.Writer) error

RecursiveCheckout recursively checks out repositories; similar to "git clone --recursive".

func RevParse

func RevParse(gitDir, ref string) (sha string, err error)

RevParse parses and formats the git rev of a given git reference.

func SafeCleanup

func SafeCleanup(path string) error

SafeCleanup recursively removes all files from a given path, which has to be a subfolder of the current working directory.

func SetMTimes

func SetMTimes(gitDir, checkoutDir, ref string) error

SetMTimes changes the modification and access time of all files in a given directory to their latest commit time.

Types

type BuildDirectory

type BuildDirectory struct {
	Name    string
	Dir     string
	Cleanup func()
}

BuildDirectory holds the git rev and path to a cloned git repository. It also holds a cleanup function to safely remove this directory.

func PrepBuildDirectory

func PrepBuildDirectory(gitDir, remote, ref string, timeout time.Duration, messages io.Writer) (*BuildDirectory, error)

PrepBuildDirectory clones a given repository and checks out the given revision, setting the timestamp of all files to their commit time and putting all submodules into a submodule cache.

type ErrMultiple

type ErrMultiple struct {
	// contains filtered or unexported fields
}

ErrMultiple holds a list of errors.

func (*ErrMultiple) Error

func (em *ErrMultiple) Error() string

type Submodule

type Submodule struct {
	Path string
	URL  string
	Rev  string // populated by GetSubmoduleRevs
}

Submodule holds the path, url, and revision to a submodule.

func ParseSubmodules

func ParseSubmodules(filename string) ([]Submodule, error)

ParseSubmodules returns all submodule definitions given a .gitmodules configuration.