history

README

history

A Go library for walking commits, file changes, trees, and blobs in Git repositories without invoking the Git binary. It reads loose and packed objects through go-git. Shallow clones, linked worktrees, alternate object stores, and SHA-1 or SHA-256 repositories are supported.

Installation

go get github.com/git-pkgs/history

Usage

Walk a revision

WalkCommits visits commits oldest-first. Each result contains the commit and its retained first-parent file changes, and commits with no retained changes are still visited.

package main

import (
	"fmt"
	"log"

	"github.com/git-pkgs/history"
)

func main() {
	repo, err := history.Open(".")
	if err != nil {
		log.Fatal(err)
	}
	defer repo.Close()

	err = repo.WalkCommits(history.CommitOptions{
		Ref:     "HEAD",
		Workers: 4,
	}, func(commit history.Commit) error {
		fmt.Printf("%s %s\n", commit.Object.Hash, history.CommitSubject(commit.Object.Message))
		for _, change := range commit.Changes {
			fmt.Printf("  %s %s\n", change.NewMode, change.Path)
		}
		return nil
	})
	if err != nil {
		log.Fatal(err)
	}
}

Set Since to exclude a revision and its ancestors, or provide Hashes to walk an exact sequence. Merges retains first-parent changes for merge commits, while PathFilter discards unrelated paths before changes are retained. The callback always runs serially in commit order, including when workers decode commits in parallel.

Walk changes across every ref

WalkChanges visits each reachable commit once in date order across all refs. The callback runs serially. Set Merges to include first-parent changes for merge commits.

err := repo.WalkChanges(history.ChangeOptions{
	Merges:  true,
	Workers: 4,
}, func(change history.Change) error {
	fmt.Printf("%s %s %s\n", change.Commit, change.NewOID, change.Path)
	return nil
})

Walk stored blobs

WalkBlobs visits each stored blob once, including objects supplied by Git alternates. Blob callbacks can run concurrently and must be safe for concurrent use.

count, err := repo.WalkBlobs(history.BlobOptions{
	Workers: 4,
	Limit: func(string) int64 {
		return 1024 * 1024
	},
}, func(blob history.Blob) error {
	fmt.Printf("%s %d\n", blob.OID, blob.Size)
	return nil
})

The returned count includes blobs excluded by Limit. Use Skip when the caller needs their object IDs.

Open options

Open uses the package's benchmarked storage defaults. OpenWithOptions exposes repository discovery, storage tuning, and an alternate-object filesystem for callers that need different behaviour.

License

MIT

Documentation

Overview

Package history walks the full commit history and blob store of a git repository using go-git, without invoking the git binary. It is the shared object-graph layer shared by secrets, git-spdx, and git-pkgs.

Index

• func CommitSubject(message string) string
• func ShallowGit(repo string) (bool, error)
• func WalkChangesGit(repo string, merges bool, visit func(Change) error) error
• type Blob
• type BlobOptions
• type Change
• type ChangeOptions
• type Commit
• type CommitOptions
• type OpenOptions
• type Repo
  • func Open(path string) (*Repo, error)
  • func OpenWithOptions(path string, opts OpenOptions) (*Repo, error)
  • func (r *Repo) Close() error
  • func (r *Repo) CommitHashes(opts CommitOptions) ([]plumbing.Hash, error)
  • func (r *Repo) CommonDir() string
  • func (r *Repo) GitDir() string
  • func (r *Repo) Repository() *gogit.Repository
  • func (r *Repo) Roots() ([]plumbing.Hash, error)
  • func (r *Repo) Shallow() (bool, error)
  • func (r *Repo) VisitCommitTrees(visit func(plumbing.Hash) error) error
  • func (r *Repo) WalkBlobs(opts BlobOptions, visit func(Blob) error) (total int, resultErr error)
  • func (r *Repo) WalkChanges(opts ChangeOptions, visit func(Change) error) error
  • func (r *Repo) WalkCommits(opts CommitOptions, visit func(Commit) error) error
  • func (r *Repo) WalkTreeEntries(hash plumbing.Hash, visit func(TreeEntry) error) (resultErr error)
• type Spool
  • func NewSpool(prefix string) (*Spool, error)
  • func (s *Spool) Close()
  • func (s *Spool) Ready() error
  • func (s *Spool) Replay(visit func(Change)) error
  • func (s *Spool) Write(c Change)
• type TreeEntry
• type Tuning
  • func DefaultTuning() Tuning

Functions

func CommitSubject

func CommitSubject(message string) string

CommitSubject returns the first paragraph of a commit message as a single line, matching git log --format=%s.

func ShallowGit

func ShallowGit(repo string) (bool, error)

ShallowGit reports whether the repository is a shallow clone by shelling out to git rev-parse.

func WalkChangesGit

func WalkChangesGit(repo string, merges bool, visit func(Change) error) error

WalkChangesGit produces the same Change stream as Repo.WalkChanges by shelling out to the git binary. It is also a benchmark baseline for the in-process walker.

Types

type Blob

type Blob struct {
	OID  string
	Size int64
	Data []byte
}

Blob is one decoded object from the repository's blob store.

type BlobOptions

type BlobOptions struct {
	// Workers is the number of concurrent object readers.
	Workers int
	// Limit returns the maximum size to read for a blob; larger blobs are
	// reported to Skip and not visited. Nil means no limit.
	Limit func(oid string) int64
	// Skip is called for blobs excluded by Limit. Nil is a no-op.
	Skip func(oid string)
}

BlobOptions controls a blob walk.

type Change

type Change struct {
	Commit, Date, Subject string
	OldOID, NewOID, Path  string
	OldMode, NewMode      string
	Merge                 bool
}

Change is one file-level modification in a single commit, matching a raw entry from git log --raw. OIDs and modes are hex/octal strings so callers can round-trip them through the git-binary backend unchanged.

type ChangeOptions

type ChangeOptions struct {
	// Merges includes merge commits in the output.
	Merges bool
	// Workers is the number of concurrent commit readers and diff workers.
	Workers int
}

ChangeOptions controls a change walk.

type Commit

type Commit struct {
	Object  *object.Commit
	Changes []Change
}

Commit is one commit and its first-parent file changes.

type CommitOptions

type CommitOptions struct {
	// Hashes is an exact commit sequence. Ref and Since are ignored when set.
	Hashes []plumbing.Hash
	// Ref is the revision to walk. An empty value uses HEAD.
	Ref string
	// Since excludes this revision and its ancestors.
	Since string
	// Merges includes first-parent changes for merge commits.
	Merges bool
	// Workers is the number of concurrent commit readers and diff workers.
	Workers int
	// PathFilter retains changes whose path returns true.
	PathFilter func(string) bool
}

CommitOptions controls an ordered commit walk.

type OpenOptions

type OpenOptions struct {
	Tuning       Tuning
	DetectDotGit bool
	AlternatesFS billy.Filesystem
}

OpenOptions controls repository discovery and storage.

type Repo

type Repo struct {
	// contains filtered or unexported fields
}

Repo wraps an open go-git repository configured for history walking.

func Open

func Open(path string) (*Repo, error)

Open opens the repository at path with the default tuning.

func OpenWithOptions

func OpenWithOptions(path string, opts OpenOptions) (*Repo, error)

OpenWithOptions opens the repository at path with repository discovery and alternate-object storage options.

func (*Repo) Close

func (r *Repo) Close() error

Close releases the underlying go-git repository.

func (*Repo) CommitHashes

func (r *Repo) CommitHashes(opts CommitOptions) ([]plumbing.Hash, error)

CommitHashes returns commits oldest-first without reading their trees.

func (*Repo) CommonDir

func (r *Repo) CommonDir() string

CommonDir returns the Git directory shared by linked worktrees.

func (*Repo) GitDir

func (r *Repo) GitDir() string

GitDir returns the worktree-specific Git directory.

func (*Repo) Repository

func (r *Repo) Repository() *gogit.Repository

Repository returns the underlying go-git handle for callers that need direct storer access.

func (*Repo) Roots

func (r *Repo) Roots() ([]plumbing.Hash, error)

Roots returns every commit reachable from a ref, deduplicated.

func (*Repo) Shallow

func (r *Repo) Shallow() (bool, error)

Shallow reports whether the repository is a shallow clone.

func (*Repo) VisitCommitTrees

func (r *Repo) VisitCommitTrees(visit func(plumbing.Hash) error) error

VisitCommitTrees walks every commit reachable from the repository's refs and invokes visit with each commit's root tree hash. Commits beyond a shallow boundary are treated as having no parents.

func (*Repo) WalkBlobs

func (r *Repo) WalkBlobs(opts BlobOptions, visit func(Blob) error) (total int, resultErr error)

WalkBlobs decodes every blob object in the repository exactly once and invokes visit from opts.Workers goroutines concurrently. visit must be safe for concurrent use. The returned int is the total blob count including those excluded by opts.Limit.

func (*Repo) WalkChanges

func (r *Repo) WalkChanges(opts ChangeOptions, visit func(Change) error) error

WalkChanges visits every file-level change in date order across all refs. visit is called from a single goroutine in commit order regardless of opts.Workers.

func (*Repo) WalkCommits

func (r *Repo) WalkCommits(opts CommitOptions, visit func(Commit) error) error

WalkCommits visits commits oldest-first. The callback runs serially in the requested order, including for commits with no retained changes.

func (*Repo) WalkTreeEntries

func (r *Repo) WalkTreeEntries(hash plumbing.Hash, visit func(TreeEntry) error) (resultErr error)

WalkTreeEntries visits the immediate children of a tree without materializing an object.Tree.

type Spool

type Spool struct {
	// contains filtered or unexported fields
}

Spool buffers a Change stream to a temporary file so it can be replayed after a separate blob-scanning pass has built its index.

func NewSpool

func NewSpool(prefix string) (*Spool, error)

NewSpool creates a temporary spool file. Prefix is used in the temp file name for debugging.

func (*Spool) Close

func (s *Spool) Close()

Close removes the spool file.

func (*Spool) Ready

func (s *Spool) Ready() error

Ready flushes the spool and rewinds it for Replay.

func (*Spool) Replay

func (s *Spool) Replay(visit func(Change)) error

Replay visits every change written before Ready.

func (*Spool) Write

func (s *Spool) Write(c Change)

Write appends a change to the spool.

type TreeEntry

type TreeEntry struct {
	Name []byte
	Hash plumbing.Hash
	Mode filemode.FileMode
}

TreeEntry is one immediate child of a Git tree. Name is valid only until the visitor returns and must be copied if retained.

type Tuning

type Tuning struct {
	MemoryIndex   bool
	Mmap          bool
	CacheBytes    uint64
	CacheShards   int
	KlauspostZlib bool
}

Tuning controls go-git storage behaviour. The zero value matches go-git defaults.

func DefaultTuning

func DefaultTuning() Tuning

DefaultTuning returns the settings git-spdx converged on after benchmarking.