cloner

package
v0.0.0-...-68956d0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package cloner — read-only audit for clone manifests.

PlanCloneAudit parses a structured source file (CSV/JSON/text) and, for every record, computes the `git clone`/`git pull` command that `gitmap clone` would run — without actually running it. The result is a diff-style report:

  • clone path (url, branch, strategy) target missing ~ pull path (url) existing git repo = skip path (url) cache fingerprint matches ! invalid path no clone URL on the record ? conflict path target exists but is not a git repo

Audit never touches the network and never writes anything outside stdout. It honors the same branch-selection strategy as the live clone path (pickCloneStrategy) so what you see is exactly what `gitmap clone` would execute.

Package cloner — clone-cache.go

Idempotent clone cache. Stores per-record fingerprints under <targetDir>/.gitmap/clone-cache.json so repeated `gitmap clone` runs can detect repos that are already cloned at the desired URL/branch and skip them when both the local HEAD and the remote tip match the cached values.

Cache schema (versioned for forward compatibility): 

{
  "version": 1,
  "entries": {
    "<relativePath>": {
      "url":       "https://github.com/owner/repo.git",
      "branch":    "main",
      "headSHA":   "abc123...",
      "remoteSHA": "abc123...",
      "updatedAt": "2025-04-21T10:00:00Z"
    }
  }
}

Package cloner re-clones repos from structured files.

Package cloner — concurrent.go

Bounded worker pool for parallel clone/pull execution. Wired in by cloneAll() when CloneOptions.MaxConcurrency > 1; the sequential path stays unchanged for the default invocation.

Each worker pulls one record at a time from a buffered job channel, performs the clone-or-pull through the same cloneOrPullOne path used by the sequential runner, and reports outcomes back through a single result channel. The collector goroutine serializes Progress / cache / summary updates so the public Progress + CloneCache types only need the lightweight mutex they already carry.

Concurrency invariants:

  • Progress.Begin / .Done / .Skip / .Fail are guarded internally by Progress.mu (see progress.go), so concurrent calls cannot interleave a half-written stderr line.
  • CloneCache.Record is guarded by CloneCache.mu (see cache.go).
  • Order of progress lines matches completion order, NOT input order. Manifest rows are still cloned into their recorded RelativePath so the on-disk hierarchy is unaffected.

Package cloner — pulldiag.go provides pull diagnosis and file-lock remediation.

Package cloner — runners.go

Dispatcher and the sequential runner, split out of cloner.go to keep each file focused (cloner.go = entry points + parsing, runners.go = orchestration, concurrent.go = worker-pool path). The dispatcher (cloneAll) is the single point that decides between sequential and parallel execution and the only writer of the "parallel enabled" header line.

Package cloner re-clones repos from structured files.

Package cloner — branch selection strategy.

pickCloneStrategy decides whether `git clone` should be invoked with an explicit `-b <branch>` flag or without it (letting the remote's default HEAD decide). The decision is driven by ScanRecord.BranchSource so that untrustworthy values (literal "HEAD", a detached SHA, or "unknown") never reach the git command line and produce "Remote branch not found" errors.

Package cloner — summary.go

Result-shape helpers split out of cloner.go to keep that file focused on entry points + parsing. Both runners (sequential in runners.go and parallel in concurrent.go) share these helpers, so a single source of truth for "what counts as success / skipped" prevents the two paths from drifting.

Index

Constants

View Source 
const CloneCacheVersion = 1

CloneCacheVersion is the on-disk schema version.

Variables

This section is empty.

Functions

func CloneFromFile 

func CloneFromFile(sourcePath, targetDir string, safePull bool) (model.CloneSummary, error)

CloneFromFile reads a source file and clones all repos under targetDir.

func CloneFromFileQuiet 

func CloneFromFileQuiet(sourcePath, targetDir string, safePull bool) (model.CloneSummary, error)

CloneFromFileQuiet reads a source file and clones with suppressed progress.

func CloneFromFileWithOptions 

func CloneFromFileWithOptions(sourcePath, targetDir string, opts CloneOptions) (model.CloneSummary, error)

CloneFromFileWithOptions is the full-control entry point. The legacy helpers above are thin wrappers that fill in CloneOptions defaults.

func IsGitRepo 

func IsGitRepo(path string) bool

IsGitRepo checks whether the given path contains a .git directory.

func IsMissingRepo 

func IsMissingRepo(path string) bool

IsMissingRepo returns true when the path is not a valid git repository.

func SafePullOne 

func SafePullOne(rec model.ScanRecord, repoDir string) model.CloneResult

SafePullOne runs safe-pull on a single repo. Exported for use by the pull command.

Types

type AuditAction 

type AuditAction string

AuditAction labels the planned action for a single record. 

const (
	AuditActionClone    AuditAction = "clone"    // target missing — fresh clone
	AuditActionPull     AuditAction = "pull"     // target is a git repo — would safe-pull
	AuditActionCached   AuditAction = "cached"   // cache fingerprint matches dest
	AuditActionInvalid  AuditAction = "invalid"  // record has no clone URL
	AuditActionConflict AuditAction = "conflict" // target exists but is not a git repo
)

type AuditEntry 

type AuditEntry struct {
	Action       AuditAction
	RelativePath string
	URL          string
	Branch       string
	UseBranch    bool
	Reason       string // strategy.reason or conflict explanation
	Command      string // exact git command line that would be run, "" for invalid/cached
}

AuditEntry is one planned-vs-actual row in the audit report.

type BatchProgress 

type BatchProgress struct {
	// contains filtered or unexported fields
}

BatchProgress tracks progress for any batch operation (pull, exec, status).

func NewBatchProgress 

func NewBatchProgress(total int, operation string, quiet bool) *BatchProgress

NewBatchProgress creates a progress tracker for a named operation.

func (*BatchProgress) BeginItem 

func (p *BatchProgress) BeginItem(name string)

BeginItem prints progress for starting an item.

func (*BatchProgress) ExitCodeForBatch 

func (p *BatchProgress) ExitCodeForBatch() int

ExitCodeForBatch returns 0 if all items succeeded, or the partial-failure exit code if any items failed. Use with os.Exit() in the calling command.

func (*BatchProgress) Fail 

func (p *BatchProgress) Fail()

Fail marks an item as failed.

func (*BatchProgress) FailWithError 

func (p *BatchProgress) FailWithError(name, errMsg string)

FailWithError marks an item as failed and records the error detail.

func (*BatchProgress) Failed 

func (p *BatchProgress) Failed() int

Failed returns the failure count.

func (*BatchProgress) Failures 

func (p *BatchProgress) Failures() []FailureRecord

Failures returns all recorded failure details.

func (*BatchProgress) HasFailures 

func (p *BatchProgress) HasFailures() bool

HasFailures returns true if any items failed.

func (*BatchProgress) PrintFailureReport 

func (p *BatchProgress) PrintFailureReport()

PrintFailureReport outputs a detailed list of failed items after a batch operation. Call after PrintSummary when HasFailures() is true.

func (*BatchProgress) PrintSummary 

func (p *BatchProgress) PrintSummary()

PrintSummary prints the final summary.

func (*BatchProgress) SetStopOnFail 

func (p *BatchProgress) SetStopOnFail(v bool)

SetStopOnFail enables early termination after the first failure.

func (*BatchProgress) Skip 

func (p *BatchProgress) Skip()

Skip marks an item as skipped (e.g., missing directory).

func (*BatchProgress) Skipped 

func (p *BatchProgress) Skipped() int

Skipped returns the skip count.

func (*BatchProgress) Stopped 

func (p *BatchProgress) Stopped() bool

Stopped returns true if the batch was halted due to --stop-on-fail.

func (*BatchProgress) Succeed 

func (p *BatchProgress) Succeed()

Succeed marks an item as successful.

func (*BatchProgress) Succeeded 

func (p *BatchProgress) Succeeded() int

Succeeded returns the success count.

type CloneAuditReport 

type CloneAuditReport struct {
	Source  string
	Target  string
	Entries []AuditEntry
	Counts  map[AuditAction]int
}

CloneAuditReport is the full per-record plan plus aggregate counters.

func PlanCloneAudit 

func PlanCloneAudit(sourcePath, targetDir string) (*CloneAuditReport, error)

PlanCloneAudit loads the source manifest and produces a non-executing plan for every record. Returns an error only when the source file cannot be loaded; per-record problems are encoded as audit entries.

func (*CloneAuditReport) Print 

func (r *CloneAuditReport) Print(w io.Writer) error

Print renders the report to w in the diff-style layout described in the package doc. Returns the number of bytes written or an error if the underlying writer fails — never silent.

type CloneCache 

type CloneCache struct {
	Version int                        `json:"version"`
	Entries map[string]CloneCacheEntry `json:"entries"`
	// contains filtered or unexported fields
}

CloneCache holds all entries keyed by record.RelativePath.

func LoadCloneCache 

func LoadCloneCache(targetDir string) *CloneCache

LoadCloneCache reads the cache from <targetDir>/.gitmap/clone-cache.json. A missing file or unparseable contents yields a fresh empty cache; this is intentional so the cache never blocks a clone run.

func (*CloneCache) IsUpToDate 

func (c *CloneCache) IsUpToDate(rec model.ScanRecord, dest string) bool

IsUpToDate reports whether the repo at dest matches the cached fingerprint for rec and the remote tip still matches what we cached. When true, the caller can safely skip the clone/pull.

All git lookups here are read-only and bounded; on any error we report "not up to date" so the caller falls back to the normal clone/pull path.

func (*CloneCache) Record 

func (c *CloneCache) Record(rec model.ScanRecord, dest string)

Record stores or updates the cache entry for rec at dest. Best-effort: failures to introspect the repo are logged via skip and do not abort.

func (*CloneCache) Save 

func (c *CloneCache) Save() error

Save persists the cache to disk. Errors are non-fatal — a failure to write the cache must never abort or corrupt a clone run.

type CloneCacheEntry 

type CloneCacheEntry struct {
	URL       string    `json:"url"`
	Branch    string    `json:"branch"`
	HeadSHA   string    `json:"headSHA"`
	RemoteSHA string    `json:"remoteSHA"`
	UpdatedAt time.Time `json:"updatedAt"`
}

CloneCacheEntry is a single per-repo cache record.

type CloneOptions 

type CloneOptions struct {
	SafePull       bool
	Quiet          bool
	MaxConcurrency int
	// DefaultBranch is the fallback branch name handed to `git clone -b`
	// for any record whose recorded (Branch, BranchSource) would
	// otherwise leave the cloner with no usable branch (empty Branch,
	// detached HEAD, unknown source, etc.). Empty preserves the legacy
	// "let the remote's default HEAD decide" behavior. Plumbed in by
	// the CLI from `--default-branch` (constants.FlagScanDefaultBranch),
	// so the wording and semantics match `gitmap scan --default-branch`.
	DefaultBranch string
}

CloneOptions tunes a CloneFromFileWithOptions run. The zero value keeps the historical behavior: sequential, progress-on-stderr.

MaxConcurrency:

  • <= 1 → sequential (one repo at a time, original ordering).
  • > 1 → bounded worker pool (see concurrent.go). Per-repo paths come from each ScanRecord.RelativePath unchanged, so the on-disk nested folder hierarchy is preserved regardless of worker count.

Quiet suppresses per-repo progress lines but keeps the final summary (matches the legacy CloneFromFileQuiet behavior).

type FailureRecord 

type FailureRecord struct {
	Name  string
	Error string
}

FailureRecord stores details about a single failed batch item.

type Progress 

type Progress struct {
	// contains filtered or unexported fields
}

Progress tracks clone operation progress.

Thread-safety: all counter mutations and stderr writes go through mu so concurrent workers in the parallel runner (concurrent.go) cannot interleave half-written status lines or corrupt the running totals. The sequential runner pays only the cost of an uncontended mutex.

func NewProgress 

func NewProgress(total int, quiet bool) *Progress

NewProgress creates a progress tracker.

func (*Progress) Begin 

func (p *Progress) Begin(name string)

Begin prints the starting line for a repo.

func (*Progress) Done 

func (p *Progress) Done(result model.CloneResult, pulled bool)

Done marks a repo as successfully completed.

func (*Progress) Fail 

func (p *Progress) Fail(result model.CloneResult)

Fail marks a repo as failed.

func (*Progress) PrintSummary 

func (p *Progress) PrintSummary()

PrintSummary prints the final summary line.

func (*Progress) Skip 

func (p *Progress) Skip(result model.CloneResult)

Skip marks a repo as skipped because it was already up to date.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.