gitsync

README

git-sync

git-sync mirrors refs from a source remote to a target remote without creating a local checkout. It uses an in-memory go-git object store and talks smart HTTP directly:

• info/refs ref advertisement for source and target
• upload-pack fetch from source with target tip hashes advertised as have
• receive-pack push to target with explicit ref update commands and a streamed packfile

That keeps the target side incremental without fetching target objects into the local process first.

Why This Exists

Mirroring Git data between remotes usually means a local mirror clone followed by a mirror push. That's fine for small repos but turns a remote-to-remote operation into a local storage problem at scale, and shell glue around git fetch / git push tends to skip planning and structured output.

git-sync fills that gap. It streams source packs directly into target receive-pack when it can, plans every action before pushing, and emits typed JSON for automation.

For when to use it (and when not), see docs/architecture.md.

Commands

The main commands are:

• git-sync sync: mirror source refs into the target
• git-sync replicate: overwrite target refs to match source via relay, and fail rather than materialize locally

sync automatically bootstraps an empty target, so the same command covers initial seeding and ongoing sync. To preview what would happen without pushing, run git-sync plan — it takes the same flags as sync, and --mode replicate previews a replicate run.

For command examples, JSON output, auth, protocol flags, and advanced command notes, see docs/usage.md.

Library API

git-sync is also a Go library. Use entire.io/entire/git-sync for the stable embedding surface (Probe, Plan, Sync, Replicate, typed results, auth and HTTP injection). entire.io/entire/git-sync/unstable exposes advanced controls (Bootstrap, Fetch, batching knobs, heap measurement) and is not stable.

Installation

Homebrew (macOS, Linux)

brew tap entireio/tap
brew install --cask git-sync

go install

Requires Go 1.26 or newer.

go install entire.io/entire/git-sync/cmd/git-sync@latest

This drops a git-sync binary into $(go env GOPATH)/bin. Make sure that directory is on your PATH.

From source

git clone https://github.com/entireio/git-sync.git
cd git-sync
go build -o git-sync ./cmd/git-sync

Quick Start

git-sync sync \
  --source-token "$GITSYNC_SOURCE_TOKEN" \
  --target-token "$GITSYNC_TARGET_TOKEN" \
  https://github.com/source-org/source-repo.git \
  https://github.com/target-org/target-repo.git

https://github.com/user-attachments/assets/60adb873-4032-4ab7-b236-24d038e04681

Sync Behavior

sync picks the bootstrap relay path automatically when the target is empty. For non-empty targets, safe fast-forward updates also use a relay path that streams the source pack directly into target receive-pack without local materialization. Anything not relay-eligible (force, prune, deletes, tag retargets) falls back to a materialized path bounded by --materialized-max-objects.

For branch filtering, ref mapping, tags, pruning, protocol selection, JSON output, and auth details, see docs/usage.md.

Testing

Default suite:

env GOCACHE=/tmp/go-build go test ./...

Extended and environment-specific test instructions are in docs/testing.md.

Documentation

• docs/usage.md — CLI commands, examples, sync behavior, JSON output, auth, protocol notes
• docs/architecture.md — product rationale, package layout, operation modes vs transfer modes, memory model
• docs/protocol.md — smart HTTP, pkt-line, capability negotiation, sideband, relay framing
• docs/testing.md — test suites and integration coverage

FAQ

Does it sync complete Git history or only perform a shallow/partial sync?

git-sync syncs the complete Git object history required for the selected refs. It does not create a shallow clone. Some planning paths may use filtered fetches, but the target receives the full objects needed for valid refs.

Is it just refs, or objects as well?

Objects as well. Refs are what git-sync plans and updates, but it also transfers the commits, trees, blobs, and tags needed for those refs to exist on the target.

Is it bidirectional?

No. git-sync is one-way: source remote to target remote. To go the other way you'd run a second invocation with the endpoints swapped.

Does it support create, update, and delete actions?

Yes. It supports creating refs, updating refs, force updates with --force, and deleting managed refs with --prune. replicate can overwrite target refs, but it is relay-only and more restrictive than sync.

How does it scale?

git-sync has two transfer paths:

• Relay — pack data streams from source upload-pack directly into target receive-pack. The local process holds no object graph, so memory stays bounded regardless of repo size. Used when the target supports relay.
• Materialized fallback — when relay isn't available, git-sync fetches the needed objects into an in-memory go-git store, plans, then encodes and pushes a packfile. Memory scales with the diff being pushed and is guarded by an explicit object-count limit. Bootstrap can batch large initial syncs to keep this bounded.

Planning itself is cheap: ref-only round-trips, plus a filter tree:0 fetch for ancestry checks when the source advertises filter support.

How long does it take for a medium-sized repo?

It depends on repository size, network speed, and whether the relay path is available. As a rule of thumb, the relay path is bounded by source pack generation + network transfer + target receive-pack time — roughly the time of a git clone from the source plus a git push of the same pack. The materialized fallback adds local memory work for objects that need inspection.

For concrete numbers on your own setup, run the included benchmark tool against a representative repo; see docs/testing.md.

Does it support SSH?

No. git-sync supports smart HTTP/HTTPS only.

Does it run as a daemon or watch for changes?

No. git-sync is a one-shot CLI/library operation. To sync on a schedule or in response to events, run it from cron, CI, a worker, or another service.

Contributing

See CONTRIBUTING.md, SECURITY.md, and CODE_OF_CONDUCT.md.

Documentation

Overview

Package gitsync provides the public orchestration API for git-sync.

The public surface is intentionally narrower than the internal engine: callers express sync intent through typed probe, plan, and sync requests, while relay selection, batching, and fallback strategy remain internal.

The package is designed for embedders such as queue workers. Callers can:

• inject an HTTP client for transport, OTEL, proxy, TLS, and timeout control
• inject an auth provider that resolves source and target credentials
• inspect structured results for per-ref outcomes and aggregate counters

Current advanced engine tuning such as batch sizing, max pack thresholds, and heap measurement remains outside this stable public surface.

Index

• type Action
• type AuthProvider
• type BatchSummary
• type Client
  • func New(opts Options) *Client
  • func (c *Client) Plan(ctx context.Context, req PlanRequest) (PlanResult, error)
  • func (c *Client) Probe(ctx context.Context, req ProbeRequest) (ProbeResult, error)
  • func (c *Client) Replicate(ctx context.Context, req SyncRequest) (SyncResult, error)
  • func (c *Client) Sync(ctx context.Context, req SyncRequest) (SyncResult, error)
• type Endpoint
• type EndpointAuth
• type EndpointRole
• type ExecutionSummary
• type Measurement
• type OperationMode
• type Options
• type PlanRequest
  • func (r PlanRequest) Validate() error
• type PlanResult
• type ProbeRequest
  • func (r ProbeRequest) Validate() error
• type ProbeResult
• type ProtocolMode
• type RefInfo
• type RefKind
• type RefMapping
• type RefPlan
• type RefResult
• type RefScope
• type ServiceStats
• type StaticAuthProvider
  • func (p StaticAuthProvider) AuthFor(_ context.Context, _ Endpoint, role EndpointRole) (EndpointAuth, error)
• type Stats
• type SyncCounts
• type SyncPolicy
• type SyncRequest
  • func (r SyncRequest) Validate() error
• type SyncResult

Examples

• Client.Sync

Types

type Action

type Action = internalbridge.Action

const (
	ActionCreate Action = internalbridge.ActionCreate
	ActionUpdate Action = internalbridge.ActionUpdate
	ActionDelete Action = internalbridge.ActionDelete
	ActionSkip   Action = internalbridge.ActionSkip
	ActionBlock  Action = internalbridge.ActionBlock
)

type AuthProvider

type AuthProvider interface {
	AuthFor(ctx context.Context, endpoint Endpoint, role EndpointRole) (EndpointAuth, error)
}

AuthProvider resolves auth for a request endpoint.

type BatchSummary

type BatchSummary = internalbridge.BatchSummary

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client provides the public orchestration API for git-sync.

func New

func New(opts Options) *Client

New constructs a new Client.

func (*Client) Plan

func (c *Client) Plan(ctx context.Context, req PlanRequest) (PlanResult, error)

Plan computes ref actions without pushing.

func (*Client) Probe

func (c *Client) Probe(ctx context.Context, req ProbeRequest) (ProbeResult, error)

Probe inspects a source remote and optional target remote.

func (*Client) Replicate

func (c *Client) Replicate(ctx context.Context, req SyncRequest) (SyncResult, error)

Replicate executes source-authoritative relay-only replication between two remotes.

func (*Client) Sync

func (c *Client) Sync(ctx context.Context, req SyncRequest) (SyncResult, error)

Sync executes a sync between two remotes.

Example

client := gitsync.New(gitsync.Options{
	HTTPClient: &http.Client{},
	Auth: gitsync.StaticAuthProvider{
		Source: gitsync.EndpointAuth{Token: "source-token"},
		Target: gitsync.EndpointAuth{Token: "target-token"},
	},
})

if _, err := client.Sync(context.Background(), gitsync.SyncRequest{
	Source: gitsync.Endpoint{URL: "https://github.example/source/repo.git"},
	Target: gitsync.Endpoint{URL: "https://git.example/target/repo.git"},
	Scope:  gitsync.RefScope{Branches: []string{"main"}},
	Policy: gitsync.SyncPolicy{
		IncludeTags: true,
		Protocol:    gitsync.ProtocolAuto,
	},
}); err != nil {
	return // network error expected in example environment
}

type Endpoint

type Endpoint struct {
	URL string `json:"url"`

	// FollowInfoRefsRedirect, when true, rewrites this endpoint's
	// effective host to the final URL returned by /info/refs after
	// HTTP redirects. Subsequent git RPCs (git-upload-pack,
	// git-receive-pack) then target the redirected host directly.
	// Matches vanilla git's smart-HTTP behaviour for discovery-aware
	// servers that 307 /info/refs to a hosting replica.
	FollowInfoRefsRedirect bool `json:"followInfoRefsRedirect,omitempty"`
}

Endpoint identifies a remote Git endpoint.

type EndpointAuth

type EndpointAuth struct {
	Username      string `json:"username"`
	Token         string `json:"token"`
	BearerToken   string `json:"bearerToken"`
	SkipTLSVerify bool   `json:"skipTlsVerify"`
}

EndpointAuth carries explicit per-request auth and TLS settings. It is resolved through an AuthProvider rather than embedded in Endpoint so endpoint identity does not also become the public auth-precedence boundary.

type EndpointRole

type EndpointRole string

EndpointRole identifies whether auth is being resolved for the source or target.

const (
	SourceRole EndpointRole = "source"
	TargetRole EndpointRole = "target"
)

type ExecutionSummary

type ExecutionSummary = internalbridge.ExecutionSummary

type Measurement

type Measurement = internalbridge.Measurement

type OperationMode

type OperationMode string

OperationMode controls high-level sync semantics.

const (
	ModeSync      OperationMode = "sync"
	ModeReplicate OperationMode = "replicate"
)

type Options

type Options struct {
	HTTPClient *http.Client
	Auth       AuthProvider
}

Options configures a Client. It is intentionally small in the first public cut.

type PlanRequest

type PlanRequest struct {
	Source       Endpoint   `json:"source"`
	Target       Endpoint   `json:"target"`
	Scope        RefScope   `json:"scope"`
	Policy       SyncPolicy `json:"policy"`
	CollectStats bool       `json:"collectStats"`
}

PlanRequest computes ref actions without pushing.

func (PlanRequest) Validate

func (r PlanRequest) Validate() error

type PlanResult

type PlanResult = internalbridge.PlanResult

type ProbeRequest

type ProbeRequest struct {
	Source       Endpoint     `json:"source"`
	Target       *Endpoint    `json:"target"`
	IncludeTags  bool         `json:"includeTags"`
	Protocol     ProtocolMode `json:"protocol"`
	CollectStats bool         `json:"collectStats"`
}

ProbeRequest inspects source refs and optional target capabilities.

func (ProbeRequest) Validate

func (r ProbeRequest) Validate() error

type ProbeResult

type ProbeResult = internalbridge.ProbeResult

type ProtocolMode

type ProtocolMode string

ProtocolMode controls source-side protocol negotiation.

const (
	ProtocolAuto ProtocolMode = "auto"
	ProtocolV1   ProtocolMode = "v1"
	ProtocolV2   ProtocolMode = "v2"
)

type RefInfo

type RefInfo = internalbridge.RefInfo

type RefKind

type RefKind = internalbridge.RefKind

const (
	RefKindBranch RefKind = internalbridge.RefKindBranch
	RefKindTag    RefKind = internalbridge.RefKindTag
)

type RefMapping

type RefMapping struct {
	Source string `json:"source"`
	Target string `json:"target"`
}

RefMapping is an explicit source-to-target ref mapping.

type RefPlan

type RefPlan = internalbridge.RefPlan

type RefResult

type RefResult = internalbridge.RefResult

type RefScope

type RefScope struct {
	Branches []string     `json:"branches"`
	Mappings []RefMapping `json:"mappings"`
}

RefScope constrains which refs a request manages.

type ServiceStats

type ServiceStats = internalbridge.ServiceStats

type StaticAuthProvider

type StaticAuthProvider struct {
	Source EndpointAuth `json:"source"`
	Target EndpointAuth `json:"target"`
}

StaticAuthProvider returns fixed source and target auth values.

func (StaticAuthProvider) AuthFor

func (p StaticAuthProvider) AuthFor(_ context.Context, _ Endpoint, role EndpointRole) (EndpointAuth, error)

AuthFor implements AuthProvider.

type Stats

type Stats = internalbridge.Stats

type SyncCounts

type SyncCounts = internalbridge.SyncCounts

type SyncPolicy

type SyncPolicy struct {
	Mode        OperationMode `json:"mode"`
	IncludeTags bool          `json:"includeTags"`
	Force       bool          `json:"force"`
	Prune       bool          `json:"prune"`
	Protocol    ProtocolMode  `json:"protocol"`
}

SyncPolicy controls high-level sync behavior.

type SyncRequest

type SyncRequest struct {
	Source       Endpoint   `json:"source"`
	Target       Endpoint   `json:"target"`
	Scope        RefScope   `json:"scope"`
	Policy       SyncPolicy `json:"policy"`
	CollectStats bool       `json:"collectStats"`
}

SyncRequest executes a sync between two remotes.

func (SyncRequest) Validate

func (r SyncRequest) Validate() error

type SyncResult

type SyncResult = internalbridge.SyncResult