lfscache

README

lfs-cache

lfs-cache is a caching proxy for Git LFS servers.

The Usage section below covers a single repository. For one proxy in front of many private GitHub repositories, with no username or password to type, go straight to the walkthrough.

Usage

Docker

$ docker run --name lfscache --rm -d -v /my/cache/dir/lfs:/lfs saracen/lfscache:latest --url github.com/org/repo.git/info/lfs --http-addr :80  --directory /lfs

Binary

Download the correct binary for your system.

$ ./lfscache --url github.com/org/repo.git/info/lfs --directory /my/cache/dir/lfs --http-addr=:9876

--directory specifies the cache directory. The layout is the same used by the Git LFS client, so it might be a good idea to copy over your .git/lfs/objects directory to preload the cache (cp -r .git/lfs/objects /my/cache/dir/lfs). The tmp and incomplete directories do not need to be copied over.

Now you need to have your Git LFS client point to the proxy. There are several ways to do this. The easiest method is changing the lfs url that will be used in your local git config:

# note that repo.git/info/lfs is not required
git config lfs.url http://localhost:9876/

# you can confirm the Endpoint that will be used by running
git lfs env | grep Endpoint

Walkthrough: many private GitHub repositories, one proxy

This is the setup the scripts/ directory exists for: one proxy instance in front of a whole organisation, and clients that authenticate with a token fetched over SSH instead of typing a username and password. The sections after it explain each piece on its own.

The template and header support below is specific to this fork, so the published saracen/lfscache image does not have it — build from source:

go build -o lfscache .

1. Run the proxy

./lfscache --url "https://github.com/{}.git/info/lfs" --directory /var/cache/lfs --http-addr :8080

The {} is filled in per request, so this one instance serves every repository in the organisation, each with its own subdirectory under /var/cache/lfs.

2. Point a repository at it

From inside a clone (on Windows, from Git Bash):

/path/to/lfscache/scripts/setup-lfscache.sh --url http://lfscache.local:8080

It works out the repository and the SSH host from the origin remote, writes the git config described below, and finishes by actually fetching a token so that a broken setup fails here rather than on your next git push:

lfs.url        http://lfscache.local:8080
X-Repo         organization/repo1
ssh remote     git@github.com

checking that a token can be fetched... ok

If your SSH key does not use one of ssh's default names (id_ed25519, id_rsa, ...), it is never offered automatically and this step fails with Permission denied (publickey). Point the helper at the key:

scripts/setup-lfscache.sh --url http://lfscache.local:8080 --ssh-key ~/.ssh/my_work_key

3. Use git normally

Nothing about day to day work changes — no username or password prompt, and no token to paste anywhere:

git lfs pull
git push

To confirm what is in effect:

git lfs env | grep Endpoint

Run setup-lfscache.sh once per clone. --unset reverts it.

When something goes wrong

Symptom	Cause
X-Repo header is required (400)	The endpoint is templated but the client is not sending the header. Check git config --get-all http.<url>.extraHeader.
Permission denied (publickey)	SSH cannot authenticate to GitHub. Re-run with --ssh-key, or check ssh -T git@github.com.
Git credentials for http://... not found	The credential helper produced nothing. Run it directly to see why: sh scripts/git-credential-lfscache get </dev/null.
Credentials that are not yours	Another credential helper (the Windows credential manager, say) answered first. setup-lfscache.sh resets the helper list for the proxy url; if you configured git by hand, add an empty credential.<url>.helper entry before yours.

Multiple repositories (URL templates)

The --url option accepts a {} placeholder. When present, the repository named by each request's X-Repo header is substituted into it, so a single instance can proxy many repositories:

$ ./lfscache --url "https://github.com/{}.git/info/lfs" --directory /my/cache/dir/lfs --http-addr=:9876

Point each repository's Git LFS client at the proxy and name the repository in the header:

# X-Repo: organization/repo1 on a request to http://localhost:9876/objects/batch
# is proxied to https://github.com/organization/repo1.git/info/lfs/objects/batch
git config lfs.url http://localhost:9876
git config --add http.http://localhost:9876.extraHeader "X-Repo: organization/repo1"

Requests without the header are refused with a 400, since there would be nothing to substitute. The header only ever names a repository: it cannot point the proxy at a different host, or escape the cache directory.

If no scheme is given, https is assumed (--url "github.com/{}.git/info/lfs").

In template mode each repository gets its own subdirectory in the cache. With --directory /my/cache/dir/lfs, objects for organization/repo1 are stored under:

/my/cache/dir/lfs/organization/repo1/objects/...

So to preload a repository's cache, copy its .git/lfs/objects into <directory>/<org>/<repo>/objects.

Authenticating with git-lfs-authenticate

Git LFS authenticates to an http endpoint with a username and password, which it sends as basic authorization. GitHub, however, hands out LFS tokens over SSH via git-lfs-authenticate and expects them back as Authorization: RemoteAuth <token>, so a token cannot simply be used as a password.

Sending X-ToRemoteAuth: true makes the proxy convert the two: the password of the incoming basic authorization is taken as a token and forwarded upstream as RemoteAuth <token>. The username is ignored. Requests without credentials are passed through untouched, so the upstream can still ask for them.

Together with X-Repo this means a client only needs a credential helper that prints a token as the password, which is what scripts/setup-lfscache.sh from the walkthrough installs. What it writes, for the proxy url only, so that other remotes keep their usual credential helpers:

Config	Why
lfs.url	Send Git LFS traffic to the proxy.
lfs.<url>.access=basic	Send credentials right away, instead of waiting to be refused once.
credential.<url>.helper	An empty entry to reset the helper list, then scripts/git-credential-lfscache.
http.<url>.extraHeader	X-Repo and X-ToRemoteAuth, added twice with git config --add.
lfscache.*	The SSH host, repository and optional key the helper reads.

The helper runs ssh <host> git-lfs-authenticate <repo> <operation>, pulls the token out of the response and prints it as the password. git-lfs applies http.<url>.extraHeader to object transfers as well as to the batch API, so X-Repo reaches the proxy on downloads too, and objects are cached under the repository they belong to.

By default an upload token is requested, which grants read and write; the helper falls back to download if write access is refused. --operation download only ever asks for a read-only token.

Tokens are short lived, so the helper only answers get and never stores anything: git-lfs reuses a token for the run it is in, and each new invocation fetches a fresh one over SSH.