mirror

README

mise-mirror

mise-mirror mirrors tool binaries listed in mise lock files (mise.lock) to a local directory or an Amazon S3 bucket.

Combined with mise's url_replacements setting, you can install tools from your own mirror instead of upstream servers (GitHub Releases, nodejs.org, etc.).

Install

$ go install github.com/fujiwara/mise-mirror/cmd/mise-mirror@latest

Or download a binary from Releases.

Usage

Usage: mise-mirror <destination> [flags]

Mirror tool binaries listed in mise lock files to a local directory or S3.

Arguments:
  <destination>    Destination directory or S3 URL (e.g. /path/to/dir,
                   s3://bucket/prefix/)

Flags:
  -h, --help                       Show context-sensitive help.
  -l, --lock-file=mise.lock,...    Path to mise lock file. Can be specified
                                   multiple times ($MISE_MIRROR_LOCK_FILE).
  -p, --platform=PLATFORM,...      Mirror only the specified platforms
                                   (e.g. linux-x64). Can be specified
                                   multiple times. Default: all platforms
                                   ($MISE_MIRROR_PLATFORM).
      --concurrency=4              Number of concurrent downloads
                                   ($MISE_MIRROR_CONCURRENCY).
      --force                      Overwrite files that already exist in the
                                   destination ($MISE_MIRROR_FORCE).
      --dry-run                    Show what would be mirrored without
                                   downloading ($MISE_MIRROR_DRY_RUN).
      --debug                      Enable debug logging ($MISE_MIRROR_DEBUG).
      --mirror-url=STRING          Base URL where the destination is served
                                   (e.g. https://mirror.example.com/prefix/).
                                   If specified, print url_replacements settings
                                   for mise after mirroring ($MISE_MIRROR_URL).
  -v, --version                    Show version and exit.

Examples

Mirror to a local directory:

$ mise-mirror --lock-file mise.lock --lock-file mise.local.lock /path/to/dir

Mirror to S3 (credentials are loaded by the AWS SDK default chain):

$ mise-mirror s3://my-bucket/mise-mirror/

S3 compatible storage can be used by setting the endpoint with the AWS SDK environment variables (AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL) or endpoint_url in the shared config file. When a custom endpoint is configured, path-style addressing (http://endpoint/bucket/key) is used. Otherwise (Amazon S3), the SDK default addressing is used.

$ AWS_ENDPOINT_URL_S3=http://localhost:7070 mise-mirror s3://my-bucket/mise-mirror/

Mirror only specific platforms:

$ mise-mirror --platform linux-x64 --platform linux-arm64 s3://my-bucket/mise-mirror/

GitHub token

If GITHUB_TOKEN (or GH_TOKEN) is set, it is sent as the Authorization header to github.com and api.github.com to avoid rate limits for unauthenticated requests. The token is not sent to other hosts, including redirect destinations of GitHub release downloads.

$ GITHUB_TOKEN=$(gh auth token) mise-mirror /path/to/dir

Layout of the mirror

Each file is stored as <destination>/<host>/<path> of the url in the lock file.

[tools.jq."platforms.linux-x64"]
checksum = "sha256:5942c9b0934e510ee61eb3e30273f1b3fe2590df93933a93d7c58b81d19c8ff5"
url = "https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64"

is mirrored to

<destination>/github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64

• Downloaded files are verified by checksum in the lock file (sha256 and sha512 are supported). Files with mismatched checksums are not stored.
• Files that already exist in the destination are skipped. Use --force to download them again.
• The same URL in multiple lock files is mirrored only once.
• Query strings in URLs are ignored.

Using the mirror from mise

Serve the destination over HTTP(S) (e.g. S3 + CloudFront, or any static web server), then configure url_replacements to route download URLs to the mirror.

[settings.url_replacements]
'regex:^https?://(github\.com|nodejs\.org)/' = "https://mise-mirror.example.com/$1/"

With --mirror-url, mise-mirror prints these settings to stdout after mirroring, covering all hosts that appear in the lock files.

$ mise-mirror --mirror-url https://mise-mirror.example.com/ s3://my-bucket/
[settings.url_replacements]
'regex:^https?://(github\.com|nodejs\.org)/' = "https://mise-mirror.example.com/$1/"

If you write the settings by hand, list the hosts that appear in your lock files. Do not route every https:// URL to the mirror, because mise also sends API requests (e.g. api.github.com) through url_replacements, and those are not mirrored.

To make sure that tools are installed from the lock file URLs, use mise install --locked (or MISE_LOCKED=1).

Limitations

• Only url in lock files is mirrored. url_api (GitHub API asset URLs) is not mirrored.
• Tools without platform URLs in the lock file (e.g. installed by plugins or package managers) are not mirrored.

Development

$ make test

The S3 integration test runs against an S3 compatible server only when MISE_MIRROR_TEST_S3_BUCKET is set. For example, with versitygw (posix backend):

$ mkdir -p /tmp/versitygw-data
$ ROOT_ACCESS_KEY=testkey ROOT_SECRET_KEY=testsecret versitygw --port 127.0.0.1:7070 posix /tmp/versitygw-data &
$ AWS_ACCESS_KEY_ID=testkey AWS_SECRET_ACCESS_KEY=testsecret AWS_REGION=us-east-1 \
  AWS_ENDPOINT_URL_S3=http://localhost:7070 MISE_MIRROR_TEST_S3_BUCKET=mise-mirror-test \
  go test -run TestS3Integration -v ./...

The bucket is created by the test if it does not exist.

LICENSE

MIT

Author

FUJIWARA Shunichiro

Documentation

Index

• Variables
• func GitHubTokenFromEnv() string
• func ObjectKey(rawURL string) (string, error)
• func Run(ctx context.Context) error
• func URLReplacements(artifacts []Artifact, mirrorURL string) (string, error)
• type Artifact
  • func FilterPlatforms(artifacts []Artifact, platforms []string) []Artifact
  • func LoadLockFiles(paths []string) ([]Artifact, error)
  • func ParseLockFile(r io.Reader) ([]Artifact, error)
• type CLI
• type LocalStorage
  • func NewLocalStorage(dir string) *LocalStorage
  • func (s *LocalStorage) Exists(ctx context.Context, key string) (bool, error)
  • func (s *LocalStorage) Location(key string) string
  • func (s *LocalStorage) Put(ctx context.Context, key string, src *os.File) error
• type Mirror
  • func (m *Mirror) Run(ctx context.Context, artifacts []Artifact) error
• type S3Client
• type S3Storage
  • func NewS3Storage(client S3Client, bucket, prefix string) *S3Storage
  • func (s *S3Storage) Exists(ctx context.Context, key string) (bool, error)
  • func (s *S3Storage) Location(key string) string
  • func (s *S3Storage) Put(ctx context.Context, key string, src *os.File) error
• type Storage
  • func NewStorage(ctx context.Context, dest string) (Storage, error)

Variables

var Version = "v0.0.1"

Functions

func GitHubTokenFromEnv

func GitHubTokenFromEnv() string

GitHubTokenFromEnv returns a GitHub token from GITHUB_TOKEN or GH_TOKEN.

func ObjectKey

func ObjectKey(rawURL string) (string, error)

ObjectKey returns the key in the mirror for rawURL. The key is "<host>/<path>", so that https://example.com/foo/bar.tar.gz is mirrored as example.com/foo/bar.tar.gz.

func Run

func Run(ctx context.Context) error

func URLReplacements

func URLReplacements(artifacts []Artifact, mirrorURL string) (string, error)

URLReplacements returns mise settings of url_replacements (in TOML) that route the artifact URLs to the mirror served at mirrorURL.

Types

type Artifact

type Artifact struct {
	Tool     string
	Version  string
	Backend  string
	Platform string
	URL      string
	Checksum string
}

Artifact is a downloadable file listed in a mise lock file.

func FilterPlatforms

func FilterPlatforms(artifacts []Artifact, platforms []string) []Artifact

FilterPlatforms returns the artifacts for the given platforms. If platforms is empty, all artifacts are returned.

func LoadLockFiles

func LoadLockFiles(paths []string) ([]Artifact, error)

LoadLockFiles reads mise lock files and returns the artifacts deduplicated by URL.

func ParseLockFile

func ParseLockFile(r io.Reader) ([]Artifact, error)

ParseLockFile parses a mise lock file and returns the artifacts listed in it.

Both `[tools.<name>."platforms.<platform>"]` (current format) and `[tools.<name>.platforms.<platform>]` (nested table) forms are supported.

type CLI

type CLI struct {
	Destination string   `arg:"" help:"Destination directory or S3 URL (e.g. /path/to/dir, s3://bucket/prefix/)"`
	LockFiles   []string `` /* 138-byte string literal not displayed */
	Platforms   []string `` /* 170-byte string literal not displayed */
	Concurrency int      `help:"Number of concurrent downloads." default:"4" env:"MISE_MIRROR_CONCURRENCY"`
	Force       bool     `help:"Overwrite files that already exist in the destination." env:"MISE_MIRROR_FORCE"`
	DryRun      bool     `help:"Show what would be mirrored without downloading." env:"MISE_MIRROR_DRY_RUN"`
	Debug       bool     `help:"Enable debug logging." env:"MISE_MIRROR_DEBUG"`
	MirrorURL   string   `` /* 202-byte string literal not displayed */

	Version kong.VersionFlag `short:"v" help:"Show version and exit."`
	// contains filtered or unexported fields
}

type LocalStorage

type LocalStorage struct {
	// contains filtered or unexported fields
}

LocalStorage stores files under a local directory.

func NewLocalStorage

func NewLocalStorage(dir string) *LocalStorage

func (*LocalStorage) Exists

func (s *LocalStorage) Exists(ctx context.Context, key string) (bool, error)

func (*LocalStorage) Location

func (s *LocalStorage) Location(key string) string

func (*LocalStorage) Put

func (s *LocalStorage) Put(ctx context.Context, key string, src *os.File) error

type Mirror

type Mirror struct {
	Storage     Storage
	HTTPClient  *http.Client
	Concurrency int
	Force       bool
	DryRun      bool
	// GitHubToken is sent only to GitHub hosts (see githubHosts).
	GitHubToken string
}

Mirror downloads artifacts and stores them into Storage.

func (*Mirror) Run

func (m *Mirror) Run(ctx context.Context, artifacts []Artifact) error

Run mirrors all artifacts. It continues on failure and returns the joined errors.

type S3Client

type S3Client interface {
	HeadObject(ctx context.Context, params *s3.HeadObjectInput, optFns ...func(*s3.Options)) (*s3.HeadObjectOutput, error)
	PutObject(ctx context.Context, params *s3.PutObjectInput, optFns ...func(*s3.Options)) (*s3.PutObjectOutput, error)
}

S3Client is a subset of the S3 API used by S3Storage.

type S3Storage

type S3Storage struct {
	// contains filtered or unexported fields
}

S3Storage stores files in an S3 bucket.

func NewS3Storage

func NewS3Storage(client S3Client, bucket, prefix string) *S3Storage

func (*S3Storage) Exists

func (s *S3Storage) Exists(ctx context.Context, key string) (bool, error)

func (*S3Storage) Location

func (s *S3Storage) Location(key string) string

func (*S3Storage) Put

func (s *S3Storage) Put(ctx context.Context, key string, src *os.File) error

type Storage

type Storage interface {
	// Exists reports whether the object for key exists.
	Exists(ctx context.Context, key string) (bool, error)
	// Put stores the content of src as key.
	Put(ctx context.Context, key string, src *os.File) error
	// Location returns a human readable location of key.
	Location(key string) string
}

Storage is a destination of mirrored files.

func NewStorage

func NewStorage(ctx context.Context, dest string) (Storage, error)

NewStorage returns a Storage for dest. dest is a local directory path or an S3 URL (s3://bucket/prefix/).