getter

package module
v0.0.0-...-906e156 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 22, 2018 License: MPL-2.0 Imports: 44 Imported by: 1

README

go-getter

Build Status Build status Go Documentation

go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input.

The power of this library is being flexible in being able to download from a number of different sources (file paths, Git, HTTP, Mercurial, etc.) using a single string as input. This removes the burden of knowing how to download from a variety of sources from the implementer.

The concept of a detector automatically turns invalid URLs into proper URLs. For example: "github.com/hashicorp/go-getter" would turn into a Git URL. Or "./foo" would turn into a file URL. These are extensible.

This library is used by Terraform for downloading modules and Nomad for downloading binaries.

Installation and Usage

Package documentation can be found on GoDoc.

Installation can be done with a normal go get:

$ go get github.com/hashicorp/go-getter

go-getter also has a command you can use to test URL strings:

$ go install github.com/hashicorp/go-getter/cmd/go-getter
...

$ go-getter github.com/foo/bar ./foo
...

The command is useful for verifying URL structures.

URL Format

go-getter uses a single string URL as input to download from a variety of protocols. go-getter has various "tricks" with this URL to do certain things. This section documents the URL format.

Supported Protocols and Detectors

Protocols are used to download files/directories using a specific mechanism. Example protocols are Git and HTTP.

Detectors are used to transform a valid or invalid URL into another URL if it matches a certain pattern. Example: "github.com/user/repo" is automatically transformed into a fully valid Git URL. This allows go-getter to be very user friendly.

go-getter out of the box supports the following protocols. Additional protocols can be augmented at runtime by implementing the Getter interface.

  • Local files
  • Git
  • Mercurial
  • HTTP
  • Amazon S3

In addition to the above protocols, go-getter has what are called "detectors." These take a URL and attempt to automatically choose the best protocol for it, which might involve even changing the protocol. The following detection is built-in by default:

  • File paths such as "./foo" are automatically changed to absolute file URLs.
  • GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically changed to Git protocol over HTTP.
  • BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically changed to a Git or mercurial protocol using the BitBucket API.
Forced Protocol

In some cases, the protocol to use is ambiguous depending on the source URL. For example, "http://github.com/mitchellh/vagrant.git" could reference an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this URL.

Forced protocol can be done by prefixing the URL with the protocol followed by double colons. For example: git::http://github.com/mitchellh/vagrant.git would download the given HTTP URL using the Git protocol.

Forced protocols will also override any detectors.

In the absense of a forced protocol, detectors may be run on the URL, transforming the protocol anyways. The above example would've used the Git protocol either way since the Git detector would've detected it was a GitHub URL.

Protocol-Specific Options

Each protocol can support protocol-specific options to configure that protocol. For example, the git protocol supports specifying a ref query parameter that tells it what ref to checkout for that Git repository.

The options are specified as query parameters on the URL (or URL-like string) given to go-getter. Using the Git example above, the URL below is a valid input to go-getter:

github.com/hashicorp/go-getter?ref=abcd1234

The protocol-specific options are documented below the URL format section. But because they are part of the URL, we point it out here so you know they exist.

Subdirectories

If you want to download only a specific subdirectory from a downloaded directory, you can specify a subdirectory after a double-slash //. go-getter will first download the URL specified before the double-slash (as if you didn't specify a double-slash), but will then copy the path after the double slash into the target directory.

For example, if you're downloading this GitHub repository, but you only want to download the test-fixtures directory, you can do the following:

https://github.com/hashicorp/go-getter.git//test-fixtures

If you downloaded this to the /tmp directory, then the file /tmp/archive.gz would exist. Notice that this file is in the test-fixtures directory in this repository, but because we specified a subdirectory, go-getter automatically copied only that directory contents.

Subdirectory paths may contain may also use filesystem glob patterns. The path must match exactly one entry or go-getter will return an error. This is useful if you're not sure the exact directory name but it follows a predictable naming structure.

For example, the following URL would also work:

https://github.com/hashicorp/go-getter.git//test-*
Checksumming

For file downloads of any protocol, go-getter can automatically verify a checksum for you. Note that checksumming only works for downloading files, not directories, but checksumming will work for any protocol.

To checksum a file, append a checksum query parameter to the URL. The paramter value should be in the format of type:value, where type is "md5", "sha1", "sha256", or "sha512". The "value" should be the actual checksum value. go-getter will parse out this query parameter automatically and use it to verify the checksum. An example URL is shown below:

./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21

The checksum query parameter is never sent to the backend protocol implementation. It is used at a higher level by go-getter itself.

Unarchiving

go-getter will automatically unarchive files into a file or directory based on the extension of the file being requested (over any protocol). This works for both file and directory downloads.

go-getter looks for an archive query parameter to specify the format of the archive. If this isn't specified, go-getter will use the extension of the path to see if it appears archived. Unarchiving can be explicitly disabled by setting the archive query parameter to false.

The following archive formats are supported:

  • tar.gz and tgz
  • tar.bz2 and tbz2
  • tar.xz and txz
  • zip
  • gz
  • bz2
  • xz

For example, an example URL is shown below:

./foo.zip

This will automatically be inferred to be a ZIP file and will be extracted. You can also be explicit about the archive type:

./some/other/path?archive=zip

And finally, you can disable archiving completely:

./some/path?archive=false

You can combine unarchiving with the other features of go-getter such as checksumming. The special archive query parameter will be removed from the URL before going to the final protocol downloader.

Protocol-Specific Options

This section documents the protocol-specific options that can be specified for go-getter. These options should be appended to the input as normal query parameters. Depending on the usage of go-getter, applications may provide alternate ways of inputting options. For example, Nomad provides a nice options block for specifying options rather than in the URL.

General (All Protocols)

The options below are available to all protocols:

  • archive - The archive format to use to unarchive this file, or "" (empty string) to disable unarchiving. For more details, see the complete section on archive support above.

  • checksum - Checksum to verify the downloaded file or archive. See the entire section on checksumming above for format and more details.

  • filename - When in file download mode, allows specifying the name of the downloaded file on disk. Has no effect in directory mode.

Local Files (file)

None

Git (git)

  • ref - The Git ref to checkout. This is a ref, so it can point to a commit SHA, a branch name, etc. If it is a named ref such as a branch name, go-getter will update it to the latest on each get.

  • sshkey - An SSH private key to use during clones. The provided key must be a base64-encoded string. For example, to generate a suitable sshkey from a private key file on disk, you would run base64 -w0 <file>.

    Note: Git 2.3+ is required to use this feature.

Mercurial (hg)
  • rev - The Mercurial revision to checkout.
HTTP (http)
Basic Authentication

To use HTTP basic authentication with go-getter, simply prepend username:password@ to the hostname in the URL such as https://Aladdin:OpenSesame@www.example.com/index.html. All special characters, including the username and password, must be URL encoded.

S3 (s3)

S3 takes various access configurations in the URL. Note that it will also read these from standard AWS environment variables if they're set. S3 compliant servers like Minio are also supported. If the query parameters are present, these take priority.

  • aws_access_key_id - AWS access key.
  • aws_access_key_secret - AWS access key secret.
  • aws_access_token - AWS access token if this is being used.
Using IAM Instance Profiles with S3

If you use go-getter and want to use an EC2 IAM Instance Profile to avoid using credentials, then just omit these and the profile, if available will be used automatically.

Using S3 with Minio

If you use go-gitter for Minio support, you must consider the following:

  • aws_access_key_id (required) - Minio access key.
  • aws_access_key_secret (required) - Minio access key secret.
  • region (optional - defaults to us-east-1) - Region identifier to use.
  • version (optional - defaults to Minio default) - Configuration file format.
S3 Bucket Examples

S3 has several addressing schemes used to reference your bucket. These are listed here: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

Some examples for these addressing schemes:

Documentation

Overview

getter is a package for downloading files or directories from a variety of protocols.

getter is unique in its ability to download both directories and files. It also detects certain source strings to be protocol-specific URLs. For example, "github.com/hashicorp/go-getter" would turn into a Git URL and use the Git protocol.

Protocols and detectors are extensible.

To get started, see Client.

Index

Constants

This section is empty.

Variables

View Source 
var Decompressors map[string]Decompressor

Decompressors is the mapping of extension to the Decompressor implementation that will decompress that extension/type.

View Source 
var Detectors []Detector

Detectors is the list of detectors that are tried on an invalid URL. This is also the order they're tried (index 0 is first).

View Source 
var Getters map[string]Getter

Getters is the mapping of scheme to the Getter implementation that will be used to get a dependency.

Functions

func Detect 

func Detect(src string, pwd string, ds []Detector) (string, error)

Detect turns a source string into another source string if it is detected to be of a known pattern.

The third parameter should be the list of detectors to use in the order to try them. If you don't want to configure this, just use the global Detectors variable.

This is safe to be called with an already valid source string: Detect will just return it.

func Get 

func Get(dst, src string) error

Get downloads the directory specified by src into the folder specified by dst. If dst already exists, Get will attempt to update it.

src is a URL, whereas dst is always just a file path to a folder. This folder doesn't need to exist. It will be created if it doesn't exist.

func GetAny 

func GetAny(dst, src string) error

GetAny downloads a URL into the given destination. Unlike Get or GetFile, both directories and files are supported.

dst must be a directory. If src is a file, it will be downloaded into dst with the basename of the URL. If src is a directory or archive, it will be unpacked directly into dst.

func GetFile 

func GetFile(dst, src string) error

GetFile downloads the file specified by src into the path specified by dst.

func SourceDirSubdir 

func SourceDirSubdir(src string) (string, string)

SourceDirSubdir takes a source and returns a tuple of the URL without the subdir and the URL with the subdir.

func SubdirGlob 

func SubdirGlob(dst, subDir string) (string, error)

SubdirGlob returns the actual subdir with globbing processed.

dst should be a destination directory that is already populated (the download is complete) and subDir should be the set subDir. If subDir is an empty string, this returns an empty string.

The returned path is the full absolute path.

func TestDecompressor 

func TestDecompressor(t testing.T, d Decompressor, cases []TestDecompressCase)

TestDecompressor is a helper function for testing generic decompressors.

Types

type BitBucketDetector 

type BitBucketDetector struct{}

BitBucketDetector implements Detector to detect BitBucket URLs and turn them into URLs that the Git or Hg Getter can understand.

func (*BitBucketDetector) Detect 

func (d *BitBucketDetector) Detect(src, _ string) (string, bool, error)

type Bzip2Decompressor 

type Bzip2Decompressor struct{}

Bzip2Decompressor is an implementation of Decompressor that can decompress bz2 files.

func (*Bzip2Decompressor) Decompress 

func (d *Bzip2Decompressor) Decompress(dst, src string, dir bool) error

type Client 

type Client struct {
	// Src is the source URL to get.
	//
	// Dst is the path to save the downloaded thing as. If Dir is set to
	// true, then this should be a directory. If the directory doesn't exist,
	// it will be created for you.
	//
	// Pwd is the working directory for detection. If this isn't set, some
	// detection may fail. Client will not default pwd to the current
	// working directory for security reasons.
	Src string
	Dst string
	Pwd string

	// Mode is the method of download the client will use. See ClientMode
	// for documentation.
	Mode ClientMode

	// Detectors is the list of detectors that are tried on the source.
	// If this is nil, then the default Detectors will be used.
	Detectors []Detector

	// Decompressors is the map of decompressors supported by this client.
	// If this is nil, then the default value is the Decompressors global.
	Decompressors map[string]Decompressor

	// Getters is the map of protocols supported by this client. If this
	// is nil, then the default Getters variable will be used.
	Getters map[string]Getter

	// Dir, if true, tells the Client it is downloading a directory (versus
	// a single file). This distinction is necessary since filenames and
	// directory names follow the same format so disambiguating is impossible
	// without knowing ahead of time.
	//
	// WARNING: deprecated. If Mode is set, that will take precedence.
	Dir bool
}

Client is a client for downloading things.

Top-level functions such as Get are shortcuts for interacting with a client. Using a client directly allows more fine-grained control over how downloading is done, as well as customizing the protocols supported.

func (*Client) Get 

func (c *Client) Get() error

Get downloads the configured source to the destination.

type ClientMode 

type ClientMode uint

ClientMode is the mode that the client operates in. 

const (
	ClientModeInvalid ClientMode = iota

	// ClientModeAny downloads anything it can. In this mode, dst must
	// be a directory. If src is a file, it is saved into the directory
	// with the basename of the URL. If src is a directory or archive,
	// it is unpacked directly into dst.
	ClientModeAny

	// ClientModeFile downloads a single file. In this mode, dst must
	// be a file path (doesn't have to exist). src must point to a single
	// file. It is saved as dst.
	ClientModeFile

	// ClientModeDir downloads a directory. In this mode, dst must be
	// a directory path (doesn't have to exist). src must point to an
	// archive or directory (such as in s3).
	ClientModeDir
)

type Decompressor 

type Decompressor interface {
	// Decompress should decompress src to dst. dir specifies whether dst
	// is a directory or single file. src is guaranteed to be a single file
	// that exists. dst is not guaranteed to exist already.
	Decompress(dst, src string, dir bool) error
}

Decompressor defines the interface that must be implemented to add support for decompressing a type.

Important: if you're implementing a decompressor, please use the containsDotDot helper in this file to ensure that files can't be decompressed outside of the specified directory.

type Detector 

type Detector interface {
	// Detect will detect whether the string matches a known pattern to
	// turn it into a proper URL.
	Detect(string, string) (string, bool, error)
}

Detector defines the interface that an invalid URL or a URL with a blank scheme is passed through in order to determine if its shorthand for something else well-known.

type FileDetector 

type FileDetector struct{}

FileDetector implements Detector to detect file paths.

func (*FileDetector) Detect 

func (d *FileDetector) Detect(src, pwd string) (string, bool, error)

type FileGetter 

type FileGetter struct {
	// Copy, if set to true, will copy data instead of using a symlink
	Copy bool
}

FileGetter is a Getter implementation that will download a module from a file scheme.

func (*FileGetter) ClientMode 

func (g *FileGetter) ClientMode(u *url.URL) (ClientMode, error)

func (*FileGetter) Get 

func (g *FileGetter) Get(dst string, u *url.URL) error

func (*FileGetter) GetFile 

func (g *FileGetter) GetFile(dst string, u *url.URL) error

type FolderStorage 

type FolderStorage struct {
	// StorageDir is the directory where the modules will be stored.
	StorageDir string
}

FolderStorage is an implementation of the Storage interface that manages modules on the disk.

func (*FolderStorage) Dir 

func (s *FolderStorage) Dir(key string) (d string, e bool, err error)

Dir implements Storage.Dir

func (*FolderStorage) Get 

func (s *FolderStorage) Get(key string, source string, update bool) error

Get implements Storage.Get

type Getter 

type Getter interface {
	// Get downloads the given URL into the given directory. This always
	// assumes that we're updating and gets the latest version that it can.
	//
	// The directory may already exist (if we're updating). If it is in a
	// format that isn't understood, an error should be returned. Get shouldn't
	// simply nuke the directory.
	Get(string, *url.URL) error

	// GetFile downloads the give URL into the given path. The URL must
	// reference a single file. If possible, the Getter should check if
	// the remote end contains the same file and no-op this operation.
	GetFile(string, *url.URL) error

	// ClientMode returns the mode based on the given URL. This is used to
	// allow clients to let the getters decide which mode to use.
	ClientMode(*url.URL) (ClientMode, error)
}

Getter defines the interface that schemes must implement to download things.

type GitGetter 

type GitGetter struct{}

GitGetter is a Getter implementation that will download a module from a git repository.

func (*GitGetter) ClientMode 

func (g *GitGetter) ClientMode(_ *url.URL) (ClientMode, error)

func (*GitGetter) Get 

func (g *GitGetter) Get(dst string, u *url.URL) error

func (*GitGetter) GetFile 

func (g *GitGetter) GetFile(dst string, u *url.URL) error

GetFile for Git doesn't support updating at this time. It will download the file every time.

type GitHubDetector 

type GitHubDetector struct{}

GitHubDetector implements Detector to detect GitHub URLs and turn them into URLs that the Git Getter can understand.

func (*GitHubDetector) Detect 

func (d *GitHubDetector) Detect(src, _ string) (string, bool, error)

type GzipDecompressor 

type GzipDecompressor struct{}

GzipDecompressor is an implementation of Decompressor that can decompress gzip files.

func (*GzipDecompressor) Decompress 

func (d *GzipDecompressor) Decompress(dst, src string, dir bool) error

type HgGetter 

type HgGetter struct{}

HgGetter is a Getter implementation that will download a module from a Mercurial repository.

func (*HgGetter) ClientMode 

func (g *HgGetter) ClientMode(_ *url.URL) (ClientMode, error)

func (*HgGetter) Get 

func (g *HgGetter) Get(dst string, u *url.URL) error

func (*HgGetter) GetFile 

func (g *HgGetter) GetFile(dst string, u *url.URL) error

GetFile for Hg doesn't support updating at this time. It will download the file every time.

type HttpGetter 

type HttpGetter struct {
	// Netrc, if true, will lookup and use auth information found
	// in the user's netrc file if available.
	Netrc bool

	// Client is the http.Client to use for Get requests.
	// This defaults to a cleanhttp.DefaultClient if left unset.
	Client *http.Client
}

HttpGetter is a Getter implementation that will download from an HTTP endpoint.

For file downloads, HTTP is used directly.

The protocol for downloading a directory from an HTTP endpoing is as follows:

An HTTP GET request is made to the URL with the additional GET parameter "terraform-get=1". This lets you handle that scenario specially if you wish. The response must be a 2xx.

First, a header is looked for "X-Terraform-Get" which should contain a source URL to download.

If the header is not present, then a meta tag is searched for named "terraform-get" and the content should be a source URL.

The source URL, whether from the header or meta tag, must be a fully formed URL. The shorthand syntax of "github.com/foo/bar" or relative paths are not allowed.

func (*HttpGetter) ClientMode 

func (g *HttpGetter) ClientMode(u *url.URL) (ClientMode, error)

func (*HttpGetter) Get 

func (g *HttpGetter) Get(dst string, u *url.URL) error

func (*HttpGetter) GetFile 

func (g *HttpGetter) GetFile(dst string, u *url.URL) error

type MockGetter 

type MockGetter struct {
	// Proxy, if set, will be called after recording the calls below.
	// If it isn't set, then the *Err values will be returned.
	Proxy Getter

	GetCalled bool
	GetDst    string
	GetURL    *url.URL
	GetErr    error

	GetFileCalled bool
	GetFileDst    string
	GetFileURL    *url.URL
	GetFileErr    error
}

MockGetter is an implementation of Getter that can be used for tests.

func (*MockGetter) ClientMode 

func (g *MockGetter) ClientMode(u *url.URL) (ClientMode, error)

func (*MockGetter) Get 

func (g *MockGetter) Get(dst string, u *url.URL) error

func (*MockGetter) GetFile 

func (g *MockGetter) GetFile(dst string, u *url.URL) error

type S3Detector 

type S3Detector struct{}

S3Detector implements Detector to detect S3 URLs and turn them into URLs that the S3 getter can understand.

func (*S3Detector) Detect 

func (d *S3Detector) Detect(src, _ string) (string, bool, error)

type S3Getter 

type S3Getter struct{}

S3Getter is a Getter implementation that will download a module from a S3 bucket.

func (*S3Getter) ClientMode 

func (g *S3Getter) ClientMode(u *url.URL) (ClientMode, error)

func (*S3Getter) Get 

func (g *S3Getter) Get(dst string, u *url.URL) error

func (*S3Getter) GetFile 

func (g *S3Getter) GetFile(dst string, u *url.URL) error

type Storage 

type Storage interface {
	// Dir returns the directory on local disk where the directory source
	// can be loaded from.
	Dir(string) (string, bool, error)

	// Get will download and optionally update the given directory.
	Get(string, string, bool) error
}

Storage is an interface that knows how to lookup downloaded directories as well as download and update directories from their sources into the proper location.

type TarBzip2Decompressor 

type TarBzip2Decompressor struct{}

TarBzip2Decompressor is an implementation of Decompressor that can decompress tar.bz2 files.

func (*TarBzip2Decompressor) Decompress 

func (d *TarBzip2Decompressor) Decompress(dst, src string, dir bool) error

type TarGzipDecompressor 

type TarGzipDecompressor struct{}

TarGzipDecompressor is an implementation of Decompressor that can decompress tar.gzip files.

func (*TarGzipDecompressor) Decompress 

func (d *TarGzipDecompressor) Decompress(dst, src string, dir bool) error

type TarXzDecompressor 

type TarXzDecompressor struct{}

TarXzDecompressor is an implementation of Decompressor that can decompress tar.xz files.

func (*TarXzDecompressor) Decompress 

func (d *TarXzDecompressor) Decompress(dst, src string, dir bool) error

type TestDecompressCase 

type TestDecompressCase struct {
	Input   string     // Input is the complete path to the input file
	Dir     bool       // Dir is whether or not we're testing directory mode
	Err     bool       // Err is whether we expect an error or not
	DirList []string   // DirList is the list of files for Dir mode
	FileMD5 string     // FileMD5 is the expected MD5 for a single file
	Mtime   *time.Time // Mtime is the optionally expected mtime for a single file (or all files if in Dir mode)
}

TestDecompressCase is a single test case for testing decompressors

type XzDecompressor 

type XzDecompressor struct{}

XzDecompressor is an implementation of Decompressor that can decompress xz files.

func (*XzDecompressor) Decompress 

func (d *XzDecompressor) Decompress(dst, src string, dir bool) error

type ZipDecompressor 

type ZipDecompressor struct{}

ZipDecompressor is an implementation of Decompressor that can decompress tar.gzip files.

func (*ZipDecompressor) Decompress 

func (d *ZipDecompressor) Decompress(dst, src string, dir bool) error

Source Files

View all Source files

Directories

Path Synopsis
cmd
go-getter
helper
url

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.