gitfs

README

gitfs

Package gitfs is a complete solution for static files in Go code.

This package enables loading any file from remote git repository in Go code. The static files are not required to be packed into the go binary in order to be used. They can be loaded in runtime when accessed, or preloaded when the program starts.

The following features are supported:

• Loading of specific version/tag/branch.

• For debug purposes, the files can be loaded from local path instead of the remote repository. The transition from remote project to local files is smooth.

• Files are loaded lazily by default or they can be preloaded if required.

• Files can be packed to the Go binary using a command line tool, which provides a smooth transition from using remote repository to binary packed.

• This project is using the standard http.FileSystem interface for the loaded static files, and provides tooling around it.

Usage

First, create a filesystem using the New function. This function accepts the project path with pattern: github.com/<owner>/<repo>(/<path>)?(@<ref>)?. If no path is specified, the root of the project will be used. ref can be any git branch using heads/<branch name> or any git tag using tags/<tag>. If the tag is of Semver format, the tags/ prefix is not required. If no ref is specified, the default branch will be used.

In the following example, the repository github.com/x/y at version v1.2.3 and internal path "static" is loaded:

fs, err := gitfs.New(ctx, "github.com/x/y/static@v1.2.3")

Reading a file from the repository can be done using the Open method. This function accepts a path, relative to the root of the defined filesystem.

f, err := fs.Open("index.html")

The variable fs implements http.FileSystem, and can be used for anything that accepts it. For example, it can be used for serving static content using the standard library:

http.Handle("/", http.FileServer(fs))

Private Repositories

When used with private github repository, it should be accessed with appropriate credentials. The credentials can be passed by providing an HTTP client. For example, to use a Github Token from environnement variable GITHUB_TOKEN:

token := os.Getenv("GITHUB_TOKEN")
client := oauth2.NewClient(
	context.Background(),
	oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token}))
fs, err := gitfs.New(ctx, "github.com/x/y", gitfs.OptClient(client))

Development

For quick development purposes, it is easier and faster to use local static content and not remote content that was pushed to the remote repository. This is enabled by the OptLocal option. To use this option only in local development and not in production system, it can be used as follow:

local := os.Getenv("LOCAL_DEBUG")
fs, err := gitfs.New(ctx, "github.com/x/y", gitfs.OptLocal(local))

In this example, we stored the value for OptLocal in an environment variable. As a result, when running the program with LOCAL_DEBUG=. local files will be used, while running without it will result in using the remote files. (the value of the environment variable should point to any directory within the github project).

Binary Packing

Using gitfs does not mean that files are required to be remotely fetched. When binary packing of the files is needed, a command line tool can pack them and no other changes in the code are required.

To get the tool run: go get github.com/posener/gitfs/cmd/gitfs.

Running the tool is by gitfs <patterns>. It will generate A file in the current directory that will contain all the required filesystem. The filesystems are detected by searching for gitfs.New calls.

Sub Packages

• bin: Package bin is a proxy to the internal/binfs.Register function.

• fsutil: Package fsutil provides useful utility functions for http.FileSystem.

Examples

Fsutil

The ./fsutil package is a collection of useful functions that can work with any http.FileSystem implementation. For example, here we will use a function that loads go templates from the filesystem.

ctx := context.Background()

// Open a git remote repository `posener/gitfs` in path `examples/templates`.
fs, err := New(ctx, "github.com/posener/gitfs/examples/templates")
if err != nil {
    log.Fatalf("Failed initialize filesystem: %s", err)
}

// Use util function that loads all templates according to a glob pattern.
tmpls, err := fsutil.TmplParseGlob(fs, nil, "*.gotmpl")
if err != nil {
    log.Fatalf("Failed parsing templates: %s", err)
}

// Execute the template and write to stdout.
tmpls.ExecuteTemplate(os.Stdout, "tmpl1.gotmpl", "Foo")

Open

With gitfs you can open a remote git repository, and load any file, including non-go files. In this example, the README.md file of a remote repository is loaded.

ctx := context.Background()

// The load path is of the form: github.com/<owner>/<repo>(/<path>)?(@<ref>)?.
// `ref` can reference any git tag or branch. If github releases are in Semver format,
// the `tags/` prefix is not needed in the `ref` part.
fs, err := New(ctx, "github.com/kelseyhightower/helloworld@3.0.0")
if err != nil {
    log.Fatalf("Failed initialize filesystem: %s", err)
}

// Open any file in the github repository, using the `Open` function. Both files
// and directory can be opened. The content is not loaded until it is actually being
// read. The content is loaded only once.
f, err := fs.Open("README.md")
if err != nil {
    log.Fatalf("Failed opening file: %s", err)
}

// Copy the content to stdout.
io.Copy(os.Stdout, f)

---

Created by goreadme

Documentation

Overview

Package gitfs is a complete solution for static files in Go code.

This package enables loading any file from remote git repository in Go code. The static files are not required to be packed into the go binary in order to be used. They can be loaded in runtime when accessed, or preloaded when the program starts.

The following features are supported:

* Loading of specific version/tag/branch.

* For debug purposes, the files can be loaded from local path instead of the remote repository. The transition from remote project to local files is smooth.

* Files are loaded lazily by default or they can be preloaded if required.

* Files can be packed to the Go binary using a command line tool, which provides a smooth transition from using remote repository to binary packed.

* This project is using the standard `http.FileSystem` interface for the loaded static files, and provides tooling around it.

Usage

First, create a filesystem using the `New` function. This function accepts the project path with pattern: `github.com/<owner>/<repo>(/<path>)?(@<ref>)?`. If no `path` is specified, the root of the project will be used. `ref` can be any git branch using `heads/<branch name>` or any git tag using `tags/<tag>`. If the tag is of Semver format, the `tags/` prefix is not required. If no `ref` is specified, the default branch will be used.

In the following example, the repository `github.com/x/y` at version v1.2.3 and internal path "static" is loaded:

fs, err := gitfs.New(ctx, "github.com/x/y/static@v1.2.3")

Reading a file from the repository can be done using the `Open` method. This function accepts a path, relative to the root of the defined filesystem.

f, err := fs.Open("index.html")

The variable `fs` implements `http.FileSystem`, and can be used for anything that accepts it. For example, it can be used for serving static content using the standard library:

http.Handle("/", http.FileServer(fs))

Private Repositories

When used with private github repository, it should be accessed with appropriate credentials. The credentials can be passed by providing an HTTP client. For example, to use a Github Token from environnement variable `GITHUB_TOKEN`:

token := os.Getenv("GITHUB_TOKEN")
client := oauth2.NewClient(
	context.Background(),
	oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token}))
fs, err := gitfs.New(ctx, "github.com/x/y", gitfs.OptClient(client))

Development

For quick development purposes, it is easier and faster to use local static content and not remote content that was pushed to the remote repository. This is enabled by the `OptLocal` option. To use this option only in local development and not in production system, it can be used as follow:

local := os.Getenv("LOCAL_DEBUG")
fs, err := gitfs.New(ctx, "github.com/x/y", gitfs.OptLocal(local))

In this example, we stored the value for `OptLocal` in an environment variable. As a result, when running the program with `LOCAL_DEBUG=.` local files will be used, while running without it will result in using the remote files. (the value of the environment variable should point to any directory within the github project).

Binary Packing

Using gitfs does not mean that files are required to be remotely fetched. When binary packing of the files is needed, a command line tool can pack them and no other changes in the code are required.

To get the tool run: `go get github.com/posener/gitfs/cmd/gitfs`.

Running the tool is by `gitfs <patterns>`. It will generate A file in the current directory that will contain all the required filesystem. The filesystems are detected by searching for `gitfs.New` calls.

Example (Fsutil)

The ./fsutil package is a collection of useful functions that can work with any `http.FileSystem` implementation. For example, here we will use a function that loads go templates from the filesystem.

ctx := context.Background()

// Open a git remote repository `posener/gitfs` in path `examples/templates`.
fs, err := New(ctx, "github.com/posener/gitfs/examples/templates")
if err != nil {
	log.Fatalf("Failed initialize filesystem: %s", err)
}

// Use util function that loads all templates according to a glob pattern.
tmpls, err := fsutil.TmplParseGlob(fs, nil, "*.gotmpl")
if err != nil {
	log.Fatalf("Failed parsing templates: %s", err)
}

// Execute the template and write to stdout.
tmpls.ExecuteTemplate(os.Stdout, "tmpl1.gotmpl", "Foo")

Output:

Hello, Foo

Example (Open)

With gitfs you can open a remote git repository, and load any file, including non-go files. In this example, the README.md file of a remote repository is loaded.

ctx := context.Background()

// The load path is of the form: github.com/<owner>/<repo>(/<path>)?(@<ref>)?.
// `ref` can reference any git tag or branch. If github releases are in Semver format,
// the `tags/` prefix is not needed in the `ref` part.
fs, err := New(ctx, "github.com/kelseyhightower/helloworld@3.0.0")
if err != nil {
	log.Fatalf("Failed initialize filesystem: %s", err)
}

// Open any file in the github repository, using the `Open` function. Both files
// and directory can be opened. The content is not loaded until it is actually being
// read. The content is loaded only once.
f, err := fs.Open("README.md")
if err != nil {
	log.Fatalf("Failed opening file: %s", err)
}

// Copy the content to stdout.
io.Copy(os.Stdout, f)

Output:

# helloworld

Index

• func New(ctx context.Context, project string, opts ...option) (http.FileSystem, error)
• func OptClient(client *http.Client) func(*config)
• func OptLocal(path string) func(*config)
• func OptPrefetch(prefetch bool) func(*config)
• func SetLogger(logger log.Logger)
• func WithContext(f http.File, ctx context.Context) http.File

Examples

• Package (Fsutil)
• Package (Open)

Functions

func New

func New(ctx context.Context, project string, opts ...option) (http.FileSystem, error)

New returns a new git filesystem for the given project.

Github: If the given project is a github project (of the form github.com/<owner>/<repo>(@<ref>)?(#<path>)? ), the returned filesystem will be fetching files from the given project. ref is optional and can be any github ref:

• `heads/<branch name>` for a branch.
• `tags/<tag>` for releases or git tags.
• `<version>` for Semver compatible releases (e.g. v1.2.3).

If no ref is set, the default branch will be used.

func OptClient

func OptClient(client *http.Client) func(*config)

OptClient sets up an HTTP client to perform request to the remote repository. This client can be used for authorization credentials.

func OptLocal

func OptLocal(path string) func(*config)

OptLocal result in looking for local git repository before accessing remote repository. The given path should be contained in a git repository which has a remote URL that matches the requested project.

func OptPrefetch

func OptPrefetch(prefetch bool) func(*config)

OptPrefetch sets prefetching all files in the filesystem when it is initially loaded.

func SetLogger

func SetLogger(logger log.Logger)

SetLogger sets informative logging for gitfs. If nil, no logging will be done.

func WithContext

func WithContext(f http.File, ctx context.Context) http.File

WithContext applies context to an http.File if it implements the contexter interface.

Usage example:

f, err := fs.Open("file")
// Handle err...
f = gitfs.WithContext(f, ctx)
_, err = f.Read(...)