brbundle

package module
Version: v1.1.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2020 License: Apache-2.0 Imports: 26 Imported by: 4

README

BRBundle

BRBundle is an asset bundling tool for Go. It is inspired by go-assets, go.rice and so on.

It supports four options to bundle assets to help building libraries, CLI applications, web applications, mobile applications, JavaScript(Gopher.js), including debugging process.

Install

$ go get go.pyspa.org/brbundle/...

Bundle Type Flow Chart

+--------------+      Yes
| go gettable? |+------------> Embedded Bundle
+----+---------+                    ^
     |                              |
     | No                           | Yes
     v                              |
+----------------+    Yes      +------------+    No
| Single Binary? |+----------> | Gopher.js? +---------->Exe Bundle
+----+-----------+             +------------+
     |
     | No
     v
+--------+            Yes
| Debug? +--------------------> Folder Bundle
+----+---+
     |
     | No
     v
   Packed Bundle

by asciiflow

Bundling Options

This tool supports 4 options to bundle assets:

  • Embedded Bundle

    This tool generates .go file. You have to generate .go file before compiling your application. This option is go-gettable.

    brbundle embedded [src-dir]

  • Exe Bundle

    This tool appends content files to your application. You can add them after compiling your application.

    brbundle bundle [exe-file] [src-dir]

  • Packed Bundle

    This tool generates one single binary that includes content files. It can use for DLC.

    brbundle pack [out-file.pb] [src-dir]

  • Folder Bundle

    For debugging. You can access content files without any building tasks. You don't have to prepare with brbundle command except encryption is needed.

How To Access Content

You can get contents by using Find() function. If contents are bundled with embedded bundle or exe bundle, you don't have to call any function to load.

import (
	"go.pyspa.org/brbundle"
	"image"
	"image/png"
)

func main() {
	file, err := brbundle.Find("file.png")
	reader, err := file.Reader()
	img, err := image.Decode(reader)
}
Getting Contents outside of executable

RegisterBundle() RegisterFolder() register external contents.

BRBundle searches the contents with the following order:

  • folder
  • bundle
  • exe-bundle
  • embedded
import (
	"go.pyspa.org/brbundle"
)

func main() {
	// load packed content
	brbundle.RegisterBundle("pack.pb")
	
	// load folder content
	brbundle.RegisterFolder("static/public")
}

Compression

BRBundle uses brotli by default. brotli is higher compression ratio with faster decompression speed than gzip.

BRBundle's web application middlewares can send brotli-ed content directly. Almost all browsers supports Content-Encoding: br.

It also supports more faster decompression algorithm LZ4.

Encryption

It supports contents encryption by AES. It uses base64 encoded 44 bytes key to encrypto. You can get your key with key-gen sub command.

Each bundles (embedded, bundle, each packed bundle files, folder bundles) can use separated encryption keys.

$ brbundle key-gen
yt6TX1eCBuG9GPRl2H6SJMbPNhPLOBxEHpb4kkaWyKUDg/tAZ2aSI3A86fw=

$ brbundle pack -c yt6TX1eCBuG9GPRl2H6SJMbPNhPLOBxEHpb4kkaWyKUDg/tAZ2aSI3A86fw= [src-dir]

Web Application Support

BRBundle support its own middleware for famous web application frameworks. It doesn't provide http.FileSystem compatible interface because:

  • It returns Brotli compressed content directly when client browser supports brotli.
  • It support fallback mechanism for Single Page Application.
net/http

"go.pyspa.org/brbundle/brhttp" contains http.Handler compatible API.

package main

import (
	"fmt"
	"net/http"

	"go.pyspa.org/brbundle"
	"go.pyspa.org/brbundle/brhttp"
)

// The simplest sample
// The server only returns only brbundle's content
// "/static/index.html" returns "index.html" of bundle.
func main() {
	fmt.Println("Listening at :8080")
	http.ListenAndServe(":8080", brhttp.Mount())
}

// Use ServeMux sample to handle static assets with API handler
func main() {
	m := http.NewServeMux()
	m.Handle("/public/", http.StripPrefix("/public", brhttp.Mount()))
	m.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello World")
	})
	fmt.Println("Listening at :8080")
	http.ListenAndServe(":8080", m)
}

// Single Page Application sample
// BRBundle's SPA supports is configured by WebOption of Mount() function
// If no contents found in bundles, it returns the specified content.
func main() {
	m := http.NewServeMux()
	m.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello World")
	})
	// Single Page Application is usually served index.html at any location
	// and routing errors are handled at browser.
	//
	// You should mount at the last line, because
	// it consumes all URL requests.
	m.Handle("/",
        brhttp.Mount(brbundle.WebOption{
            SPAFallback: "index.html",
        }),
	)
	fmt.Println("Listening at :8080")
	http.ListenAndServe(":8080", m)
}
Echo

Echo is a high performance, extensible, minimalist Go web framework.

package main

import (
	"fmt"
	"net/http"

	"github.com/labstack/echo"
	"go.pyspa.org/brbundle"
	"go.pyspa.org/brbundle/brecho"
)

// The simplest sample
func main() {
    e := echo.New()
	// Asterisk is required!
    e.GET("/*", brecho.Mount())
    e.Logger.Fatal(e.Start(":1323"))
}

// Use with echo.Group 
func main() {
	e := echo.New()
    e.GET("/api/status", func (c echo.Context) error {
        return c.String(http.StatusOK, "Hello, World!")
    })
	g := e.Group("/assets")
	// Asterisk is required!
	g.GET("/*", brecho.Mount())
	e.Logger.Fatal(e.Start(":1323"))
}

// Single Page Application sample
// BRBundle's SPA supports is configured by WebOption of Mount() function
// If no contents found in bundles, it returns the specified content.
//
// Single Page Application is usually served index.html at any location
// and routing errors are handled at browser.
func main() {
	e := echo.New()
	e.GET("/api/status", func (c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})
	// Use brbundle works as an error handler
	echo.NotFoundHandler = brecho.Mount(brbundle.WebOption{
		SPAFallback: "index.html",
	})
	e.Logger.Fatal(e.Start(":1323"))
}
Chi Router

Chi router is a lightweight, idiomatic and composable router for building Go HTTP services.

package main

import (
	"fmt"
	"net/http"

	"github.com/go-chi/chi"
	"go.pyspa.org/brbundle"
	"go.pyspa.org/brbundle/brchi"
)

// Use with chi.Router
func main() {
	r := chi.NewRouter()
	fmt.Println("You can access index.html at /public/index.html")
	// Asterisk is required!
	r.Get("/public/*", brchi.Mount())
	r.Get("/api", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello World")
	})
	fmt.Println("Listening at :8080")
	http.ListenAndServe(":8080", r)
}

// Single Page Application sample
// BRBundle's SPA supports is configured by WebOption of Mount() function
// If no contents found in bundles, it returns the specified content.
//
// Single Page Application is usually served index.html at any location
// and routing errors are handled at browser.
func main() {
	r := chi.NewRouter()
	r.Get("/api/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello World")
	})
	fmt.Println("You can access index.html at any location")
	// Use brbundle as an error handler
	r.NotFound(brchi.Mount(brbundle.WebOption{
		SPAFallback: "index.html",
	}))
	fmt.Println("Listening at :8080")
	http.ListenAndServe(":8080", r)
}
fasthttp / fasthttprouter

fasthttp is a fast http package. fasthttprouter is a high performance request router that scales well for fasthttp.

package main

import (
	"fmt"

	"github.com/buaazp/fasthttprouter"
	"go.pyspa.org/brbundle"
	"go.pyspa.org/brbundle/brfasthttp"
	"github.com/valyala/fasthttp"
)

// The simplest sample
func main() {
	fmt.Println("Listening at :8080")
	fmt.Println("You can access index.html at /index.html")
	fasthttp.ListenAndServe(":8080", brfasthttp.Mount())
}

// Use with fasthttprouter
func main() {
	r := fasthttprouter.New()
	r.GET("/api/status", func (ctx *fasthttp.RequestCtx) {
		ctx.WriteString("Hello, World!")
	})
	// "*filepath" is required at the last fragment of path string
	fmt.Println("You can access index.html at /static/index.html")
	r.GET("/static/*filepath", brfasthttp.Mount())

	fmt.Println("Listening at :8080")
	fasthttp.ListenAndServe(":8080", r.Handler)
}

// Single Page Application sample
// BRBundle's SPA supports is configured by WebOption of Mount() function
// If no contents found in bundles, it returns the specified content.
//
// Single Page Application is usually served index.html at any location
// and routing errors are handled at browser.
func main() {
	r := fasthttprouter.New()
	r.GET("/api/status", func (ctx *fasthttp.RequestCtx) {
		ctx.WriteString("Hello, World!")
	})
	fmt.Println("You can access index.html at any location")
	// Use brbundle works as an error handler
	r.NotFound = brfasthttp.Mount(brbundle.WebOption{
		SPAFallback: "index.html",
	})
	fmt.Println("Listening at :8080")
	fasthttp.ListenAndServe(":8080", r.Handler)
}
Gin

Gin is a HTTP web framework written in Go (Golang). It features a Martini-like API with much better performance -- up to 40 times faster.

package main

import (
	"fmt"
	"net/http"

	"github.com/gin-gonic/gin"
	"go.pyspa.org/brbundle"
	"go.pyspa.org/brbundle/brgin"
)

// Use with gin's router
func main() {
    r := gin.Default()
    r.GET("/api/status", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "status": "ok",
        })
    })
    fmt.Println("You can access index.html at /static/index.html")
	// "*filepath" is required at the last fragment of path string
    r.GET("/static/*filepath", brgin.Mount())
    r.Run(":8080")
}

// Single Page Application sample
// BRBundle's SPA supports is configured by WebOption of Mount() function
// If no contents found in bundles, it returns the specified content.
//
// Single Page Application is usually served index.html at any location
// and routing errors are handled at browser.
func main() {
	r := gin.Default()
    r.GET("/api/status", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "status": "ok",
        })
    })
	fmt.Println("You can access index.html at any location")
	// Use brbundle works as an error handler
	r.NoRoute(brgin.Mount(brbundle.WebOption{
		SPAFallback: "index.html",
	}))
	fmt.Println("Listening at :8080")
	r.Run(":8080")
}

Internal Design

File Format

It uses zip format to make single packed file. Embedded bundles and Exe bundles also use zip format. It doesn't use Deflate algorithm. It uses Brotli or LZ4 inside it.

Selecting Compression Method

BRBundle chooses compression format Brotli and LZ4 automatically.

That option you can choose is using -f (faster) or not.

  • If your application is a web application server, always turn off -f
  • Otherwise, decide using -f from size and booting speed.

-f option makes the content compressed with LZ4.

But Brotli has some cons. If the content is already compressed (like PNG, JPEG, OpenOffice formats), compression ratio is not effective. And loading compressed contents is slower than uncompressed content. Even if turned off Brotli, BRBundle fall back to LZ4. So the content like JSON becomes smaller than original and not slower than uncompressed content so much.

Now, current code skip compression if the content size after compression is not enough small:

  • var u: int = uncompressed_size
  • var c: int = compressed_size
  • var enough_small: bool = (u - 1000 > c) || (u > 10000 && (u * 0.90 > c))

Documentation

Overview

brbundle package provides asset bundler's runtime that uses Brotli (https://github.com/google/brotli).

Source repository is here: https://github.com/pyspa/brbundle

Install

To install runtime and commandline tool, type the following command: 

$ go get go.pyspa.org/brbundle/...

Asset Handling

This package provides four kind of bundles to handle assets:

1. Embedding. Generate .go file that includes binary representation of asset files. This is best for libraries and so on that is needed to be go gettable.

2. Appended to executable. Generate .zip file internally and appended to executables. You can replace assets after building.

3. Single packed binary file. You can specify and import during runtime. I assume this used for DLC.

4. Folder. This is for debugging. You don't have to do anything to import asset

brbundle searches assets the following orders: 

Folder -> Single binary file -> Assets on executable -> Embedded assets

Generate Bundle

The following command generates .go file: 

$ brbundle embedded <src-dir>

The following command append assets to executable: 

$ brbundle bundle <exec-file-path> <src-dir>

The following command generates single packed file: 

$ brbundle pack <out-file-path> <src-dir>

The following command generates asset folder. You can use regular cp command even if you don't have to encrypto assets: 

$ brbundle folder <dest-dir> <src-dir>

Standard Usage

It is easy to use the assets: 

file, err := brbundle.Find("file.png")
reader, err := file.Reader()
img, err := image.Decode(reader)

Embedded assets and assets appended to executable are available by default. The following functions registers assets in single packed file and local folder: 

brbundle.RegisterBundle("masterdata.pb")
brbundle.RegisterFolder("frontend/bist")

Web Framework Middlewares

You can save the earth by using brbundle. brbundle middlewares brotli content directly when browser supports it. Currently, more than 90% browsers already support (https://caniuse.com/#feat=brotli). brbundle provides the following frameworks' middleware:

  • net/http
  • echo
  • gin
  • fasthttp and fastrouter
  • chi router

net/http: 

m := http.NewServeMux()
m.Handle("/static/",
  http.StripPrefix("/static",
  brhttp.Mount())
http.ListenAndServe("localhost:8000", m)

These middlewares also support SPA(Single Page Application). More detail information is on Angular web site (https://angular.io/guide/deployment#routed-apps-must-fallback-to-indexhtml).

All samples are in examples folder: https://github.com/pyspa/brbundle/tree/master/examples

Compression Option

brbundle uses Brotli by default. If you pass --fast/-f option, brbundle uses Snappy (https://github.com/google/snappy) instead of Brotli. Snappy has low compression ratio, but very fast.

Encryption

brbundle supports encryption. You can generates encryption/decryption key by the following command: 

$ brbundle key-gen
pBJ0IB3x4EogUVNqmlI4I0EV9+aGpozmIQvSfF+PLo0NfzeamIeaeXHoTqs

When creating bundles, you can pass the key via --crypto/-c option: 

$ brbundle pack -c pBJ0IB3x4EogUVNqmlI4I0E... images.pb images

Keys should be passed the following functions: 

// for embedded assets
// default name is empty string and you can change by using --name/-n option of brbundle command
brbundle.SetDecryptoKeyToEmbeddedBundle("name", "pBJ0IB3x4EogUVNqmlI4I0E...")
// for executable
brbundle.SetDecryptoKeyToExeBundle("pBJ0IB3x4EogUVNqmlI4I0E...")
// for bundle
brbundle.RegisterBundle("bundle.pb", brbundle.Option{
  DecryptoKey: "pBJ0IB3x4EogUVNqmlI4I0E...",
})
// for folder
brbundle.RegisterEncryptedFolder("public", "pBJ0IB3x4EogUVNqmlI4I0E...")

Index

Constants

View Source 
const (
	FolderBundleType   BundleType = 0
	ManifestBundleType            = 1
	PackedBundleType              = 2
	ExeBundleType                 = 3
	EmbeddedBundleType            = 4
)
View Source 
const (
	NoCompression CompressionType = iota
	Brotli

	NoEncryption EncryptionType = iota
	AES
)
View Source 
const (
	UseBrotli     = "b"
	UseLZ4        = "l"
	NotToCompress = "-"

	UseAES        = "a"
	NotToEncrypto = "-"
)
View Source 
const ZIPMethodSnappy uint16 = 65535

Variables

View Source 
var DefaultRepository = NewRepository()

DefaultRepository is a default repository instance

Functions

func NewReadCloser 

func NewReadCloser(reader io.Reader, closer io.Closer) io.ReadCloser

func ParseCommentString 

func ParseCommentString(comment string) (compressorFlag, etag, contentType string)

func RegisterBundle 

func RegisterBundle(path string, option ...Option) error

RegisterBundle registers single packed bundle file to repository

func RegisterEmbeddedBundle 

func RegisterEmbeddedBundle(data []byte, name string)

func RegisterEncryptedFolder 

func RegisterEncryptedFolder(path, key string, option ...Option) error

RegisterEncryptedFolder registers folder to repository with decryption key

func RegisterFolder 

func RegisterFolder(path string, option ...Option) error

RegisterFolder registers folder to repository

func SetDecryptoKeyToEmbeddedBundle 

func SetDecryptoKeyToEmbeddedBundle(name, key string) error

SetDecryptoKeyToEmbeddedBundle registers decrypto key for embedded encrypted assets.

func SetDecryptoKeyToExeBundle 

func SetDecryptoKeyToExeBundle(key string) error

SetDecryptoKeyToExeBundle registers decrypto key for bundled assets appended to executable.

func Unload 

func Unload(name string) error

Unload removes assets from default repository. The name is specified by option when registration.

Types

type BundleType 

type BundleType int

type CompressionType 

type CompressionType int

func (CompressionType) Flag 

func (c CompressionType) Flag() string

func (CompressionType) String 

func (c CompressionType) String() string

type Decompressor 

type Decompressor func(io.Reader) io.Reader

type Decryptor 

type Decryptor interface {
	Decrypto(input io.Reader) (io.Reader, error)
}

type EncryptionType 

type EncryptionType int

func (EncryptionType) Flag 

func (e EncryptionType) Flag() string

func (EncryptionType) String 

func (e EncryptionType) String() string

type FileEntry 

type FileEntry interface {
	Reader() (io.ReadCloser, error)
	ReadAll() ([]byte, error)
	BrotliReader() (io.ReadCloser, error)
	Stat() os.FileInfo
	CompressedSize() int64
	Name() string
	Path() string
	EtagAndContentType() (string, string)
	GetLocalPath() (string, error)
}

func Find 

func Find(path string) (FileEntry, error)

Find returns assets in default asset repository.

type ManifestEntry added in v1.1.0 

type ManifestEntry struct {
	File string `json:"file"`
	Sha1 string `json:"sha1"`
	Size int    `json:"size"`
}

type Option 

type Option struct {
	DecryptoKey         string
	MountPoint          string
	Name                string
	Priority            int
	TempFolder          string
	ResetDownloadFolder bool
	ParallelDownload    int
}

type Progress added in v1.1.0 

type Progress struct {
	// contains filtered or unexported fields
}

func (Progress) DeleteFiles added in v1.1.0 

func (p Progress) DeleteFiles() []string

func (Progress) DownloadFiles added in v1.1.0 

func (p Progress) DownloadFiles() []string

func (Progress) KeepFiles added in v1.1.0 

func (p Progress) KeepFiles() []string

func (*Progress) Wait added in v1.1.0 

func (p *Progress) Wait() error

type ROption 

type ROption struct {
	OmitExeBundle          bool
	OmitEmbeddedBundle     bool
	OmitEnvVarFolderBundle bool
}

type ReadCloser 

type ReadCloser struct {
	// contains filtered or unexported fields
}

func (ReadCloser) Close 

func (rc ReadCloser) Close() error

func (ReadCloser) Read 

func (rc ReadCloser) Read(buf []byte) (int, error)

type Repository 

type Repository struct {
	Cache *lru.TwoQueueCache
	// contains filtered or unexported fields
}

func NewRepository 

func NewRepository(option ...ROption) *Repository

func (*Repository) ClearCache 

func (r *Repository) ClearCache()

func (*Repository) Dirs added in v1.1.0 

func (r *Repository) Dirs() []string

func (*Repository) FilesInDir added in v1.1.0 

func (r *Repository) FilesInDir(dirPath string) []string

func (*Repository) Find 

func (r *Repository) Find(candidatePaths ...string) (FileEntry, error)

func (*Repository) RegisterBundle 

func (r *Repository) RegisterBundle(path string, option ...Option) error

func (*Repository) RegisterEncryptedFolder 

func (r *Repository) RegisterEncryptedFolder(path, key string, option ...Option) error

func (*Repository) RegisterFolder 

func (r *Repository) RegisterFolder(path string, option ...Option) error

func (*Repository) RegisterRemoteManifest added in v1.1.0 

func (r *Repository) RegisterRemoteManifest(manifestUrl string, option ...Option) (*Progress, error)

func (*Repository) SetCacheSize 

func (r *Repository) SetCacheSize(size int) error

func (*Repository) SetDecryptoKeyToEmbeddedBundle 

func (r *Repository) SetDecryptoKeyToEmbeddedBundle(name, key string) error

func (*Repository) SetDecryptoKeyToExeBundle 

func (r *Repository) SetDecryptoKeyToExeBundle(key string) error

func (*Repository) Unload 

func (r *Repository) Unload(name string) error

func (*Repository) Walk added in v1.1.0 

func (r *Repository) Walk(root string, walkFn WalkFunc) error

type WalkFunc added in v1.1.0 

type WalkFunc func(path string, info os.FileInfo, err error) error

type WebOption 

type WebOption struct {
	Repository     *Repository
	SPAFallback    string
	MaxAge         time.Duration
	DirectoryIndex string
}

Source Files

View all

Directories

Path Synopsis
cmd
brbundle
chi/spa
chi/with_router
echo/simple
echo/spa
echo/with_router
fasthttp/simple
fasthttp/spa
fasthttp/with_router
gin/spa
gin/with_router
http/simple
http/spa
http/with_router

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL