weasel

README

yummy-weasel

A simple frontend (App Engine app) that serves content from a Google Cloud Storage (GCS) bucket, while allowing for:

• HTTPS on custom domain
• support for Service Worker (works only over HTTPS)
• HTTP/2 push
• more robust redirect from naked custom domain
• serving directly from naked custom domain
• keep the deployment/publishing flow using just the existing GCS bucket, which has proven to be fast and reliable, with partial content updates.
• dynamic server-side logic

design

The design is simple. Suppose you have a static website www.example.com, served directly from a GCS gs://www.example.com.

A very simplified picture of the serving path can be shown as follows.

+-----+    GET http://www.example.com/dir/index.html
| GCS |  <-----------------------------------------+  visitor
+-----+                                                 :-/

Weasel enhancement consists of modifying the serving path to:

+-----+   (2) GET gs://www.example.com/dir/index.html
| GCS |  <------------------------------+--------+
+-----+                                 | yummy  |
                                        | weasel |
                                        +--------+
                                           ↑  |
                                      (1)  |  | (3)
         GET https://www.example.com/dir/  |  ↓ Link: <asset>; rel=preload

                                         visitor
                                           :-)

1. A visitor requests the website page. Note that /index.html is now optional. Due to GCS restrictions, such suffixes were previously required for sub-folders, but with this approach they no longer need to be specified. Also, requests can (and will) be made over HTTPS.

2. Weasel fetches the object content from the original GCS bucket and caches it locally. This step is necessary only if the object hasn't been cached already or the cache has expired. Cache expiration and invalidation is based on GCS object cache-control header settings.

3. Weasel responds with the GCS object contents. Note that we can optionally push additional assets related to the requested file by using Link: <asset>; rel=preload header supported by GFE.

dev flow

Just use goapp tool provided with the Google Appengine SDK for Go.

Assuming the SDK is installed in $SDK_DIR:

• goapp test runs tests
• goapp deploy deploys the app to App Engine production servers.

Documentation

Overview

Package weasel provides a simple frontend (App Engine app) that serves content from a Google Cloud Storage (GCS) bucket. See README.md for the design details.

Index

• Constants
• Variables
• func ServeObject(w http.ResponseWriter, o *Object, withBody bool) error
• func ValidMethod(m string) bool
• type FetchError
  • func (e *FetchError) Error() string
• type Object
  • func (o *Object) Redirect() string
  • func (o *Object) RedirectCode() int
• type Storage
  • func (s *Storage) CacheKey(bucket, name string) string
  • func (s *Storage) HandleChangeHook(w http.ResponseWriter, r *http.Request)
  • func (s *Storage) PurgeCache(ctx context.Context, bucket, name string) error
  • func (s *Storage) ReadFile(ctx context.Context, bucket, name string) (*Object, error)
  • func (s *Storage) ReadObject(ctx context.Context, bucket, name string) (*Object, error)
  • func (s *Storage) Stat(ctx context.Context, bucket, name string) (*Object, error)

Constants

const (
	ScopeStorageRead  = "https://www.googleapis.com/auth/devstorage.read_only"
	ScopeStorageOwner = "https://www.googleapis.com/auth/devstorage.full_control"
)

Google Cloud Storage OAuth2 scopes

Variables

var DefaultStorage = Storage{
	Base:  "https://storage.googleapis.com",
	Index: "index.html",
}

DefaultStorage is a Storage with sensible default parameters suitable for prod environments on App Engine.

Functions

func ServeObject

func ServeObject(w http.ResponseWriter, o *Object, withBody bool) error

ServeObject writes object o to w, with optional body.

func ValidMethod

func ValidMethod(m string) bool

ValidMethod reports whether m is a supported HTTP method.

Types

type FetchError

type FetchError struct {
	Msg  string
	Code int
}

FetchError contains error code and message from a GCS response.

func (*FetchError) Error

func (e *FetchError) Error() string

Error returns formatted FetchError.

type Object

type Object struct {
	Meta map[string]string
	Body []byte
}

Object represents a single GCS object.

func (*Object) Redirect

func (o *Object) Redirect() string

Redirect returns o's redirect URL, zero string otherwise.

func (*Object) RedirectCode

func (o *Object) RedirectCode() int

RedirectCode returns o's HTTP response code for redirect. It defaults to http.StatusMovedPermanently.

type Storage

type Storage struct {
	Base  string // GCS service base URL, e.g. "https://storage.googleapis.com".
	Index string // Appended to an object name in certain cases, e.g. "index.html".
}

Storage incapsulates configuration params for retrieveing and serving GCS objects.

func (*Storage) CacheKey

func (s *Storage) CacheKey(bucket, name string) string

CacheKey returns a key to cache an object under, computed from s.Base, bucket and then name.

func (*Storage) HandleChangeHook

func (s *Storage) HandleChangeHook(w http.ResponseWriter, r *http.Request)

HandleChangeHook handles Object Change Notifications as described at https://cloud.google.com/storage/docs/object-change-notification. It removes objects from cache.

func (*Storage) PurgeCache

func (s *Storage) PurgeCache(ctx context.Context, bucket, name string) error

PurgeCache removes cached object from memcache. It does not return an error in the case of cache miss.

func (*Storage) ReadFile

func (s *Storage) ReadFile(ctx context.Context, bucket, name string) (*Object, error)

ReadFile abstracts ReadObject and treats object name like a file path.

func (*Storage) ReadObject

func (s *Storage) ReadObject(ctx context.Context, bucket, name string) (*Object, error)

ReadObject retrieves GCS object name of the bucket from cache or network. Objects fetched from the network are cached before returning from this function.

func (*Storage) Stat

func (s *Storage) Stat(ctx context.Context, bucket, name string) (*Object, error)

Stat is similar to Read except the returned object.Body may be nil.