upload

README

Upload for HTTP servers

Enables you to upload files, such as build artifacts, to your HTTP server instance.

Use this with the built-in authentication, or a different authentication plugin such as jwt.

Licensed under a BSD-style license.

Highlights

• uses HTTP PUT and POST for uploads
• supports HTTP MOVE and DELETE
• imposes limits on filenames:
  • rejects those that are not conforming to Unicode NFC or NFD
  • rejects any comprised of unexpected alphabets ϟ(ツ)╯
• checks request authorization using scheme Signature
• can be configured to silently discard unauthorized requests
• (Linux only) files appear after having been written completely, not before
• limits to file- and transaction sizes independent from any transport encoding

Warnings

Use TLS when uploading, or your data and authorization tokens can be intercepted by third parties and used against you.

This plugin reveals some errors thrown by your filesystem implementation to the uploader, for example about insufficient space on the target device..

The way Golang currently decodes MIME Multipart (which is used with POST requests) results in any files you are uploading being held in memory for the duration of the upload.

Configuration Syntax

Let this be a legible shorthand for instances of ScopeConfiguration:

upload <path> {
	to                    "<directory>"

	enable_webdav
	filenames_form        <none|NFC|NFD>
	filenames_in          <u0000-uff00> [<u0000-uff00>| …]
	random_suffix_len     0..N
	promise_download_from <path>

	max_filesize          0..N
	max_transaction_size  0..N

	hmac_keys_in          <keyid_0=base64(binary)> [<keyid_1=base64(binary)>| …]
	timestamp_tolerance   <0..32>
	silent_auth_errors
}

These settings are required:

• path is the scope below which the plugin will react to any uploads. It will be stripped and no part of any resulting file and directories.
• to is an existing target directory. Must be a quoted absolute path. When using Linux it is recommended to place this on a filesystem which supports O_TMPFILE, such as (but not limited to) ext4 or XFS.

These are optional:

• enable_webdav: Enables other methods than POST and PUT, especially MOVE and DELETE. Is a flag and has no parameters.
  (disable_webdav will no longer be recognized because it's the new default.)

• filenames_form: if given, filenames and directories that are not conforming to Unicode NFC or NFD will be rejected.
  Set this to one of either values when you get errors indicating that your filesystem does not convert names properly. (If in doubt, go with NFC; on Mac PCs with NFD.)
  The default is to not enforce anything.

• filenames_in allows you to limit filenames to specified Unicode ranges. The ranges' bounds must be given in hexadecimal, and start with letter u.
  Use this setting to prevent users from uploading files in, for example, Cyrillic when expect Latin and/or Chinese alphabets only.

• random_suffix_len, if > 0, will result in all filenames getting a randomized suffix.
  The suffix will start in a _ (underscore letter) and placed before any extension.
  For example, image.png will be written as image_a107xm.png with configuration value 6. Utilize promise_download_from to get the resulting filename.
  The default is 0 for off.

• promise_download_from is a string that represents an URI reference, such as a path.
  It will be used to indicate where the uploaded file can be downloaded, by responding with HTTP header Location (multiple times if need be) for all received files.
  You will most probably want to set this to the upload path.
  The default value is "", which means no HTTP header Location will be sent.

• By max_filesize you can limit the size of individual files. Unless set to 0, which means "unlimited" and is the default value, it's in bytes.

• max_transaction_size is similar, but applies to uploads of one or more file in one request. For example, when using MIME Multipart uploads.
  The behaviour with max_filesize > max_transaction_size is currently undefined; set max_transaction_size to a multiple of max_filesize.

Some transfer encodings, such as base64, know comments. Those, or super-long headers and the such, can be exploited to transfer many more bytes than for example max_transaction_size would otherwise allow. Mitigate this by utilizing a different plugin, http.limits, which counts incoming bytes ignorant of any encoding. Set a limit of about 1.4× to 2.05× the max_transaction_size.
This plugin writes files blockwise for a better performance. Limits are rounded up by a few kilobytes to the next full block.

Optional, but required if you want to use the built-in authorization feature:

• hmac_keys_in is a space-delimited list of username → shared secret associations.
  The latter is binary data, encoded using base64, with a recommended length of 32 octets.
• timestamp_tolerance sets the validity of a request with authorization, and is used to account for clock drift difference between the uploader's and the server's computer.
  It is given as power of 2, and its default value is 2 (as in: ± four seconds = 1<<2 = 2**2). Set this to 1 or 0 with reliably synchronized clocks.
• silent_auth_errors, if set the plugin's built-in authorization will return no HTTP errors of its own.
  Instead, the request will be handed over to the next middleware, which then will most probably return a HTTP error. This is a cheap way to obscure that your site accepts uploads.

Tutorial

Setup a minimal configuration like this, or go run example.go & after copying it into a separate directory and removing the line with +build ignore (mind the port number, which is 9000 there):

upload /web/path {
 	to "/var/tmp"
 	enable_webdav
}

… and upload one file:

# HTTP PUT
curl \
  -T /etc/os-release \
  https://127.0.0.1/web/path/from-release

… or more files in one go (sub-directories will be created as needed):

# HTTP POST
curl \
  -F gitconfig=@.gitconfig \
  -F id_ed25519.pub=@.ssh/id_ed25519.pub \
  https://127.0.0.1/web/path/

… which you then can move and delete like this:

# MOVE is 'mv'
curl -X MOVE \
  -H "Destination: /web/path/to-release" \
  https://127.0.0.1/web/path/from-release

# DELETE is 'rm -r'
curl -X DELETE \
  https://127.0.0.1/web/path/to-release

Authorization: Signature

This plugin comes with support for request authorization scheme Signature, though does not support its realms or any other algorithm than hmac-sha256.
It's a HMAC scheme with a pre-shared secret and nonce||timestamp.

Send header Authorization and two other, formatted like this, with every upload request:

Authorization: Signature keyId="(username)",algorithm="hmac-sha256",headers="timestamp token",signature="(see below)"
Timestamp: (current UNIX time)
Token: (a nonce)

You can generate new keys (passwords, pre-shared keys) using BASH and OpenSSL, and encode them to base64:

SECRET="$(openssl rand -base64 32)"

# printf "%s\n" "${SECRET}"
# TWF0dCBIb2x0IGRvZXNuJ3QgdXBkYXRlIGhpcyBNYWM=

A full script for uploading something would be:

#!/bin/bash
# hmac_keys_in mark=Z2VoZWlt
#
UPLOADER="mark"
SECRET="geheim"

TIMESTAMP="$(date --utc +%s)"
# length and contents are not important, "abcdef" would work as well
NONCE="$(cat /dev/urandom | tr -d -c '[:alnum:]' | head -c $(( 32 - ${#TIMESTAMP} )))"

SIGNATURE="$(printf "${TIMESTAMP}${NONCE}" \
             | openssl dgst -sha256 -hmac "${SECRET}" -binary \
             | openssl enc -base64)"

# order does not matter; any skipped fields in Authorization will be set to defaults
exec curl -T \
  --header "Timestamp: ${TIMESTAMP}" \
  --header "Token: ${NONCE}" \
  --header "Authorization: Signature keyId='${UPLOADER}',signature='${SIGNATURE}'" \
  "<filename>" "<url>"

Configuration Examples

A host used by someone in Central and West Europe would be configured like this to accept filenames in Latin with some Greek runes and a few mathematical symbols:

upload /college/curriculum {
	to "/home/ellen_baker/inbox"
	filenames_form NFC
	filenames_in u0000–u007F u0100–u017F u0391–u03C9 u2018–u203D u2152–u217F
}

A host for Linux distribution packages can be more restrictive:

upload /binhost/gentoo {
	to "/var/portage/packages"
	filenames_in u0000–u007F
	timestamp_tolerance 0
}
tls {
	…
	clientcas /etc/ssl/dist-uploaders-CA.crt
}

… while someone in East Asia would share space on his blog with three friends like this:

upload /wp-uploads {
	to "/var/www/senpai/wp-uploads"
	enable_webdav
	max_filesize 16777216
	filenames_in u0000–u007F u0100–u017F u0391–u03C9 u2018–u203D u3000–u303f u3040–u309f u30a0–u30ff u4e00–9faf uff00–uffef

	timestamp_tolerance 3
	silent_auth_errors

	hmac_keys_in yui=eXVp hina=aGluYQ== olivia=b2xpdmlh james=amFtZXM=
}

See Also

You can find a very nice overview of Unicode Character ranges here:
http://jrgraphix.net/research/unicode_blocks.php

Here is the official list of Unicode blocks:
http://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt

For Authorization: Signature please see:

• https://tools.ietf.org/html/draft-cavage-http-signatures-05
• github.com/joyent/gosign is an implementation in Golang,
• github.com/joyent/node-http-signature for Node.js.

Documentation

Overview

Package upload contains a HTTP handler that provides facilities for uploading files.

Use flags for http server implementations other than Go's own, like this:

go build -tags "caddyserver0.9 caddyserver1.0" …

Those tags start with the first version, followed by all major.minor up to its current version. Please see how Go does it: https://golang.org/pkg/go/build/#hdr-Build_Constraints

Absent any meaningful flags use the http.Handler implementation (see the following example).

Example

var (
	scope     = "/" // prefix to http.Request.URL.Path
	directory = "/var/tmp"
	next      = http.FileServer(http.Dir(directory))
)

cfg := NewDefaultConfiguration(directory)
uploadHandler, _ := NewHandler(scope, cfg, next)

http.Handle(scope, uploadHandler)
// http.ListenAndServe(":9000", nil)

Index

• Constants
• func FinishSetup() error
• func IsAcceptableFilename(s string, reduceAcceptableRunesTo []*unicode.RangeTable, ...) bool
• func ParseUnicodeBlockList(str string) (*unicode.RangeTable, error)
• type Handler
  • func NewHandler(scope string, config *ScopeConfiguration, next http.Handler) (*Handler, error)
  • func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)
• type ScopeConfiguration
  • func NewDefaultConfiguration(targetDirectory string) *ScopeConfiguration

Examples

• Package

Constants

const (
	// AlwaysRejectRunes contains runes that are not safe to use with network shares.
	//
	// Please note that '/' is already discarded at an earlier stage.
	AlwaysRejectRunes = `"*:<>?|\`
)

Functions

func FinishSetup

func FinishSetup() error

FinishSetup communicates any collected white- and blacklists to the operating system.

Call this on systems and with seervers that support OS-based locking down. Servers that feature a dynamic reconfiguration or the like should not call this. Do not call this in unit tests.

Subject to change, has only an effect on OpenBSD.

func IsAcceptableFilename

func IsAcceptableFilename(s string, reduceAcceptableRunesTo []*unicode.RangeTable,
	enforceForm *norm.Form) bool

IsAcceptableFilename is used to enforce filenames in wanted alphabet(s). Setting 'reduceAcceptableRunesTo' reduces the supremum unicode.PrintRanges.

A string with runes – other than U+0020 (space) or U+2009 (spatium) – representing space will be rejected.

Filenames are not transliterated to prevent loops within clusters of mirrors.

func ParseUnicodeBlockList

func ParseUnicodeBlockList(str string) (*unicode.RangeTable, error)

ParseUnicodeBlockList naïvely translates a string with space-delimited Unicode ranges to Go's unicode.RangeTable.

All elements must fit into uint32. A Range must begin with its lower bound, and ranges must not overlap (we don't check this here!).

The format of one range is as follows, with 'stride' being set to '1' if left empty.

<low>-<high>[:<stride>]

Types

type Handler

type Handler struct {
	Next   http.Handler
	Config *ScopeConfiguration
	Scope  string // Basically this will be stripped from the full URL and the target path swapped in.
}

Handler implements http.Handler.

func NewHandler

func NewHandler(scope string, config *ScopeConfiguration, next http.Handler) (*Handler, error)

NewHandler creates a new instance of this plugin's upload handler, meant to be used in Go's own http server.

XXX: Its responsibility is to reject invalid or formally incorrect configurations.

'next' is optional. 'scope' is a string and the prefix of the upload destination's URL.Path, like `/dir/to/upload/destination`.

func (Handler) ServeHTTP

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP handles any uploads, else defers the request to the next handler.

type ScopeConfiguration

type ScopeConfiguration struct {
	// How big a difference between 'now' and the provided timestamp do we tolerate?
	// In seconds. Due to possible optimizations this should be an order of 2.
	// A reasonable default is 1<<2.
	TimestampTolerance uint64

	MaxFilesize        uint64
	MaxTransactionSize uint64

	// Target directory on disk that serves as upload destination.
	WriteToPath string

	// Uploaded files can be gotten back from here.
	// If ≠ "" this will trigger sending headers such as "Location".
	ApparentLocation string

	// Maps KeyIDs to shared secrets.
	// Here the latter are already decoded from base64 to binary.
	// Request verification is disabled if this is empty.
	IncomingHmacSecrets     auth.HmacSecrets
	IncomingHmacSecretsLock sync.RWMutex

	// If false, this plugin returns HTTP Errors.
	// If true, passes the given request to the next middleware
	// which could respond with an Error of its own, poorly obscuring where this plugin is used.
	SilenceAuthErrors bool

	// Enables MOVE, DELETE, and similar. Without this only POST and PUT will be recognized.
	EnableWebdav bool

	// Set this to reject any non-conforming filenames.
	UnicodeForm *struct{ Use norm.Form }

	// Limit the acceptable alphabet(s) for filenames by setting this value.
	RestrictFilenamesTo []*unicode.RangeTable

	// Append '_' and a randomized suffix of that length.
	RandomizedSuffixLength uint32
}

ScopeConfiguration represents the settings of a scope (URL path).

func NewDefaultConfiguration

func NewDefaultConfiguration(targetDirectory string) *ScopeConfiguration

NewDefaultConfiguration creates a new default configuration.