tusd

package module
v0.0.0-...-1fe2c51 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 5, 2019 License: MIT Imports: 17 Imported by: 0

README

tusd

Tus logo

tus is a protocol based on HTTP for resumable file uploads. Resumable means that an upload can be interrupted at any moment and can be resumed without re-uploading the previous data again. An interruption may happen willingly, if the user wants to pause, or by accident in case of an network issue or server outage.

tusd is the official reference implementation of the tus resumable upload protocol. The protocol specifies a flexible method to upload files to remote servers using HTTP. The special feature is the ability to pause and resume uploads at any moment allowing to continue seamlessly after e.g. network interruptions.

It is capable of accepting uploads with arbitrary sizes and storing them locally on disk, on Google Cloud Storage or on AWS S3 (or any other S3-compatible storage system). Due to its modularization and extensibility, support for nearly any other cloud provider could easily be added to tusd.

Protocol version: 1.0.0

Getting started

You can download ready-to-use packages including binaries for OS X, Linux and Windows in various formats of the latest release.

Compile from source

The only requirement for building tusd is Go 1.5 or newer. If you meet this criteria, you can clone the git repository, install the remaining dependencies and build the binary:

git clone git@github.com:tus/tusd.git
cd tusd

go get -u github.com/aws/aws-sdk-go/...
go get -u github.com/prometheus/client_golang/prometheus

go build -o tusd cmd/tusd/main.go

Running tusd

Start the tusd upload server is as simple as invoking a single command. For example, following snippet demonstrates how to start a tusd process which accepts tus uploads at http://localhost:1080/files/ (notice the trailing slash) and stores them locally in the ./data directory:

$ tusd -dir=./data
[tusd] Using './data' as directory storage.
[tusd] Using 0.00MB as maximum size.
[tusd] Using 0.0.0.0:1080 as address to listen.
[tusd] Using /files/ as the base path.
[tusd] Using /metrics as the metrics path.

Alternatively, if you want to store the uploads on an AWS S3 bucket, you only have to specify the bucket and provide the corresponding access credentials and region information using environment variables (if you want to use a S3-compatible store, use can use the -s3-endpoint option):

$ export AWS_ACCESS_KEY_ID=xxxxx
$ export AWS_SECRET_ACCESS_KEY=xxxxx
$ export AWS_REGION=eu-west-1
$ tusd -s3-bucket=my-test-bucket.com
[tusd] Using 's3://my-test-bucket.com' as S3 bucket for storage.
[tusd] Using 0.00MB as maximum size.
[tusd] Using 0.0.0.0:1080 as address to listen.
[tusd] Using /files/ as the base path.
[tusd] Using /metrics as the metrics path.

tusd is also able to read the credentials automatically from a shared credentials file (~/.aws/credentials) as described in https://github.com/aws/aws-sdk-go#configuring-credentials.

Furthermore, tusd also has support for storing uploads on Google Cloud Storage. In order to enable this feature, supply the path to your account file containing the necessary credentials:

$ export GCS_SERVICE_ACCOUNT_FILE=./account.json
$ tusd -gcs-bucket=my-test-bucket.com
[tusd] Using 'gcs://my-test-bucket.com' as GCS bucket for storage.
[tusd] Using 0.00MB as maximum size.
[tusd] Using 0.0.0.0:1080 as address to listen.
[tusd] Using /files/ as the base path.
[tusd] Using /metrics as the metrics path.

Besides these simple examples, tusd can be easily configured using a variety of command line options:

$ tusd -help
Usage of tusd:
  -base-path string
    	Basepath of the HTTP server (default "/files/")
  -behind-proxy
    	Respect X-Forwarded-* and similar headers which may be set by proxies
  -dir string
    	Directory to store uploads in (default "./data")
  -expose-metrics
    	Expose metrics about tusd usage (default true)
  -gcs-bucket string
    	Use Google Cloud Storage with this bucket as storage backend (requires the GCS_SERVICE_ACCOUNT_FILE environment variable to be set)
  -hooks-dir string
    	Directory to search for available hooks scripts
  -hooks-enabled-events string
    	Comma separated list of enabled hook events (e.g. post-create,post-finish). Leave empty to enable all events
  -hooks-http string
    	An HTTP endpoint to which hook events will be sent to
  -hooks-http-backoff int
    	Number of seconds to wait before retrying each retry (default 1)
  -hooks-http-retry int
    	Number of times to retry on a 500 or network timeout (default 3)
  -host string
    	Host to bind HTTP server to (default "0.0.0.0")
  -max-size int
    	Maximum size of a single upload in bytes
  -metrics-path string
    	Path under which the metrics endpoint will be accessible (default "/metrics")
  -port string
    	Port to bind HTTP server to (default "1080")
  -s3-bucket string
    	Use AWS S3 with this bucket as storage backend (requires the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION environment variables to be set)
  -s3-endpoint string
    	Endpoint to use S3 compatible implementations like minio (requires s3-bucket to be pass)
  -store-size int
    	Size of space allowed for storage
  -timeout int
    	Read timeout for connections in milliseconds.  A zero value means that reads will not timeout (default 30000)
  -version
    	Print tusd version information

Monitoring tusd

tusd exposes metrics at the /metrics endpoint (example) in the Prometheus Text Format. This allows you to hook up Prometheus or any other compatible service to your tusd instance and let it monitor tusd. Alternatively, there are many parsers and client libraries available for consuming the metrics format directly.

The endpoint contains details about Go's internals, general HTTP numbers and details about tus uploads and tus-specific errors. It can be completely disabled using the -expose-metrics false flag and it's path can be changed using the -metrics-path /my/numbers flag.

Using tusd manually

Besides from running tusd using the provided binary, you can embed it into your own Go program:

package main

import (
	"fmt"
	"net/http"

	"github.com/kyl2016/tusd"
	"github.com/kyl2016/tusd/filestore"
)

func main() {
	// Create a new FileStore instance which is responsible for
	// storing the uploaded file on disk in the specified directory.
	// This path _must_ exist before tusd will store uploads in it.
	// If you want to save them on a different medium, for example
	// a remote FTP server, you can implement your own storage backend
	// by implementing the tusd.DataStore interface.
	store := filestore.FileStore{
		Path: "./uploads",
	}

	// A storage backend for tusd may consist of multiple different parts which
	// handle upload creation, locking, termination and so on. The composer is a
	// place where all those separated pieces are joined together. In this example
	// we only use the file store but you may plug in multiple.
	composer := tusd.NewStoreComposer()
	store.UseIn(composer)

	// Create a new HTTP handler for the tusd server by providing a configuration.
	// The StoreComposer property must be set to allow the handler to function.
	handler, err := tusd.NewHandler(tusd.Config{
		BasePath:      "/files/",
		StoreComposer: composer,
	})
	if err != nil {
		panic(fmt.Errorf("Unable to create handler: %s", err))
	}

	// Right now, nothing has happened since we need to start the HTTP server on
	// our own. In the end, tusd will start listening on and accept request at
	// http://localhost:8080/files
	http.Handle("/files/", http.StripPrefix("/files/", handler))
	err = http.ListenAndServe(":8080", nil)
	if err != nil {
		panic(fmt.Errorf("Unable to listen: %s", err))
	}
}

Please consult the online documentation for more details about tusd's APIs and its sub-packages.

Implementing own storages

The tusd server is built to be as flexible as possible and to allow the use of different upload storage mechanisms. By default the tusd binary includes filestore which will save every upload to a specific directory on disk.

If you have different requirements, you can build your own storage backend which will save the files to S3, a remote FTP server or similar. Doing so is as simple as implementing the tusd.DataStore interface and using the new struct in the configuration object. Please consult the documentation about detailed information about the required methods.

Packages

This repository does not only contain the HTTP server's code but also other useful tools:

  • s3store: A storage backend using AWS S3
  • filestore: A storage backend using the local file system
  • gcsstore: A storage backend using Google cloud storage
  • memorylocker: An in-memory locker for handling concurrent uploads
  • consullocker: A locker using the distributed Consul service
  • etcd3locker: A locker using the distributed KV etcd3 store
  • limitedstore: A storage wrapper limiting the total used space for uploads
3rd-Party tusd Packages

The following packages are supported by 3rd-party maintainers outside of this repository. Please file issues respective to the packages in their respective repositories.

Running the testsuite

Build Status Build status

go test -v ./...

FAQ

How can I access tusd using HTTPS?

The tusd binary, once executed, listens on the provided port for only non-encrypted HTTP requests and does not accept HTTPS connections. This decision has been made to limit the functionality inside this repository which has to be developed, tested and maintained. If you want to send requests to tusd in a secure fashion - what we absolutely encourage, we recommend you to utilize a reverse proxy in front of tusd which accepts incoming HTTPS connections and forwards them to tusd using plain HTTP. More information about this topic, including sample configurations for Nginx and Apache, can be found in issue #86 and in the Apache example configuration.

Can I run tusd behind a reverse proxy?

Yes, it is absolutely possible to do so. Firstly, you should execute the tusd binary using the -behind-proxy flag indicating it to pay attention to special headers which are only relevant when used in conjunction with a proxy. Furthermore, there are additional details which should be kept in mind, depending on the used software:

  • Disable request buffering. Nginx, for example, reads the entire incoming HTTP request, including its body, before sending it to the backend, by default. This behavior defeats the purpose of resumability where an upload is processed while it's being transfered. Therefore, such as feature should be disabled.

  • Adjust maximum request size. Some proxies have default values for how big a request may be in order to protect your services. Be sure to check these settings to match the requirements of your application.

  • Forward hostname and scheme. If the proxy rewrites the request URL, the tusd server does not know the original URL which was used to reach the proxy. This behavior can lead to situations, where tusd returns a redirect to a URL which can not be reached by the client. To avoid this confusion, you can explicitly tell tusd which hostname and scheme to use by supplying the X-Forwarded-Host and X-Forwarded-Proto headers.

Explicit examples for the above points can be found in the Nginx configuration which is used to power the master.tus.io instace.

Can I run custom verification/authentication checks before an upload begins?

Yes, this is made possible by the hook system inside the tusd binary. It enables custom routines to be executed when certain events occurs, such as a new upload being created which can be handled by the pre-create hook. Inside the corresponding hook file, you can run your own validations against the provided upload metadata to determine whether the action is actually allowed or should be rejected by tusd. Please have a look at the corresponding documentation for a more detailed explanation.

Can I run tusd inside a VM/Vagrant/VirtualBox?

Yes, you can absolutely do so without any modifications. However, there is one known problem: If you are using tusd inside VirtualBox (the default provider for Vagrant) and are storing the files inside a shared/synced folder, you might get TemporaryErrors (Lockfile created, but doesn't exist) when trying to upload. This happens because shared folders do not support hard links which are necessary for tusd. Please use another non-shared folder for storing files (see https://github.com/kyl2016/tusd/issues/201).

I am getting TemporaryErrors (Lockfile created, but doesn't exist)! What can I do?

This error can occur when you are running tusd's disk storage on a file system which does not support hard links. These hard links are used to create lock files for ensuring that an upload's data is consistent. For example, this problem can happen when running tusd inside VirtualBox (see the answer above for more details) or when using file system interfaces to cloud storage providers (see https://github.com/kyl2016/tusd/issues/257). We recommend you to ensure that your file system supports hard links, use a different file system, or use one of tusd's cloud storage abilities. If the problem still persists, please open a bug report.

How can I prevent users from downloading the uploaded files?

tusd allows any user to retrieve a previously uploaded file by issuing a HTTP GET request to the corresponding upload URL. This is possible as long as the uploaded files on the datastore have not been deleted or moved to another location. While it is a handy feature for debugging and testing your setup, we know that there are situations where you don't want to allow downloads or where you want more control about who downloads what. In these scenarios we recommend to place a proxy in front of tusd which takes on the task of access control or even preventing HTTP GET requests entirely. tusd has no feature built in for controling or disabling downloads on its own because the main focus is on accepting uploads, not serving files.

License

This project is licensed under the MIT license, see LICENSE.txt.

Documentation

Overview

Package tusd provides ways to accept tus 1.0 calls using HTTP.

tus is a protocol based on HTTP for resumable file uploads. Resumable means that an upload can be interrupted at any moment and can be resumed without re-uploading the previous data again. An interruption may happen willingly, if the user wants to pause, or by accident in case of an network issue or server outage (http://tus.io).

The basics of tusd

tusd was designed in way which allows an flexible and customizable usage. We wanted to avoid binding this package to a specific storage system – particularly a proprietary third-party software. Therefore tusd is an abstract layer whose only job is to accept incoming HTTP requests, validate them according to the specification and finally passes them to the data store.

The data store is another important component in tusd's architecture whose purpose is to do the actual file handling. It has to write the incoming upload to a persistent storage system and retrieve information about an upload's current state. Therefore it is the only part of the system which communicates directly with the underlying storage system, whether it be the local disk, a remote FTP server or cloud providers such as AWS S3.

Using a store composer

The only hard requirements for a data store can be found in the DataStore interface. It contains methods for creating uploads (NewUpload), writing to them (WriteChunk) and retrieving their status (GetInfo). However, there are many more features which are not mandatory but may still be used. These are contained in their own interfaces which all share the *DataStore suffix. For example, GetReaderDataStore which enables downloading uploads or TerminaterDataStore which allows uploads to be terminated.

The store composer offers a way to combine the basic data store - the core - implementation and these additional extensions: 

composer := tusd.NewStoreComposer()
composer.UseCore(dataStore) // Implements DataStore
composer.UseTerminater(terminater) // Implements TerminaterDataStore
composer.UseLocker(locker) // Implements LockerDataStore

The corresponding methods for adding an extension to the composer are prefixed with Use* followed by the name of the corresponding interface. However, most data store provide multiple extensions and adding all of them manually can be tedious and error-prone. Therefore, all data store distributed with tusd provide an UseIn() method which does this job automatically. For example, this is the S3 store in action (see S3Store.UseIn): 

store := s3store.New(…)
locker := memorylocker.New()
composer := tusd.NewStoreComposer()
store.UseIn(composer)
locker.UseIn(composer)

Finally, once you are done with composing your data store, you can pass it inside the Config struct in order to create create a new tusd HTTP handler: 

config := tusd.Config{
  StoreComposer: composer,
  BasePath: "/files/",
}
handler, err := tusd.NewHandler(config)

This handler can then be mounted to a specific path, e.g. /files: 

http.Handle("/files/", http.StripPrefix("/files/", handler))

Index

Examples

Constants

View Source 
const UploadLengthDeferred = "1"

Variables

View Source 
var (
	ErrUnsupportedVersion               = NewHTTPError(errors.New("unsupported version"), http.StatusPreconditionFailed)
	ErrMaxSizeExceeded                  = NewHTTPError(errors.New("maximum size exceeded"), http.StatusRequestEntityTooLarge)
	ErrInvalidContentType               = NewHTTPError(errors.New("missing or invalid Content-Type header"), http.StatusBadRequest)
	ErrInvalidUploadLength              = NewHTTPError(errors.New("missing or invalid Upload-Length header"), http.StatusBadRequest)
	ErrInvalidOffset                    = NewHTTPError(errors.New("missing or invalid Upload-Offset header"), http.StatusBadRequest)
	ErrNotFound                         = NewHTTPError(errors.New("upload not found"), http.StatusNotFound)
	ErrFileLocked                       = NewHTTPError(errors.New("file currently locked"), 423) // Locked (WebDAV) (RFC 4918)
	ErrMismatchOffset                   = NewHTTPError(errors.New("mismatched offset"), http.StatusConflict)
	ErrSizeExceeded                     = NewHTTPError(errors.New("resource's size exceeded"), http.StatusRequestEntityTooLarge)
	ErrNotImplemented                   = NewHTTPError(errors.New("feature not implemented"), http.StatusNotImplemented)
	ErrUploadNotFinished                = NewHTTPError(errors.New("one of the partial uploads is not finished"), http.StatusBadRequest)
	ErrInvalidConcat                    = NewHTTPError(errors.New("invalid Upload-Concat header"), http.StatusBadRequest)
	ErrModifyFinal                      = NewHTTPError(errors.New("modifying a final upload is not allowed"), http.StatusForbidden)
	ErrUploadLengthAndUploadDeferLength = NewHTTPError(errors.New("provided both Upload-Length and Upload-Defer-Length"), http.StatusBadRequest)
	ErrInvalidUploadDeferLength         = NewHTTPError(errors.New("invalid Upload-Defer-Length header"), http.StatusBadRequest)
	ErrUploadStoppedByServer            = NewHTTPError(errors.New("upload has been stopped by server"), http.StatusBadRequest)
)

Functions

func LogEvent 

func LogEvent(logger *log.Logger, eventName string, details ...string)

func ParseMetadataHeader 

func ParseMetadataHeader(header string) map[string]string

ParseMetadataHeader parses the Upload-Metadata header as defined in the File Creation extension. e.g. Upload-Metadata: name bHVucmpzLnBuZw==,type aW1hZ2UvcG5n

func SerializeMetadataHeader 

func SerializeMetadataHeader(meta map[string]string) string

SerializeMetadataHeader serializes a map of strings into the Upload-Metadata header format used in the response for HEAD requests. e.g. Upload-Metadata: name bHVucmpzLnBuZw==,type aW1hZ2UvcG5n

Types

type ConcaterDataStore 

type ConcaterDataStore interface {
	// ConcatUploads concatenations the content from the provided partial uploads
	// and write the result in the destination upload which is specified by its
	// ID. The caller (usually the handler) must and will ensure that this
	// destination upload has been created before with enough space to hold all
	// partial uploads. The order, in which the partial uploads are supplied,
	// must be respected during concatenation.
	ConcatUploads(destination string, partialUploads []string) error
}

ConcaterDataStore is the interface required to be implemented if the Concatenation extension should be enabled. Only in this case, the handler will parse and respect the Upload-Concat header.

type Config 

type Config struct {
	// DataStore implementation used to store and retrieve the single uploads.
	// The usage of this field is deprecated and should be avoided in favor of
	// StoreComposer.
	DataStore DataStore
	// StoreComposer points to the store composer from which the core data store
	// and optional dependencies should be taken. May only be nil if DataStore is
	// set.
	StoreComposer *StoreComposer
	// MaxSize defines how many bytes may be stored in one single upload. If its
	// value is is 0 or smaller no limit will be enforced.
	MaxSize int64
	// BasePath defines the URL path used for handling uploads, e.g. "/files/".
	// If no trailing slash is presented it will be added. You may specify an
	// absolute URL containing a scheme, e.g. "http://tus.io"
	BasePath string

	// NotifyCompleteUploads indicates whether sending notifications about
	// completed uploads using the CompleteUploads channel should be enabled.
	NotifyCompleteUploads bool
	// NotifyTerminatedUploads indicates whether sending notifications about
	// terminated uploads using the TerminatedUploads channel should be enabled.
	NotifyTerminatedUploads bool
	// NotifyUploadProgress indicates whether sending notifications about
	// the upload progress using the UploadProgress channel should be enabled.
	NotifyUploadProgress bool
	// NotifyCreatedUploads indicates whether sending notifications about
	// the upload having been created using the CreatedUploads channel should be enabled.
	NotifyCreatedUploads bool
	// Logger is the logger to use internally, mostly for printing requests.
	Logger *log.Logger
	// Respect the X-Forwarded-Host, X-Forwarded-Proto and Forwarded headers
	// potentially set by proxies when generating an absolute URL in the
	// response to POST requests.
	RespectForwardedHeaders bool
	// contains filtered or unexported fields
}

Config provides a way to configure the Handler depending on your needs.

type DataStore 

type DataStore interface {
	// Create a new upload using the size as the file's length. The method must
	// return an unique id which is used to identify the upload. If no backend
	// (e.g. Riak) specifes the id you may want to use the uid package to
	// generate one. The properties Size and MetaData will be filled.
	NewUpload(info FileInfo) (id string, err error)
	// Write the chunk read from src into the file specified by the id at the
	// given offset. The handler will take care of validating the offset and
	// limiting the size of the src to not overflow the file's size. It may
	// return an os.ErrNotExist which will be interpreted as a 404 Not Found.
	// It will also lock resources while they are written to ensure only one
	// write happens per time.
	// The function call must return the number of bytes written.
	WriteChunk(id string, offset int64, src io.Reader) (int64, error)
	// Read the fileinformation used to validate the offset and respond to HEAD
	// requests. It may return an os.ErrNotExist which will be interpreted as a
	// 404 Not Found.
	GetInfo(id string) (FileInfo, error)
}

type ErrorsTotalMap 

type ErrorsTotalMap struct {
	// contains filtered or unexported fields
}

ErrorsTotalMap stores the counters for the different HTTP errors.

func (*ErrorsTotalMap) Load 

func (e *ErrorsTotalMap) Load() map[HTTPError]*uint64

Load retrieves the map of the counter pointers atomically

type FileInfo 

type FileInfo struct {
	ID string
	// Total file size in bytes specified in the NewUpload call
	Size int64
	// Indicates whether the total file size is deferred until later
	SizeIsDeferred bool
	// Offset in bytes (zero-based)
	Offset   int64
	MetaData MetaData
	// Indicates that this is a partial upload which will later be used to form
	// a final upload by concatenation. Partial uploads should not be processed
	// when they are finished since they are only incomplete chunks of files.
	IsPartial bool
	// Indicates that this is a final upload
	IsFinal bool
	// If the upload is a final one (see IsFinal) this will be a non-empty
	// ordered slice containing the ids of the uploads of which the final upload
	// will consist after concatenation.
	PartialUploads []string
	// contains filtered or unexported fields
}

func (FileInfo) StopUpload 

func (f FileInfo) StopUpload()

StopUpload interrupts an running upload from the server-side. This means that the current request body is closed, so that the data store does not get any more data. Furthermore, a response is sent to notify the client of the interrupting and the upload is terminated (if supported by the data store), so the upload cannot be resumed anymore.

type FinisherDataStore 

type FinisherDataStore interface {
	// FinishUpload executes additional operations for the finished upload which
	// is specified by its ID.
	FinishUpload(id string) error
}

FinisherDataStore is the interface which can be implemented by DataStores which need to do additional operations once an entire upload has been completed. These tasks may include but are not limited to freeing unused resources or notifying other services. For example, S3Store uses this interface for removing a temporary object.

type GetReaderDataStore 

type GetReaderDataStore interface {
	// GetReader returns a reader which allows iterating of the content of an
	// upload specified by its ID. It should attempt to provide a reader even if
	// the upload has not been finished yet but it's not required.
	// If the returned reader also implements the io.Closer interface, the
	// Close() method will be invoked once everything has been read.
	// If the given upload could not be found, the error tusd.ErrNotFound should
	// be returned.
	GetReader(id string) (io.Reader, error)
}

GetReaderDataStore is the interface which must be implemented if handler should expose and support the GET route. It will allow clients to download the content of an upload regardless whether it's finished or not. Please, be aware that this feature is not part of the official tus specification. Instead it's a custom mechanism by tusd.

type HTTPError 

type HTTPError interface {
	error
	StatusCode() int
	Body() []byte
}

HTTPError represents an error with an additional status code attached which may be used when this error is sent in a HTTP response. See the net/http package for standardized status codes.

func NewHTTPError 

func NewHTTPError(err error, statusCode int) HTTPError

NewHTTPError adds the given status code to the provided error and returns the new error instance. The status code may be used in corresponding HTTP responses. See the net/http package for standardized status codes.

type Handler 

type Handler struct {
	*UnroutedHandler
	http.Handler
}

Handler is a ready to use handler with routing (using pat)

func NewHandler 

func NewHandler(config Config) (*Handler, error)

NewHandler creates a routed tus protocol handler. This is the simplest way to use tusd but may not be as configurable as you require. If you are integrating this into an existing app you may like to use tusd.NewUnroutedHandler instead. Using tusd.NewUnroutedHandler allows the tus handlers to be combined into your existing router (aka mux) directly. It also allows the GET and DELETE endpoints to be customized. These are not part of the protocol so can be changed depending on your needs.

type LengthDeferrerDataStore 

type LengthDeferrerDataStore interface {
	DeclareLength(id string, length int64) error
}

LengthDeferrerDataStore is the interface that must be implemented if the creation-defer-length extension should be enabled. The extension enables a client to upload files when their total size is not yet known. Instead, the client must send the total size as soon as it becomes known.

type LockerDataStore 

type LockerDataStore interface {
	// LockUpload attempts to obtain an exclusive lock for the upload specified
	// by its id.
	// If this operation fails because the resource is already locked, the
	// tusd.ErrFileLocked must be returned. If no error is returned, the attempt
	// is consider to be successful and the upload to be locked until UnlockUpload
	// is invoked for the same upload.
	LockUpload(id string) error
	// UnlockUpload releases an existing lock for the given upload.
	UnlockUpload(id string) error
}

LockerDataStore is the interface required for custom lock persisting mechanisms. Common ways to store this information is in memory, on disk or using an external service, such as ZooKeeper. When multiple processes are attempting to access an upload, whether it be by reading or writing, a synchronization mechanism is required to prevent data corruption, especially to ensure correct offset values and the proper order of chunks inside a single upload.

type MetaData 

type MetaData map[string]string

type Metrics 

type Metrics struct {
	// RequestTotal counts the number of incoming requests per method
	RequestsTotal map[string]*uint64
	// ErrorsTotal counts the number of returned errors by their message
	ErrorsTotal       *ErrorsTotalMap
	BytesReceived     *uint64
	UploadsFinished   *uint64
	UploadsCreated    *uint64
	UploadsTerminated *uint64
}

Metrics provides numbers about the usage of the tusd handler. Since these may be accessed from multiple goroutines, it is necessary to read and modify them atomically using the functions exposed in the sync/atomic package, such as atomic.LoadUint64. In addition the maps must not be modified to prevent data races.

type StoreComposer 

type StoreComposer struct {
	Core DataStore

	UsesTerminater     bool
	Terminater         TerminaterDataStore
	UsesFinisher       bool
	Finisher           FinisherDataStore
	UsesLocker         bool
	Locker             LockerDataStore
	UsesGetReader      bool
	GetReader          GetReaderDataStore
	UsesConcater       bool
	Concater           ConcaterDataStore
	UsesLengthDeferrer bool
	LengthDeferrer     LengthDeferrerDataStore
}

StoreComposer represents a composable data store. It consists of the core data store and optional extensions. Please consult the package's overview for a more detailed introduction in how to use this structure.

func NewStoreComposer 

func NewStoreComposer() *StoreComposer

NewStoreComposer creates a new and empty store composer.

Example 
package main

import (
	"github.com/kyl2016/tusd"
	"github.com/kyl2016/tusd/filestore"
	"github.com/kyl2016/tusd/limitedstore"
	"github.com/kyl2016/tusd/memorylocker"
)

func main() {
	composer := tusd.NewStoreComposer()

	fs := filestore.New("./data")
	fs.UseIn(composer)

	ml := memorylocker.New()
	ml.UseIn(composer)

	ls := limitedstore.New(1024*1024*1024, composer.Core, composer.Terminater)
	ls.UseIn(composer)

	config := tusd.Config{
		StoreComposer: composer,
	}

	_, _ = tusd.NewHandler(config)
}

Output:

func (*StoreComposer) Capabilities 

func (store *StoreComposer) Capabilities() string

Capabilities returns a string representing the provided extensions in a human-readable format meant for debugging.

func (*StoreComposer) UseConcater 

func (store *StoreComposer) UseConcater(ext ConcaterDataStore)

func (*StoreComposer) UseCore 

func (store *StoreComposer) UseCore(core DataStore)

UseCore will set the used core data store. If the argument is nil, the property will be unset.

func (*StoreComposer) UseFinisher 

func (store *StoreComposer) UseFinisher(ext FinisherDataStore)

func (*StoreComposer) UseGetReader 

func (store *StoreComposer) UseGetReader(ext GetReaderDataStore)

func (*StoreComposer) UseLengthDeferrer 

func (store *StoreComposer) UseLengthDeferrer(ext LengthDeferrerDataStore)

func (*StoreComposer) UseLocker 

func (store *StoreComposer) UseLocker(ext LockerDataStore)

func (*StoreComposer) UseTerminater 

func (store *StoreComposer) UseTerminater(ext TerminaterDataStore)

type TerminaterDataStore 

type TerminaterDataStore interface {
	// Terminate an upload so any further requests to the resource, both reading
	// and writing, must return os.ErrNotExist or similar.
	Terminate(id string) error
}

TerminaterDataStore is the interface which must be implemented by DataStores if they want to receive DELETE requests using the Handler. If this interface is not implemented, no request handler for this method is attached.

type UnroutedHandler 

type UnroutedHandler struct {

	// CompleteUploads is used to send notifications whenever an upload is
	// completed by a user. The FileInfo will contain information about this
	// upload after it is completed. Sending to this channel will only
	// happen if the NotifyCompleteUploads field is set to true in the Config
	// structure. Notifications will also be sent for completions using the
	// Concatenation extension.
	CompleteUploads chan FileInfo
	// TerminatedUploads is used to send notifications whenever an upload is
	// terminated by a user. The FileInfo will contain information about this
	// upload gathered before the termination. Sending to this channel will only
	// happen if the NotifyTerminatedUploads field is set to true in the Config
	// structure.
	TerminatedUploads chan FileInfo
	// UploadProgress is used to send notifications about the progress of the
	// currently running uploads. For each open PATCH request, every second
	// a FileInfo instance will be send over this channel with the Offset field
	// being set to the number of bytes which have been transfered to the server.
	// Please be aware that this number may be higher than the number of bytes
	// which have been stored by the data store! Sending to this channel will only
	// happen if the NotifyUploadProgress field is set to true in the Config
	// structure.
	UploadProgress chan FileInfo
	// CreatedUploads is used to send notifications about the uploads having been
	// created. It triggers post creation and therefore has all the FileInfo incl.
	// the ID available already. It facilitates the post-create hook. Sending to
	// this channel will only happen if the NotifyCreatedUploads field is set to
	// true in the Config structure.
	CreatedUploads chan FileInfo
	// Metrics provides numbers of the usage for this handler.
	Metrics Metrics
	// contains filtered or unexported fields
}

UnroutedHandler exposes methods to handle requests as part of the tus protocol, such as PostFile, HeadFile, PatchFile and DelFile. In addition the GetFile method is provided which is, however, not part of the specification.

func NewUnroutedHandler 

func NewUnroutedHandler(config Config) (*UnroutedHandler, error)

NewUnroutedHandler creates a new handler without routing using the given configuration. It exposes the http handlers which need to be combined with a router (aka mux) of your choice. If you are looking for preconfigured handler see NewHandler.

func (*UnroutedHandler) DelFile 

func (handler *UnroutedHandler) DelFile(w http.ResponseWriter, r *http.Request)

DelFile terminates an upload permanently.

func (*UnroutedHandler) GetFile 

func (handler *UnroutedHandler) GetFile(w http.ResponseWriter, r *http.Request)

GetFile handles requests to download a file using a GET request. This is not part of the specification.

func (*UnroutedHandler) HeadFile 

func (handler *UnroutedHandler) HeadFile(w http.ResponseWriter, r *http.Request)

HeadFile returns the length and offset for the HEAD request

func (*UnroutedHandler) Middleware 

func (handler *UnroutedHandler) Middleware(h http.Handler) http.Handler

Middleware checks various aspects of the request and ensures that it conforms with the spec. Also handles method overriding for clients which cannot make PATCH AND DELETE requests. If you are using the tusd handlers directly you will need to wrap at least the POST and PATCH endpoints in this middleware.

func (*UnroutedHandler) PatchFile 

func (handler *UnroutedHandler) PatchFile(w http.ResponseWriter, r *http.Request)

PatchFile adds a chunk to an upload. This operation is only allowed if enough space in the upload is left.

func (*UnroutedHandler) PostFile 

func (handler *UnroutedHandler) PostFile(w http.ResponseWriter, r *http.Request)

PostFile creates a new file upload using the datastore after validating the length and parsing the metadata.

Source Files

View all Source files

Directories

Path Synopsis
Package cachestore provide a storage backend based on the memory, only cache the bytes.
 Package cachestore provide a storage backend based on the memory, only cache the bytes.
cmd
tusd
tusd/cli
tusd/cli/hooks
Package consullocker provides a locking mechanism using a Consul server.
 Package consullocker provides a locking mechanism using a Consul server.
Package etcd3locker provides a locking mechanism using an etcd3 cluster.
 Package etcd3locker provides a locking mechanism using an etcd3 cluster.
Package filestore provide a storage backend based on the local file system.
 Package filestore provide a storage backend based on the local file system.
Package gcsstore provides a Google cloud storage based backend.
 Package gcsstore provides a Google cloud storage based backend.
Package limitedstore provides a storage with a limited space.
 Package limitedstore provides a storage with a limited space.
Package memorylocker provides an in-memory locking mechanism.
 Package memorylocker provides an in-memory locking mechanism.
package prometheuscollector allows to expose metrics for Prometheus.
 package prometheuscollector allows to expose metrics for Prometheus.
Package s3store provides a storage backend using AWS S3 or compatible servers.
 Package s3store provides a storage backend using AWS S3 or compatible servers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.