Version: v2.0.0-...-eedbe7b Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Mar 5, 2021 License: LGPL-3.0 Imports: 3 Imported by: 2



utils/filestorage provides types for abstracting and implementing a system that stores files, including their metadata.

Each file in the system is identified by a unique ID, determined by the system at the time the file is stored.

File metadata includes such information as the size of the file, its checksum, and when it was created. Regardless of how it is stored in the system, at the abstraction level it is represented as a document.

Metadata can exist in the system without an associated file. However, every file must have a corresponding metadata doc stored in the system. A file can be added for a metadata doc that does not have one already.

The main type is the FileStorage interface. It exposes the core functionality of such a system. This includes adding/removing files, retrieving them or their metadata, and listing all files in the system.

The package also provides a basic implementation of FileStorage, available through NewFileStorage(). This implementation simply wraps two more focused systems: doc storage and raw file storage. The wrapper uses the doc storage to store the metadata and raw file storage to store the files.

The two subsystems are exposed via corresponding interfaces: DocStorage (and its specialization MetadataStorage) and RawFileStorage. While a single type could implement both, in practice they will be separate. The doc storage is responsible to generating the unique IDs. The raw file storage defers to the doc storage for any information about the file, including the ID.



This section is empty.


This section is empty.


This section is empty.


type Doc

type Doc struct {
	Raw RawDoc

Doc wraps a document in the Document interface.

func (*Doc) ID

func (d *Doc) ID() string

ID returns the document's unique identifier.

func (*Doc) SetID

func (d *Doc) SetID(id string) bool

SetID sets the document's unique identifier. If the ID is already set, SetID() returns true (false otherwise).

type DocStorage

type DocStorage interface {

	// Doc returns the doc that matches the ID.  If there is no match,
	// an error is returned (see errors.IsNotFound).  Any other problem
	// also results in an error.
	Doc(id string) (Document, error)

	// ListDocs returns a list of all the docs in the storage.
	ListDocs() ([]Document, error)

	// AddDoc adds the doc to the storage.  If successful, the storage-
	// generated ID for the doc is returned.  Otherwise an error is
	// returned.
	AddDoc(doc Document) (string, error)

	// RemoveDoc removes the matching doc from the storage.  If there
	// is no match an error is returned (see errors.IsNotFound).  Any
	// other problem also results in an error.
	RemoveDoc(id string) error

DocStorage is an abstraction for a system that can store docs (structs). The system is expected to generate its own unique ID for each doc.

type Document

type Document interface {
	// ID returns the unique ID of the document.
	ID() string

	// SetID sets the ID of the document.  If the ID is already set,
	// SetID() should return true (false otherwise).
	SetID(id string) (alreadySet bool)

Document represents a document that can be identified uniquely by a string.

type FileMetadata

type FileMetadata struct {
	Raw RawFileMetadata

FileMetadata contains the metadata for a single stored file.

func NewMetadata

func NewMetadata() *FileMetadata

NewMetadata returns a new Metadata for a stored file.

func (*FileMetadata) Checksum

func (m *FileMetadata) Checksum() string

func (*FileMetadata) ChecksumFormat

func (m *FileMetadata) ChecksumFormat() string

func (*FileMetadata) SetFileInfo

func (m *FileMetadata) SetFileInfo(size int64, checksum, format string) error

func (*FileMetadata) SetStored

func (m *FileMetadata) SetStored(timestamp *time.Time)

func (*FileMetadata) Size

func (m *FileMetadata) Size() int64

func (*FileMetadata) Stored

func (m *FileMetadata) Stored() *time.Time

type FileStorage

type FileStorage interface {

	// Metadata returns a file's metadata.
	Metadata(id string) (Metadata, error)

	// Get returns a file and its metadata.
	Get(id string) (Metadata, io.ReadCloser, error)

	// List returns the metadata for each stored file.
	List() ([]Metadata, error)

	// Add stores a file and its metadata.
	Add(meta Metadata, archive io.Reader) (string, error)

	// SetFile stores a file for an existing metadata entry.
	SetFile(id string, file io.Reader) error

	// Remove removes a file from storage.
	Remove(id string) error

FileStorage is an abstraction that can be used for the storage of files.

func NewFileStorage

func NewFileStorage(meta MetadataStorage, files RawFileStorage) FileStorage

NewFileStorage returns a new FileStorage value that wraps a MetadataStorage and a RawFileStorage. It coordinates the two even though they may not be designed to be compatible (or the two may be the same value).

A stored file will always have a metadata value stored. However, it is not required to have a raw file stored.

type Metadata

type Metadata interface {

	// Size is the size of the file (in bytes).
	Size() int64

	// Checksum is the checksum for the file.
	Checksum() string

	// ChecksumFormat is the kind (and encoding) of checksum.
	ChecksumFormat() string

	// Stored returns when the file was last stored.  If it has not been
	// stored yet, nil is returned.  If it has been stored but the
	// timestamp is not available, a zero value is returned
	// (see Time.IsZero).
	Stored() *time.Time

	// SetFileInfo sets the file info on the metadata.
	SetFileInfo(size int64, checksum, checksumFormat string) error

	// SetStored records when the file was last stored.  If the previous
	// value matters, be sure to call Stored() first.
	SetStored(timestamp *time.Time)

Metadata is the meta information for a stored file.

func Convert

func Convert(doc Document) (Metadata, error)

Convert turns a Document into a Metadata if possible.

type MetadataDocStorage

type MetadataDocStorage struct {

MetadataDocStorage provides the MetadataStorage methods than can be derived from DocStorage methods. To fully implement MetadataStorage, this type must be embedded in a type that implements the remaining methods.

func (*MetadataDocStorage) AddMetadata

func (s *MetadataDocStorage) AddMetadata(meta Metadata) (string, error)

ListMetadata implements MetadataStorage.ListMetadata.

func (*MetadataDocStorage) ListMetadata

func (s *MetadataDocStorage) ListMetadata() ([]Metadata, error)

ListMetadata implements MetadataStorage.ListMetadata.

func (*MetadataDocStorage) Metadata

func (s *MetadataDocStorage) Metadata(id string) (Metadata, error)

Metadata implements MetadataStorage.Metadata.

func (*MetadataDocStorage) RemoveMetadata

func (s *MetadataDocStorage) RemoveMetadata(id string) error

ListMetadata implements MetadataStorage.ListMetadata.

type MetadataStorage

type MetadataStorage interface {

	// Metadata returns the matching Metadata.  It fails if there is no
	// match (see errors.IsNotFound).  Any other problems likewise
	// results in an error.
	Metadata(id string) (Metadata, error)

	// ListMetadata returns a list of all metadata in the storage.
	ListMetadata() ([]Metadata, error)

	// AddMetadata adds the metadata to the storage.  If successful, the
	// storage-generated ID for the metadata is returned.  Otherwise an
	// error is returned.
	AddMetadata(meta Metadata) (string, error)

	// RemoveMetadata removes the matching metadata from the storage.
	// If there is no match an error is returned (see errors.IsNotFound).
	// Any other problem also results in an error.
	RemoveMetadata(id string) error

	// SetStored updates the stored metadata to indicate that the
	// associated file has been successfully stored in a RawFileStorage
	// system.  If it does not find a stored metadata with the matching
	// ID, it will return an error (see errors.IsNotFound).  It also
	// returns an error if it fails to update the stored metadata.
	SetStored(id string) error

MetadataStorage is an extension of DocStorage adapted to file metadata.

type RawDoc

type RawDoc struct {
	// ID is the unique identifier for the document.
	ID string

RawDoc is a basic, uniquely identifiable document.

type RawFileMetadata

type RawFileMetadata struct {
	// Size is the size (in bytes) of the stored file.
	Size int64
	// Checksum is the checksum of the stored file.
	Checksum string
	// ChecksumFormat describes the kind of the checksum.
	ChecksumFormat string
	// Stored records the timestamp of when the file was last stored.
	Stored *time.Time

RawFileMetadata holds info specific to stored files.

type RawFileStorage

type RawFileStorage interface {

	// File returns the matching file.  If there is no match an error is
	// returned (see errors.IsNotFound).  Any other problem also results
	// in an error.
	File(id string) (io.ReadCloser, error)

	// AddFile adds the file to the storage.  If it fails to do so,
	// it returns an error.  If a file is already stored for the ID,
	// AddFile() fails (see errors.IsAlreadyExists).
	AddFile(id string, file io.Reader, size int64) error

	// RemoveFile removes the matching file from the storage.  It fails
	// if there is no such file (see errors.IsNotFound).  Any other problem
	// also results in an error.
	RemoveFile(id string) error

RawFileStorage is an abstraction around a system that can store files. The system is expected to rely on the user for unique IDs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL