server

package
v0.0.0-...-bd9ca74 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2021 License: Apache-2.0 Imports: 33 Imported by: 0

Documentation

Index

Constants

View Source 
const MaxConcurrentCommits = 2

the number of transaction commits to tape we allow at a given time. If there are more they will wait in a queue.

Variables

View Source 
var (
	ErrInvalidPos = errors.New("Seek: cannot seek before read position")
	ErrWhence     = errors.New("Seek: invalid whence")
)
View Source 
var Version = "2019.2"

The version of this package. It is here so we can serve it on the welcome page. It is pulled from git during the make process.

Functions

func NewReadSeekCloser 

func NewReadSeekCloser(r store.ReadAtCloser, size int64) io.ReadSeekCloser

NewReadSeekCloser converts a ReadAtCloser into a ReadSeekCloser.

func NotImplementedHandler 

func NotImplementedHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

NotImplementedHandler will return a 501 not implemented error.

func VarHandler 

func VarHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

VarHandler adapts the expvar default handler to the httprouter three parameter handler.

func WelcomeHandler 

func WelcomeHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

WelcomeHandler handles requests from GET /

Types

type BlobDB 

type BlobDB interface {
	// Look up the metadata for the given item+blob id. Returns error if error encountered.
	// returns nil,nil if the blob was not found in the index.
	FindBlob(item string, blobid int) (*items.Blob, error)

	// Look up blob metadata using an item+version+slot name combo. Returns error if one
	// happened. Returns nil,nil if no such blob is in the index, so a missing item is not an error.
	// The slot name needs to be exact, no wildcard expansion is done.
	// Use version = 0 to refer to the most recent version of the item.
	FindBlobBySlot(item string, version int, slot string) (*items.Blob, error)

	// Index the given item using the given id.
	// (The item id should already be in the item structure. can that parameter be removed?)
	IndexItem(itemid string, item *items.Item) error

	// GetItemList returns a list of item information for a listing page.
	GetItemList(offset int, pagesize int, sortorder string) ([]SimpleItem, error)
}

BlobDB are the methods we need to interact with the new item metadata caching. This interface is expected to grow as more functionality is moved to the database.

The goal is to remove the original database Cache interface along with its hooks into the item package.

type ContentStatus 

type ContentStatus int
const (
	ContentUnknown ContentStatus = iota
	ContentCached                // the content is sourced from the cache
	ContentLarge                 // the content is very big and is not cached
	ContentWaiting               // the content is being copied into the cache
)

type Fixity 

type Fixity struct {
	ID            int64     // id of this record
	Item          string    // the item being verified
	ScheduledTime time.Time `json:"Scheduled_time"` // scheduled time
	Status        string    // "scheduled", "mismatch", "ok", "error"
	Notes         string    // mostly only used in case of an error
}

A Fixity represents a single past or future fixity check.

type FixityDB 

type FixityDB interface {
	// NextItem returns the fixity record id of the earliest pending record
	// that is before the cutoff time. If there are no such records 0 is returned.
	NextFixity(cutoff time.Time) int64

	// GetFixty retuens the fixity record with the given id.
	// Returns nil if no such record was found, or on error.
	GetFixity(id int64) *Fixity

	// SearchFixty returns the fixity records matching the provided arguments.
	// It returns an empty slice if there are no matching records or on error.
	// Status can be 'scheduled', 'error', 'ok', or 'mismatch'
	// Use the zero value for a parameter to represent a wildcard.
	SearchFixity(start time.Time, end time.Time, item string, status string) []*Fixity

	// UpdateItem will either update a fixity record or create a new one. If
	// the ID field is empty (i.e. = 0) then a new fixity record is created. If
	// an ID was passed, and the status of the record in the database is
	// "scheduled", then the database is updated to match the passed in record.
	// The id of the created/updated record is returned. If record has an empty
	// status, it will default to "scheduled".
	UpdateFixity(record Fixity) (int64, error)

	// Delete fixity record id, only if the status is "scheduled". Error otherwise.
	DeleteFixity(id int64) error

	// LookupCheck takes an item id and returns the time of earliest pending
	// fixity check for that item. If no fixity check is pending, returns
	// the zero time.
	LookupCheck(item string) (time.Time, error)
}

A FixityDB tracks the information the fixity service needs to know what items have been checked, what needs to be checked, and any fixity errors found. It is presumed to be backed by a database, but that is not assumed. Methods should be safe to be called by multiple goroutines.

The fixity records in the "scheduled" state are free to be modified. But records in other states should be considered immutable.

type InvalidValidator 

type InvalidValidator struct{}

An InvalidValidator is a TokenValidator for which every token is invalid. That is, it always returns the user "" with the Unknown role.

func (InvalidValidator) TokenValid 

func (InvalidValidator) TokenValid(token string) (string, Role, error)

TokenValid will always return an invalid user with no permissions.

type MsqlCache 

type MsqlCache struct {
	// contains filtered or unexported fields
}

MsqlCache implements the items.ItemCache interface and the FixityDB interface using MySQL as the backing store.

func NewMysqlCache 

func NewMysqlCache(dial string) (*MsqlCache, error)

NewMysqlCache connects to a MySQL database and returns an item satisifying both the ItemCache and FixityDB interfaces.

func (*MsqlCache) DeleteFixity 

func (mc *MsqlCache) DeleteFixity(id int64) error

func (*MsqlCache) FindBlob 

func (ms *MsqlCache) FindBlob(item string, blobid int) (*items.Blob, error)

func (*MsqlCache) FindBlobBySlot 

func (ms *MsqlCache) FindBlobBySlot(item string, version int, slot string) (*items.Blob, error)

func (*MsqlCache) GetFixity 

func (mc *MsqlCache) GetFixity(id int64) *Fixity

GetFixity

func (*MsqlCache) GetItemList 

func (ms *MsqlCache) GetItemList(offset int, pagesize int, sortorder string) ([]SimpleItem, error)

func (*MsqlCache) IndexItem 

func (ms *MsqlCache) IndexItem(item string, thisItem *items.Item) error

IndexItem adds row entries for every version, slot, and blob for the given item. It is ok if some pieces are already in the tables.

func (*MsqlCache) Lookup 

func (ms *MsqlCache) Lookup(id string) *items.Item

Lookup returns a cached Item, if one exists in the database. Otherwise it returns nil.

func (*MsqlCache) LookupCheck 

func (mc *MsqlCache) LookupCheck(item string) (time.Time, error)

LookupCheck will return the time of the earliest scheduled fixity check for the given item. If there is no pending fixity check for the item, it returns the zero time.

func (*MsqlCache) NextFixity 

func (mc *MsqlCache) NextFixity(cutoff time.Time) int64

NextFixity returns the earliest scheduled fixity record that is before the cutoff time. If there is no such record it returns 0

func (*MsqlCache) SearchFixity 

func (mc *MsqlCache) SearchFixity(start, end time.Time, item string, status string) []*Fixity

SearchFixity

func (*MsqlCache) Set 

func (ms *MsqlCache) Set(id string, thisItem *items.Item)

Set adds the given item to the cache under the key id.

func (*MsqlCache) UpdateFixity 

func (mc *MsqlCache) UpdateFixity(record Fixity) (int64, error)

UpdateFixity updates or creates the given fixity record. The record is created if ID is == 0. Otherwise the given record is updated so long as the record in the database has status "scheduled". The ID of the new or updated record is returned.

type NobodyValidator 

type NobodyValidator struct{}

A NobodyValidator is a TokenValidator that for every possible token returns a user named "nobody" with the Admin role.

func (NobodyValidator) TokenValid 

func (NobodyValidator) TokenValid(token string) (string, Role, error)

TokenValid will always return an Admin user named "nobody".

type QlCache 

type QlCache struct {
	// contains filtered or unexported fields
}

A QlCache implements an item.ItemCache and a FixityDB interface backed by a QL database.

func NewQlCache 

func NewQlCache(filename string) (*QlCache, error)

NewQlCache makes a QL database cache. filename is the name of the file to save the database to. The filname beginning with "mem--" means to keep everything in memory.

func (*QlCache) DeleteFixity 

func (qc *QlCache) DeleteFixity(id int64) error

func (*QlCache) FindBlob 

func (qc *QlCache) FindBlob(item string, blobid int) (*items.Blob, error)

func (*QlCache) FindBlobBySlot 

func (qc *QlCache) FindBlobBySlot(item string, version int, slot string) (*items.Blob, error)

func (*QlCache) GetFixity 

func (qc *QlCache) GetFixity(id int64) *Fixity

GetFixityById

func (*QlCache) GetItemList 

func (qc *QlCache) GetItemList(offset int, pagesize int, sortorder string) ([]SimpleItem, error)

func (*QlCache) IndexItem 

func (qc *QlCache) IndexItem(item string, thisItem *items.Item) error

IndexItem adds row entries for every version, slot, and blob for the given item. It is ok if some pieces are already in the tables.

func (*QlCache) Lookup 

func (qc *QlCache) Lookup(item string) *items.Item

Lookup returns an item from the cache if it exists. If there is nothing under that key, it will return nil.

func (*QlCache) LookupCheck 

func (qc *QlCache) LookupCheck(item string) (time.Time, error)

LookupCheck returns the earliest scheduled fixity check for the given item. If there is no scheduled fixity check, it returns the zero time.

func (*QlCache) NextFixity 

func (qc *QlCache) NextFixity(cutoff time.Time) int64

NextFixity will return the item id of the earliest scheduled fixity check that is before the cutoff time. If there is no such record 0 is returned.

func (*QlCache) SearchFixity 

func (qc *QlCache) SearchFixity(start, end time.Time, item string, status string) []*Fixity

SearchFixity

func (*QlCache) Set 

func (qc *QlCache) Set(item string, thisItem *items.Item)

Set adds the given item to the cache under the key item.

func (*QlCache) UpdateFixity 

func (qc *QlCache) UpdateFixity(record Fixity) (int64, error)

UpdateFixity updates or creates the given fixity record. The record is created if ID is == 0. Otherwise the given record is updated so long as the record in the database has status "scheduled".

type RESTServer 

type RESTServer struct {
	// Port number to run bendo on. defaults to 14000
	PortNumber string

	// Port to run the profiler/inspector on. If empty, the goroutine is not
	// started. It provides information on the "/debug/" route.
	PProfPort string

	// Items is the base item store (that is, tape system or file system where
	// preservation content is stored). Run() will panic if Items is nil.
	Items *items.Store

	// Validator does authentication by validating any user tokens
	// presented to the API. If this is nil then no authentication will be
	// done.
	Validator TokenValidator

	// TxStore keeps information on transactions in progress. If this is
	// nil, transactions will be kept inside the cache directory.
	TxStore *transaction.Store

	// FileStore keeps the uploaded file waiting to be saved into the Item
	// store. If nil, the files will be stored inside the cache directory.
	FileStore *fragment.Store

	// Cache keeps smallish blobs retreived from tape.
	Cache blobcache.T

	// BlobDB holds the item/version/slot/blob information in a structured way
	// so we can query it without needing to read and parse the JSON structures
	// describing items.
	BlobDB BlobDB

	// Fixity stores the records tracking past and future fixity checks.
	FixityDatabase FixityDB
	DisableFixity  bool
	// contains filtered or unexported fields
}

RESTServer holds the configuration for a Bendo REST API server.

Set all the public fields and then call Run. Run will listen on the given port and handle requests. At the moment there is no maximum simultaneous request limit. Do not change any fields after calling Run.

Run will also start a goroutine to handle serializing new file uploads into storage bags, and a goroutine to do fixity checking.

There are two levels of configuration. It should be enough to only set CacheDir and Items. The other fields are exposed to allow more customization.

func (*RESTServer) AppendFileHandler 

func (s *RESTServer) AppendFileHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

AppendFileHandler handles requests to both POST /upload and POST /upload/:fileid

func (*RESTServer) BundleListHandler 

func (s *RESTServer) BundleListHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

BundleListHandler handles GET requests to "/bundle/list".

func (*RESTServer) BundleListPrefixHandler 

func (s *RESTServer) BundleListPrefixHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

BundleListPrefixHandler handles GET requests to "/bundle/list/:prefix".

func (*RESTServer) BundleOpenHandler 

func (s *RESTServer) BundleOpenHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

BundleOpenHandler handles GET requests to "/bundle/open/:key"

func (*RESTServer) CancelTxHandler 

func (s *RESTServer) CancelTxHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

CancelTxHandler handles requests to POST /transaction/:tid/cancel

func (*RESTServer) DeleteFileHandler 

func (s *RESTServer) DeleteFileHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

DeleteFileHandler handles requests to DELETE /upload/:fileid This deletes a file which has been uploaded and is in the temporary holding area.

func (*RESTServer) DeleteFixityHandler 

func (s *RESTServer) DeleteFixityHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

DeleteFixity handles requests to DELETE /fixty/:id This deletes a scheduled fixity check for the given id . reurns 404 ff no request is found in the 'scheduled' state for the give id, 200 otherwise.

func (*RESTServer) DisableTapeUse 

func (s *RESTServer) DisableTapeUse()

DisableTapeUse disables the tape use flag. The server will not try to access the tape device while tape use is turned off.

func (*RESTServer) EnableTapeUse 

func (s *RESTServer) EnableTapeUse()

EnableTapeUse turns on the tape use flag. This allows the server to access the tape storage.

func (*RESTServer) GetFileHandler 

func (s *RESTServer) GetFileHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

GetFileHandler handles requests to GET /upload/:fileid

func (*RESTServer) GetFileInfoHandler 

func (s *RESTServer) GetFileInfoHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

GetFileInfoHandler handles requests to GET /upload/:fileid/metadata

func (*RESTServer) GetFixityHandler 

func (s *RESTServer) GetFixityHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

func (*RESTServer) GetFixityIdHandler 

func (s *RESTServer) GetFixityIdHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

func (*RESTServer) GetTapeUseHandler 

func (s *RESTServer) GetTapeUseHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

GetTapeUseHandler handles requests from GET /admin/use_tape

func (*RESTServer) Handler 

func (s *RESTServer) Handler() http.Handler

func (*RESTServer) IndexItem 

func (s *RESTServer) IndexItem(id string) error

IndexItem loads an item from the item store and indexes it into our blob database

func (*RESTServer) ItemHandler 

func (s *RESTServer) ItemHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

ItemHandler handles requests to GET /item/:id

func (*RESTServer) ListFileHandler 

func (s *RESTServer) ListFileHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

ListFileHandler handles requests to GET /upload

func (*RESTServer) ListTxHandler 

func (s *RESTServer) ListTxHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

ListTxHandler handles requests to GET /transaction

func (*RESTServer) NewTxHandler 

func (s *RESTServer) NewTxHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

NewTxHandler handles requests to POST /item/:id/transaction

func (*RESTServer) PostFixityHandler 

func (s *RESTServer) PostFixityHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

PostFixityHandler handles requests to POST /fixty/:item

func (*RESTServer) PutFixityHandler 

func (s *RESTServer) PutFixityHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

func (*RESTServer) Run 

func (s *RESTServer) Run() error

Run initializes and starts all the goroutines used by the server. It then blocks listening for and handling http requests.

func (*RESTServer) SetFileInfoHandler 

func (s *RESTServer) SetFileInfoHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

SetFileInfoHandler handles requests to PUT /upload/:fileid/metadata

func (*RESTServer) SetTapeUseHandler 

func (s *RESTServer) SetTapeUseHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

SetTapeUseHandler handles requests to PUT /admin/use_tape/:status

func (*RESTServer) SlotHandler 

func (s *RESTServer) SlotHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

SlotHandler handles requests to GET /item/:id/*slot 

and requests to HEAD /item/:id/*slot

func (*RESTServer) StartFixity 

func (s *RESTServer) StartFixity()

StartFixity starts the background goroutines to check item fixity. It returns immediately and does not block.

func (*RESTServer) Stop 

func (s *RESTServer) Stop() error

Stop will stop the server and return when all the server goroutines have exited and the socked closed.

func (*RESTServer) TxCleaner 

func (s *RESTServer) TxCleaner()

TxCleaner will loop forever removing old transactions and old orphened uploaded files. That means any finished transactions which are older than a few days. Both the transaction and any uploaded files referenced by the transaction are deleted. This function will never return.

func (*RESTServer) TxInfoHandler 

func (s *RESTServer) TxInfoHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

TxInfoHandler handles requests to GET /transaction/:tid

func (*RESTServer) UIItemsHandler 

func (s *RESTServer) UIItemsHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params)

UIItemsHandler handles requests from GET /ui/items

type Role 

type Role int

A Role is an enumeration describing the permission level a given user has. 

const (
	RoleUnknown Role = iota
	RoleMDOnly
	RoleRead
	RoleWrite
	RoleAdmin
)

The enumerations for a Role. They form a linear order with later entries having more permissions than earlier ones.

func AtoRole 

func AtoRole(s string) Role

AtoRole converts a string into a Role. The strings are case-insensitive, and are "mdonly", "read", "write", "admin". If the string cannot be decoded RoleUnknown is returned.

type SimpleItem 

type SimpleItem struct {
	ID        string
	MaxBundle int // largest bundle id used by this item
	Created   time.Time
	Modified  time.Time
	Size      int64
}

A SimpleItem is like an Item, but does not contain the blob and version information. It is used to simplify the item list displays.

type TokenValidator 

type TokenValidator interface {
	TokenValid(token string) (user string, role Role, err error)
}

A TokenValidator validates and decodes user tokens passed into the web API. If the given token is not valid, for whatever reason, the user "" with a role of RoleUnknown is returned. An error is returned only if there is some kind of error doing the lookup and the ultimate status of the token is unknown.

func NewListValidator 

func NewListValidator(r io.Reader) (TokenValidator, error)

NewListValidator returns a validator backed by a predefined list of users, which are read from r upon creation. The reader r should consist of a sequence of user entries, separated by newlines. Each entry has the form: 

<user name>  <role>  <token>

The fields are delineated by whitespace (spaces or tabs). This decoder does not permit spaces in either the user name or the token. The role is one of "MDOnly", "Read", "Write", "Admin" (case insensitive). Empty lines and lines beginning with a hash '#' are skipped.

func NewListValidatorFile 

func NewListValidatorFile(fname string) (TokenValidator, error)

NewListValidatorFile is a convenience function that reads the contents of the given file into a ListValidator. The file should have the same format that NewListValidator expects.

func NewListValidatorString 

func NewListValidatorString(data string) (TokenValidator, error)

NewListValidatorString is a convenience function that passes the given string into a ListValidator. The format of the string is the same as that expected by NewListValidator.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.