backup

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 7, 2023 License: AGPL-3.0 Imports: 16 Imported by: 0

Documentation

Overview

Package backup is providing commands to inspect a file system (hard-drive, USB, Android, S3) and backup medias to a remote DPhoto storage.

Index

Constants

View Source 
const (
	MediaTypeImage MediaType = "IMAGE"
	MediaTypeVideo MediaType = "VIDEO"
	MediaTypeOther MediaType = "OTHER"

	OrientationUpperLeft  ImageOrientation = "UPPER_LEFT"
	OrientationLowerRight ImageOrientation = "LOWER_RIGHT"
	OrientationUpperRight ImageOrientation = "UPPER_RIGHT"
	OrientationLowerLeft  ImageOrientation = "LOWER_LEFT"

	ProgressEventScanComplete   ProgressEventType = "scan-complete"     // ProgressEventScanComplete set the total of files
	ProgressEventAnalysed       ProgressEventType = "analysed"          // ProgressEventAnalysed is not useful for progress, it will be fined grained before upload
	ProgressEventCatalogued     ProgressEventType = "catalogued"        // ProgressEventCatalogued is not useful for progress, it will be fined grained before upload
	ProgressEventWrongAlbum     ProgressEventType = "wrong-album"       // ProgressEventWrongAlbum count files in filtered out albums (if filter used), subtracted from ProgressEventScanComplete
	ProgressEventAlreadyExists  ProgressEventType = "duplicate-catalog" // ProgressEventAlreadyExists count files already known in catalog, subtracted from ProgressEventScanComplete
	ProgressEventDuplicate      ProgressEventType = "duplicate-unique"  // ProgressEventDuplicate count files present twice in this backup/scan process, subtracted from ProgressEventScanComplete
	ProgressEventReadyForUpload ProgressEventType = "upload-ready"      // ProgressEventReadyForUpload files remaining after analysis and filters: ProgressEventReadyForUpload = ProgressEventScanComplete - ProgressEventDuplicate - ProgressEventAlreadyExists - ProgressEventWrongAlbum
	ProgressEventUploaded       ProgressEventType = "uploaded"          // ProgressEventUploaded files uploaded, is equals to ProgressEventReadyForUpload when complete
	ProgressEventAlbumCreated   ProgressEventType = "album-created"     // ProgressEventAlbumCreated notify when a new album is created
)

Variables

View Source 
var (
	ConcurrentAnalyser   = 1
	ConcurrentCataloguer = 1
	ConcurrentUploader   = 1
	BatchSize            = 1
)
View Source 
var (
	MediaCounterZero = MediaCounter{}
)
View Source 
var SupportedExtensions = map[string]MediaType{
	"jpg":  MediaTypeImage,
	"jpeg": MediaTypeImage,
	"png":  MediaTypeImage,
	"gif":  MediaTypeImage,
	"webp": MediaTypeImage,
	"raw":  MediaTypeImage,

	"svg": MediaTypeImage,
	"eps": MediaTypeImage,

	"mkv":  MediaTypeVideo,
	"mts":  MediaTypeVideo,
	"avi":  MediaTypeVideo,
	"mp4":  MediaTypeVideo,
	"mpeg": MediaTypeVideo,
	"mov":  MediaTypeVideo,
	"wmv":  MediaTypeVideo,
	"webm": MediaTypeVideo,
}

SupportedExtensions is used by SourceVolume adapters to filter files they find

Functions

func Init 

func Init(catalog CatalogAdapter, archive BArchiveAdapter)

func RegisterDetailsReader 

func RegisterDetailsReader(reader DetailsReaderAdapter)

func Scan 

func Scan(owner string, volume SourceVolume, optionSlice ...Options) ([]*ScannedFolder, []FoundMedia, error)

Scan a source to discover albums based on original folder structure. Use listeners will be notified on the progress of the scan.

Types

type AnalysedMedia 

type AnalysedMedia struct {
	FoundMedia FoundMedia    // FoundMedia is the reference of the file, implementation depends on the VolumeType
	Type       MediaType     // Type is 'photo' or 'video'
	Sha256Hash string        // Sha256Hash sha256 of the file
	Details    *MediaDetails // Details are data found within the file (location, date, ...)
}

AnalysedMedia is a FoundMedia to which has been attached its type (photo / video) and other details usually found within the file.

type BArchiveAdapter 

type BArchiveAdapter interface {
	// ArchiveMedia uploads the file in the right folder but might change the name to avoid clash with other existing files. Use files name is always returned.
	ArchiveMedia(owner string, media *BackingUpMediaRequest) (string, error)
}

type BackingUpMediaRequest 

type BackingUpMediaRequest struct {
	AnalysedMedia *AnalysedMedia
	Id            string
	FolderName    string
}

BackingUpMediaRequest is the requests that must be executed to back up the media

type CatalogAdapter 

type CatalogAdapter interface {
	// GetAlbumsTimeline create and initialise a timeline optimised to find album from a date, and create missing albums.
	GetAlbumsTimeline(owner string) (TimelineAdapter, error)

	// AssignIdsToNewMedias filter out existing medias and generate an ID for new ones.
	AssignIdsToNewMedias(owner string, medias []*AnalysedMedia) (map[*AnalysedMedia]string, error)

	// IndexMedias add to the catalog following medias
	IndexMedias(owner string, requests []*CatalogMediaRequest) error
}

type CatalogMediaRequest 

type CatalogMediaRequest struct {
	BackingUpMediaRequest *BackingUpMediaRequest
	ArchiveFilename       string // ArchiveFilename is a normalised named generated and used in archive.
}

CatalogMediaRequest is the request passed to Archive domain

type ClosableFoundMedia 

type ClosableFoundMedia interface {
	Close() error
}

ClosableFoundMedia can be implemented alongside FoundMedia if the implementation requires to release resources once the media has been handled.

type CompletionReport 

type CompletionReport interface {
	NewAlbums() []string
	Skipped() MediaCounter
	CountPerAlbum() map[string]*TypeCounter
}

func Backup 

func Backup(owner string, volume SourceVolume, optionsSlice ...Options) (CompletionReport, error)

Backup is analysing each media and is backing it up if not already in the catalog.

type DetailsReaderAdapter 

type DetailsReaderAdapter interface {
	// Supports returns true if the file can be parsed with this reader. False otherwise.
	Supports(media FoundMedia, mediaType MediaType) bool
	// ReadDetails extracts metadata from the content of the file.
	ReadDetails(reader io.Reader, options DetailsReaderOptions) (*MediaDetails, error)
}

type DetailsReaderOptions 

type DetailsReaderOptions struct {
	Fast bool // Fast true indicate the parser should focus at extracting the date, nothing else
}

type FoundMedia 

type FoundMedia interface {
	// MediaPath return breakdown of the absolute path of the media.
	MediaPath() MediaPath
	// ReadMedia reads content of the file ; it might not be optimised to call it several times (see VolumeToBackup)
	ReadMedia() (io.ReadCloser, error)
	// Size returns the size of the file
	Size() int

	String() string
}

FoundMedia represents files found on the scanned volume

func NewInmemoryMedia 

func NewInmemoryMedia(name string, date time.Time, content []byte) FoundMedia

NewInmemoryMedia creates a new FoundMedia for TESTING PURPOSE ONLY

type FullMediaSignature 

type FullMediaSignature struct {
	Sha256 string
	Size   uint
}

FullMediaSignature is the business key of the media, unique per user

func (*FullMediaSignature) String 

func (s *FullMediaSignature) String() string

type ImageOrientation 

type ImageOrientation string

ImageOrientation is teh start point of stored data

type InmemoryMedia 

type InmemoryMedia struct {
	// contains filtered or unexported fields
}

func (*InmemoryMedia) MediaPath 

func (i *InmemoryMedia) MediaPath() MediaPath

func (*InmemoryMedia) ReadMedia 

func (i *InmemoryMedia) ReadMedia() (io.ReadCloser, error)

func (*InmemoryMedia) Size 

func (i *InmemoryMedia) Size() int

func (*InmemoryMedia) String 

func (i *InmemoryMedia) String() string

type MediaCounter 

type MediaCounter struct {
	Count int // Count is the number of medias
	Size  int // Size is the sum of the size of the medias
}

func NewMediaCounter 

func NewMediaCounter(count int, size int) MediaCounter

func (MediaCounter) Add 

func (c MediaCounter) Add(count int, size int) MediaCounter

Add creates a new MediaCounter with the delta applied ; initial MediaCounter is not updated.

func (MediaCounter) AddCounter 

func (c MediaCounter) AddCounter(counter MediaCounter) MediaCounter

AddCounter creates a new MediaCounter which is the sum of the 2 counters provided.

func (MediaCounter) IsZero 

func (c MediaCounter) IsZero() bool

IsZero returns true if it's the default value

type MediaDetails 

type MediaDetails struct {
	Width, Height             int
	DateTime                  time.Time
	Orientation               ImageOrientation
	Make                      string
	Model                     string
	GPSLatitude, GPSLongitude float64
	Duration                  int64  // Duration is the length, in milliseconds, of a video
	VideoEncoding             string // VideoEncoding is the codec used to encode the video (ex: 'H264')
}

func (*MediaDetails) String 

func (s *MediaDetails) String() string

type MediaPath 

type MediaPath struct {
	ParentFullPath string // ParentFullPath is the absolute path of the media folder (URL = ParentFullPath + Filename)
	Root           string // Root is the path or URL representing the volume in which the media has been found. (URL = Root + Path + Filename)
	Path           string // Path is the directory path relative to Root: URL = Root + Path + Filename
	Filename       string // Filename does not contain any slash, and contains the extension
	ParentDir      string // ParentDir is the name of the directory: dirname(Root + Path)
}

MediaPath is a breakdown of an absolute path, or URL, agnostic of its origin.

type MediaType 

type MediaType string

MediaType is photo or video

type Options 

type Options struct {
	RestrictedAlbumFolderName map[string]interface{} // RestrictedAlbumFolderName will restrict the media to only back up medias that are in one of these albums
	Listener                  interface{}            // Listener will receive progress events.
	SkipRejects               bool                   // SkipRejects mode will report any analysis error, or missing timestamp, and continue.
}

func OptionOnlyAlbums 

func OptionOnlyAlbums(albums ...string) Options

OptionOnlyAlbums restricts backed up medias to those in these albums

func OptionSkipRejects 

func OptionSkipRejects(skip bool) Options

OptionSkipRejects disables the strict mode and ignores invalid files (wrong / no date, ...)

func OptionWithListener 

func OptionWithListener(listener interface{}) Options

OptionWithListener creates an option with a listener

type PostAnalyseFilter 

type PostAnalyseFilter interface {
	// AcceptAnalysedMedia returns TRUE if the media should be backed-up.
	AcceptAnalysedMedia(media *AnalysedMedia, folderName string) bool
}

type ProgressEvent 

type ProgressEvent struct {
	Type      ProgressEventType // Type defines what's count, and size are about ; some might not be used.
	Count     int               // Count is the number of media
	Size      int               // Size is the sum of the size of the media concerned by this event
	Album     string            // Album is the folder name of the medias concerned by this event
	MediaType MediaType         // MediaType is the type of media ; only mandatory with 'uploaded' event
}

type ProgressEventType 

type ProgressEventType string

type ScannedFolder 

type ScannedFolder struct {
	Name         string
	RelativePath string                  // RelativePath can be used for display purpose
	FolderName   string                  // FolderName is the original folder name (Name with date prefix that have been removed)
	AbsolutePath string                  // AbsolutePath is used to create a new SourceVolume
	Start, End   time.Time               // Start and End are the beginning of the day of the first media, and the beginning of the day following the last media.
	Distribution map[string]MediaCounter // Distribution is the number of media found for each day (format YYYY-MM-DD)
}

ScannedFolder represents a (sub)folder in the scanned target

func (*ScannedFolder) PushBoundaries 

func (f *ScannedFolder) PushBoundaries(date time.Time, size int)

PushBoundaries is updating the ScannedFolder dates, and update the counter.

type SourceVolume 

type SourceVolume interface {
	String() string
	FindMedias() ([]FoundMedia, error)
}

type TimelineAdapter 

type TimelineAdapter interface {
	FindOrCreateAlbum(mediaTime time.Time) (folderName string, created bool, err error)
	FindAlbum(dateTime time.Time) (folderName string, exists bool, err error)
}

type TrackAnalysed 

type TrackAnalysed interface {
	OnAnalysed(done, total MediaCounter)
}

type TrackEvents 

type TrackEvents interface {
	OnEvent(event ProgressEvent)
}

type TrackScanComplete 

type TrackScanComplete interface {
	OnScanComplete(total MediaCounter)
}

type TrackUploaded 

type TrackUploaded interface {
	OnUploaded(done, total MediaCounter)
}

TrackUploaded includes both uploaded and skipped

type Tracker 

type Tracker struct {
	Done chan struct{} // Done is closed when all events have been processed.
	// contains filtered or unexported fields
}

Tracker is consuming progress channel, keep a record of counts, and call listeners

func NewTracker 

func NewTracker(progressChannel chan *ProgressEvent, listeners ...interface{}) *Tracker

NewTracker creates the Tracker and start consuming (async)

func (*Tracker) CountPerAlbum 

func (t *Tracker) CountPerAlbum() map[string]*TypeCounter

func (*Tracker) NewAlbums 

func (t *Tracker) NewAlbums() []string

func (*Tracker) Skipped 

func (t *Tracker) Skipped() MediaCounter

func (*Tracker) WaitToComplete 

func (t *Tracker) WaitToComplete()

type TypeCounter 

type TypeCounter struct {
	// contains filtered or unexported fields
}

func NewTypeCounter 

func NewTypeCounter(mediaType MediaType, count int, size int) *TypeCounter

NewTypeCounter is a convenience method for testing or mocking 'backup' domain

func (*TypeCounter) GetMediaIndex 

func (c *TypeCounter) GetMediaIndex(mediaType MediaType) int

func (*TypeCounter) IncrementCounter 

func (c *TypeCounter) IncrementCounter(counter *[numberOfMediaType]int, mediaType MediaType, delta int)

func (*TypeCounter) IncrementFoundCounter 

func (c *TypeCounter) IncrementFoundCounter(mediaType MediaType, count int, size int) *TypeCounter

func (*TypeCounter) OfType 

func (c *TypeCounter) OfType(mediaType MediaType) MediaCounter

func (*TypeCounter) Total 

func (c *TypeCounter) Total() MediaCounter

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.