image

package
v1.12.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 21, 2023 License: MIT Imports: 31 Imported by: 0

Documentation

Overview

Package image handles storing, resizing and retrieval of images Provides Store with Save and Load implementations on top of local file system and bolt db. Service object encloses Store and add common methods, this is the one consumer should use.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CachedImgID 

func CachedImgID(imgURL string) (string, error)

CachedImgID generates ID for a cached image. ID would look like: "cached_images/<sha1-of-image-url-hostname>-<sha1-of-image-entire-url>" <sha1-of-image-url-hostname> - would allow us to identify all images from particular site if ever needed <sha1-of-image-entire-url> - would allow us to avoid storing duplicates of the same image (as accurate as deduplication based on potentially mutable url can be)

func Sha1Str 

func Sha1Str(s string) string

Sha1Str converts provided string to sha1

Types

type Bolt 

type Bolt struct {
	// contains filtered or unexported fields
}

Bolt provides image Store for images keeping data in bolt DB, restricts max size. It uses 3 buckets to manage images data. Two buckets contains image data (staged and committed images). Third bucket holds insertion timestamps.

func NewBoltStorage 

func NewBoltStorage(fileName string, options bolt.Options) (*Bolt, error)

NewBoltStorage create bolt image store

func (*Bolt) Cleanup 

func (b *Bolt) Cleanup(_ context.Context, ttl time.Duration) error

Cleanup runs scan of staging and removes old data based on ttl

func (*Bolt) Commit 

func (b *Bolt) Commit(id string) error

Commit file stored in staging bucket by copying it to permanent bucket Data from staging bucket not removed immediately, but would be removed on Cleanup

func (*Bolt) Info 

func (b *Bolt) Info() (StoreInfo, error)

Info returns meta information about storage

func (*Bolt) Load 

func (b *Bolt) Load(id string) ([]byte, error)

Load image from DB

func (*Bolt) ResetCleanupTimer added in v1.9.0 

func (b *Bolt) ResetCleanupTimer(id string) error

ResetCleanupTimer resets cleanup timer for the image

func (*Bolt) Save 

func (b *Bolt) Save(id string, img []byte) error

Save saves image for given id to staging bucket in DB

type FileSystem 

type FileSystem struct {
	Location   string
	Staging    string
	Partitions int
	// contains filtered or unexported fields
}

FileSystem provides image Store for local files. Saves and loads files from Location, restricts max size.

func (*FileSystem) Cleanup 

func (f *FileSystem) Cleanup(_ context.Context, ttl time.Duration) error

Cleanup runs scan of staging and removes old files based on ttl

func (*FileSystem) Commit 

func (f *FileSystem) Commit(id string) error

Commit file stored in staging location by moving it to permanent location

func (*FileSystem) Info 

func (f *FileSystem) Info() (StoreInfo, error)

Info returns meta information about storage

func (*FileSystem) Load 

func (f *FileSystem) Load(id string) ([]byte, error)

Load image from FS. Uses id to get partition subdirectory.

func (*FileSystem) ResetCleanupTimer added in v1.9.0 

func (f *FileSystem) ResetCleanupTimer(id string) error

ResetCleanupTimer resets cleanup timer for the image

func (*FileSystem) Save 

func (f *FileSystem) Save(id string, img []byte) error

Save saves image with given id to local FS, staging directory. Files partitioned across multiple subdirectories, and the final path includes part, i.e. /location/user1/03/123-4567

type RPC 

type RPC struct {
	jrpc.Client
}

RPC implements remote engine and delegates all Calls to remote http server

func (*RPC) Cleanup 

func (r *RPC) Cleanup(_ context.Context, ttl time.Duration) error

Cleanup runs scan of staging and removes old files based on ttl

func (*RPC) Commit 

func (r *RPC) Commit(id string) error

Commit file stored in staging location by moving it to permanent location

func (*RPC) Info 

func (r *RPC) Info() (StoreInfo, error)

Info returns meta information about storage

func (*RPC) Load 

func (r *RPC) Load(id string) ([]byte, error)

Load image with given id

func (*RPC) ResetCleanupTimer added in v1.9.0 

func (r *RPC) ResetCleanupTimer(id string) error

ResetCleanupTimer resets cleanup timer for the image

func (*RPC) Save 

func (r *RPC) Save(id string, img []byte) error

Save saves image with given id to staging.

type Service 

type Service struct {
	ServiceParams
	// contains filtered or unexported fields
}

Service wraps Store with common functions needed for any store implementation It also provides async Submit with func param retrieving all submitting IDs. Submitted IDs committed (i.e. moved from staging to final) on ServiceParams.EditDuration expiration.

func NewService 

func NewService(s Store, p ServiceParams) *Service

NewService returns new Service instance

func (*Service) Cleanup 

func (s *Service) Cleanup(ctx context.Context)

Cleanup runs periodic cleanup with 1.5*ServiceParams.EditDuration. Blocking loop, should be called inside of goroutine by consumer

func (*Service) Close 

func (s *Service) Close(ctx context.Context)

Close flushes all in-progress submits and enforces waiting commits

func (*Service) Commit added in v1.12.0 

func (s *Service) Commit(idsFn func() []string) error

Commit multiple ids immediately

func (*Service) ExtractNonProxiedPictures added in v1.12.0 

func (s *Service) ExtractNonProxiedPictures(commentHTML string) (ids []string)

ExtractNonProxiedPictures gets list of non-proxied images from the doc html and convert from urls to ids, i.e. user/pic.png This method is used in image check on post preview and load, as proxied images have lazy loading and wouldn't be present on disk but still valid as they will be loaded the first time someone requests them.

func (*Service) ExtractPictures 

func (s *Service) ExtractPictures(commentHTML string) (ids []string)

ExtractPictures gets list of images from the doc html and convert from urls to ids, i.e. user/pic.png

func (*Service) ImgContentType 

func (s *Service) ImgContentType(img []byte) string

ImgContentType returns content type for provided image

func (*Service) Info 

func (s *Service) Info() (StoreInfo, error)

Info returns meta information about storage

func (*Service) Load 

func (s *Service) Load(id string) ([]byte, error)

Load wraps storage Load function.

func (*Service) ResetCleanupTimer added in v1.9.0 

func (s *Service) ResetCleanupTimer(id string) error

ResetCleanupTimer resets cleanup timer for the image

func (*Service) Save 

func (s *Service) Save(userID string, r io.Reader) (id string, err error)

Save wraps storage Save function, validating and resizing the image before calling it.

func (*Service) SaveWithID 

func (s *Service) SaveWithID(id string, r io.Reader) error

SaveWithID wraps storage Save function, validating and resizing the image before calling it.

func (*Service) Submit 

func (s *Service) Submit(idsFn func() []string)

Submit multiple ids via function for delayed commit

type ServiceParams 

type ServiceParams struct {
	EditDuration time.Duration // edit period for comments
	ImageAPI     string        // image api matching path
	ProxyAPI     string        // proxy api matching path
	MaxSize      int
	MaxHeight    int
	MaxWidth     int
}

ServiceParams contains externally adjustable parameters of Service

type Store 

type Store interface {
	Info() (StoreInfo, error)         // get meta information about storage
	Save(id string, img []byte) error // store image with passed id to staging
	Load(id string) ([]byte, error)   // load image by ID

	ResetCleanupTimer(id string) error                    // resets cleanup timer for the image, called on comment preview
	Commit(id string) error                               // move image from staging to permanent
	Cleanup(ctx context.Context, ttl time.Duration) error // run removal loop for old images on staging
}

Store defines interface for saving and loading pictures. Declares two-stage save with Commit. Save stores to staging area and Commit moves to the final location. Two-stage commit scheme is used for not storing images which are uploaded but later never used in the comments, e.g. when somebody uploaded a picture but did not sent the comment.

type StoreInfo 

type StoreInfo struct {
	FirstStagingImageTS time.Time
}

StoreInfo contains image store meta information

type StoreMock added in v1.11.0 

type StoreMock struct {
	// CleanupFunc mocks the Cleanup method.
	CleanupFunc func(ctx context.Context, ttl time.Duration) error

	// CommitFunc mocks the Commit method.
	CommitFunc func(id string) error

	// InfoFunc mocks the Info method.
	InfoFunc func() (StoreInfo, error)

	// LoadFunc mocks the Load method.
	LoadFunc func(id string) ([]byte, error)

	// ResetCleanupTimerFunc mocks the ResetCleanupTimer method.
	ResetCleanupTimerFunc func(id string) error

	// SaveFunc mocks the Save method.
	SaveFunc func(id string, img []byte) error
	// contains filtered or unexported fields
}

StoreMock is a mock implementation of Store. 

func TestSomethingThatUsesStore(t *testing.T) {

	// make and configure a mocked Store
	mockedStore := &StoreMock{
		CleanupFunc: func(ctx context.Context, ttl time.Duration) error {
			panic("mock out the Cleanup method")
		},
		CommitFunc: func(id string) error {
			panic("mock out the Commit method")
		},
		InfoFunc: func() (StoreInfo, error) {
			panic("mock out the Info method")
		},
		LoadFunc: func(id string) ([]byte, error) {
			panic("mock out the Load method")
		},
		ResetCleanupTimerFunc: func(id string) error {
			panic("mock out the ResetCleanupTimer method")
		},
		SaveFunc: func(id string, img []byte) error {
			panic("mock out the Save method")
		},
	}

	// use mockedStore in code that requires Store
	// and then make assertions.

}

func (*StoreMock) Cleanup added in v1.11.0 

func (mock *StoreMock) Cleanup(ctx context.Context, ttl time.Duration) error

Cleanup calls CleanupFunc.

func (*StoreMock) CleanupCalls added in v1.11.0 

func (mock *StoreMock) CleanupCalls() []struct {
	Ctx context.Context
	TTL time.Duration
}

CleanupCalls gets all the calls that were made to Cleanup. Check the length with: 

len(mockedStore.CleanupCalls())

func (*StoreMock) Commit added in v1.11.0 

func (mock *StoreMock) Commit(id string) error

Commit calls CommitFunc.

func (*StoreMock) CommitCalls added in v1.11.0 

func (mock *StoreMock) CommitCalls() []struct {
	ID string
}

CommitCalls gets all the calls that were made to Commit. Check the length with: 

len(mockedStore.CommitCalls())

func (*StoreMock) Info added in v1.11.0 

func (mock *StoreMock) Info() (StoreInfo, error)

Info calls InfoFunc.

func (*StoreMock) InfoCalls added in v1.11.0 

func (mock *StoreMock) InfoCalls() []struct {
}

InfoCalls gets all the calls that were made to Info. Check the length with: 

len(mockedStore.InfoCalls())

func (*StoreMock) Load added in v1.11.0 

func (mock *StoreMock) Load(id string) ([]byte, error)

Load calls LoadFunc.

func (*StoreMock) LoadCalls added in v1.11.0 

func (mock *StoreMock) LoadCalls() []struct {
	ID string
}

LoadCalls gets all the calls that were made to Load. Check the length with: 

len(mockedStore.LoadCalls())

func (*StoreMock) ResetCleanupTimer added in v1.11.0 

func (mock *StoreMock) ResetCleanupTimer(id string) error

ResetCleanupTimer calls ResetCleanupTimerFunc.

func (*StoreMock) ResetCleanupTimerCalls added in v1.11.0 

func (mock *StoreMock) ResetCleanupTimerCalls() []struct {
	ID string
}

ResetCleanupTimerCalls gets all the calls that were made to ResetCleanupTimer. Check the length with: 

len(mockedStore.ResetCleanupTimerCalls())

func (*StoreMock) Save added in v1.11.0 

func (mock *StoreMock) Save(id string, img []byte) error

Save calls SaveFunc.

func (*StoreMock) SaveCalls added in v1.11.0 

func (mock *StoreMock) SaveCalls() []struct {
	ID  string
	Img []byte
}

SaveCalls gets all the calls that were made to Save. Check the length with: 

len(mockedStore.SaveCalls())

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.