tarcache

Documentation

Overview

Package tarcache supports the creation and running of a pipeline that receives files, tars up the contents, and uploads everything when the tarfile is big enough or the contents are old enough.

Index

• type TarCache
  • func New(rootDirectory filename.System, datatype string, metadata *flagx.KeyValue, ...) (*TarCache, chan<- filename.System)
  • func (t *TarCache) ListenForever(termCtx context.Context, killCtx context.Context)

Types

type TarCache

type TarCache struct {
	// contains filtered or unexported fields
}

TarCache contains everything you need to incrementally create a tarfile. Once enough time has passed since the first file was added OR the resulting tar file has become big enough, it will call the uploadAndDelete() method. To upload a lot of tarfiles, you should only have to create one TarCache. The TarCache takes care of creating each tarfile and getting it uploaded.

func New

func New(rootDirectory filename.System, datatype string, metadata *flagx.KeyValue, sizeThreshold bytecount.ByteCount, ageThreshold memoryless.Config, uploader uploader.Uploader) (*TarCache, chan<- filename.System)

New creates a new TarCache object and returns a pointer to it and the channel used to send data to the TarCache.

func (*TarCache) ListenForever

func (t *TarCache) ListenForever(termCtx context.Context, killCtx context.Context)

ListenForever waits for new files and then uploads them. Using this approach allows us to ensure that all file processing happens in this single thread, no matter whether the processing is happening due to age thresholds or size thresholds.