Version: v0.0.0-...-d33463d Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jun 12, 2020 License: BSD-3-Clause Imports: 12 Imported by: 0



Package storage of the Dataflow kit describes Store interface for read/ write operations with downloaded data and parsed results.

Storage types like Diskv, MongoDB implement methods which satisfy Store interface.



View Source
const (
	CACHE        = "Cache"
	COOKIES      = "Cookies"
	INTERMEDIATE = "Intermediate"

Storage types


This section is empty.


This section is empty.


type Record

type Record struct {
	Type    string
	Key     string
	Value   []byte
	ExpTime int64

Record struct keeps Key/Value and expiration time of specified type

type Store

type Store interface {
	//Reads value from storage by specified key
	Read(rec Record) (value []byte, err error)
	//Writes specified pair key value to storage.
	//expTime value sets TTL for Redis storage.
	//expTime set Metadata Expires value for S3Storage
	Write(rec Record) error
	// IsExists checkes whether specified by key record exists
	IsExists(rec Record) bool
	//Is key expired ? It checks if parse results storage item is expired. Set up  Expiration as "ITEM_EXPIRE_IN" environment variable.
	//html pages cache stores this info in sResponse.Expires . It is not used for fetch endpoint.
	Expired(rec Record) bool
	//Delete deletes specified item from the store
	Delete(rec Record) error
	//DeleteAll erases all items from the store
	DeleteAll() error
	// Close storage connection

Store is the key interface of storage. All other structs implement methods wchich satisfy that interface.

func NewStore

func NewStore(sType string) Store

NewStore creates New initialized Store instance with predefined parameters Storage Types: S3, Spaces, Redis, Diskv

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL