archiver

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 26, 2019 License: MIT Imports: 35 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConvertHeaderToTags 

func ConvertHeaderToTags(header *HistoryBlobHeader) (map[string]string, error)

ConvertHeaderToTags converts header into metadata tags for blob

func GetHighestVersion added in v0.5.9 

func GetHighestVersion(tags map[string]string) (*int64, error)

GetHighestVersion returns the highest version from index blob tags

func IsLast 

func IsLast(tags map[string]string) bool

IsLast returns true if tags indicate blob is the last blob in archived history, false otherwise

func MaxArchivalIterationTimeout added in v0.5.9 

func MaxArchivalIterationTimeout() time.Duration

MaxArchivalIterationTimeout returns the max allowed timeout for a single iteration of archival workflow

func NewHistoryBlobKey 

func NewHistoryBlobKey(domainID, workflowID, runID string, closeFailoverVersion int64, pageToken int) (blob.Key, error)

NewHistoryBlobKey returns a key for history blob

func NewHistoryIndexBlobKey added in v0.5.9 

func NewHistoryIndexBlobKey(domainID, workflowID, runID string) (blob.Key, error)

NewHistoryIndexBlobKey returns a key for history index blob

func NewNonDeterministicBlobKey added in v0.5.9 

func NewNonDeterministicBlobKey(key blob.Key) (blob.Key, error)

NewNonDeterministicBlobKey returns a key for the non-deterministic history blob given the key for the other blob

func NewReplayMetricsClient 

func NewReplayMetricsClient(client metrics.Client, ctx workflow.Context) metrics.Client

NewReplayMetricsClient creates a metrics client which is aware of cadence's replay mode

func NewReplayMetricsScope added in v0.5.9 

func NewReplayMetricsScope(scope metrics.Scope, ctx workflow.Context) metrics.Scope

NewReplayMetricsScope creates a metrics scope which is aware of cadence's replay mode

Types

type ArchiveRequest 

type ArchiveRequest struct {
	ShardID              int
	DomainID             string
	DomainName           string
	WorkflowID           string
	RunID                string
	EventStoreVersion    int32
	BranchToken          []byte
	NextEventID          int64
	CloseFailoverVersion int64
	BucketName           string
}

ArchiveRequest is request to Archive

type Archiver 

type Archiver interface {
	Start()
	Finished() []uint64
}

Archiver is used to process archival requests

func NewArchiver 

func NewArchiver(
	ctx workflow.Context,
	logger log.Logger,
	metricsClient metrics.Client,
	concurrency int,
	requestCh workflow.Channel,
) Archiver

NewArchiver returns a new Archiver

type BootstrapContainer 

type BootstrapContainer struct {
	PublicClient     workflowserviceclient.Interface
	MetricsClient    metrics.Client
	Logger           log.Logger
	ClusterMetadata  cluster.Metadata
	HistoryManager   persistence.HistoryManager
	HistoryV2Manager persistence.HistoryV2Manager
	Blobstore        blobstore.Client
	DomainCache      cache.DomainCache
	Config           *Config
	HistoryArchivers map[string]carchiver.HistoryArchiver

	// the following are only set in testing code
	HistoryBlobReader     HistoryBlobReader
	HistorySizeEstimator  SizeEstimator
	HistoryBlobDownloader HistoryBlobDownloader
}

BootstrapContainer contains everything need for bootstrapping

type Client 

type Client interface {
	Archive(*ArchiveRequest) error
}

Client is used to archive workflow histories

func NewClient 

func NewClient(
	metricsClient metrics.Client,
	logger log.Logger,
	publicClient workflowserviceclient.Interface,
	numWorkflows dynamicconfig.IntPropertyFn,
	requestRPS dynamicconfig.IntPropertyFn,
) Client

NewClient creates a new Client

type ClientMock 

type ClientMock struct {
	mock.Mock
}

ClientMock is an autogenerated mock type for the Client type

func (*ClientMock) Archive 

func (_m *ClientMock) Archive(_a0 *ArchiveRequest) error

Archive provides a mock function with given fields: _a0

type ClientWorker 

type ClientWorker interface {
	Start() error
	Stop()
}

ClientWorker is a cadence client worker

func NewClientWorker 

func NewClientWorker(container *BootstrapContainer) ClientWorker

NewClientWorker returns a new ClientWorker

type Config 

type Config struct {
	EnableArchivalCompression                 dynamicconfig.BoolPropertyFnWithDomainFilter
	HistoryPageSize                           dynamicconfig.IntPropertyFnWithDomainFilter
	TargetArchivalBlobSize                    dynamicconfig.IntPropertyFnWithDomainFilter
	ArchiverConcurrency                       dynamicconfig.IntPropertyFn
	ArchivalsPerIteration                     dynamicconfig.IntPropertyFn
	DeterministicConstructionCheckProbability dynamicconfig.FloatPropertyFn
	BlobIntegrityCheckProbability             dynamicconfig.FloatPropertyFn
	TimeLimitPerArchivalIteration             dynamicconfig.DurationPropertyFn
}

Config for ClientWorker

type DownloadBlobRequest added in v0.5.9 

type DownloadBlobRequest struct {
	NextPageToken        []byte
	ArchivalBucket       string
	DomainID             string
	WorkflowID           string
	RunID                string
	CloseFailoverVersion *int64
}

DownloadBlobRequest is request to DownloadBlob

type DownloadBlobResponse added in v0.5.9 

type DownloadBlobResponse struct {
	NextPageToken []byte
	HistoryBlob   *HistoryBlob
}

DownloadBlobResponse is response from DownloadBlob

type HistoryBlob 

type HistoryBlob struct {
	Header *HistoryBlobHeader `json:"header"`
	Body   *shared.History    `json:"body"`
}

HistoryBlob is the serializable data that forms the body of a blob

type HistoryBlobDownloader added in v0.5.9 

type HistoryBlobDownloader interface {
	DownloadBlob(context.Context, *DownloadBlobRequest) (*DownloadBlobResponse, error)
}

HistoryBlobDownloader is used to download history blobs

func NewHistoryBlobDownloader added in v0.5.9 

func NewHistoryBlobDownloader(blobstoreClient blobstore.Client) HistoryBlobDownloader

NewHistoryBlobDownloader returns a new HistoryBlobDownloader

type HistoryBlobDownloaderMock added in v0.5.9 

type HistoryBlobDownloaderMock struct {
	mock.Mock
}

HistoryBlobDownloaderMock is an autogenerated mock type for the HistoryBlobDownloader type

func (*HistoryBlobDownloaderMock) DownloadBlob added in v0.5.9 

func (_m *HistoryBlobDownloaderMock) DownloadBlob(_a0 context.Context, _a1 *DownloadBlobRequest) (*DownloadBlobResponse, error)

DownloadBlob provides a mock function with given fields: _a0, _a1

type HistoryBlobHeader 

type HistoryBlobHeader struct {
	DomainName           *string `json:"domain_name,omitempty"`
	DomainID             *string `json:"domain_id,omitempty"`
	WorkflowID           *string `json:"workflow_id,omitempty"`
	RunID                *string `json:"run_id,omitempty"`
	CurrentPageToken     *int    `json:"current_page_token,omitempty"`
	NextPageToken        *int    `json:"next_page_token,omitempty"`
	IsLast               *bool   `json:"is_last,omitempty"`
	FirstFailoverVersion *int64  `json:"first_failover_version,omitempty"`
	LastFailoverVersion  *int64  `json:"last_failover_version,omitempty"`
	FirstEventID         *int64  `json:"first_event_id,omitempty"`
	LastEventID          *int64  `json:"last_event_id,omitempty"`
	UploadDateTime       *string `json:"upload_date_time,omitempty"`
	UploadCluster        *string `json:"upload_cluster,omitempty"`
	EventCount           *int64  `json:"event_count,omitempty"`
	CloseFailoverVersion *int64  `json:"close_failover_version,omitempty"`
}

HistoryBlobHeader is the header attached to all history blobs

type HistoryBlobIterator 

type HistoryBlobIterator interface {
	Next() (*HistoryBlob, error)
	HasNext() bool
	GetState() ([]byte, error)
}

HistoryBlobIterator is used to get history blobs

func NewHistoryBlobIterator 

func NewHistoryBlobIterator(
	request ArchiveRequest,
	container *BootstrapContainer,
	domainName string,
	clusterName string,
	initialState []byte,
) (HistoryBlobIterator, error)

NewHistoryBlobIterator returns a new HistoryBlobIterator

type HistoryBlobIteratorMock 

type HistoryBlobIteratorMock struct {
	mock.Mock
}

HistoryBlobIteratorMock is an autogenerated mock type for the HistoryBlobIterator type

func (*HistoryBlobIteratorMock) GetState added in v0.6.0 

func (_m *HistoryBlobIteratorMock) GetState() ([]byte, error)

GetState provides a mock function with given fields:

func (*HistoryBlobIteratorMock) HasNext 

func (_m *HistoryBlobIteratorMock) HasNext() bool

HasNext provides a mock function with given fields:

func (*HistoryBlobIteratorMock) Next 

func (_m *HistoryBlobIteratorMock) Next() (*HistoryBlob, error)

Next provides a mock function with given fields:

type HistoryBlobReader 

type HistoryBlobReader interface {
	GetBlob(pageToken int) (*HistoryBlob, error)
}

HistoryBlobReader is used to read history blobs

func NewHistoryBlobReader 

func NewHistoryBlobReader(itr HistoryBlobIterator) HistoryBlobReader

NewHistoryBlobReader returns a new HistoryBlobReader

type HistoryBlobReaderMock 

type HistoryBlobReaderMock struct {
	mock.Mock
}

HistoryBlobReaderMock is an autogenerated mock type for the HistoryBlobReader type

func (*HistoryBlobReaderMock) GetBlob 

func (_m *HistoryBlobReaderMock) GetBlob(pageToken int) (*HistoryBlob, error)

GetBlob provides a mock function with given fields: pageToken

type HistoryBlobUploader added in v0.6.0 

type HistoryBlobUploader interface {
	UploadHistory(context.Context, *ArchiveRequest) (*uploadResult, error)
}

HistoryBlobUploader is used to upload history blobs

func NewHistoryBlobUploader added in v0.6.0 

func NewHistoryBlobUploader() HistoryBlobUploader

NewHistoryBlobUploader returns a new HistoryBlobUploader

type MockArchiver 

type MockArchiver struct {
	mock.Mock
}

MockArchiver is an autogenerated mock type for the Archiver type

func (*MockArchiver) Finished 

func (_m *MockArchiver) Finished() []uint64

Finished provides a mock function with given fields:

func (*MockArchiver) Start 

func (_m *MockArchiver) Start()

Start provides a mock function with given fields:

type Pump 

type Pump interface {
	Run() PumpResult
}

Pump pumps archival requests into request channel

func NewPump 

func NewPump(
	ctx workflow.Context,
	logger log.Logger,
	metricsClient metrics.Client,
	carryover []ArchiveRequest,
	timeout time.Duration,
	requestLimit int,
	requestCh workflow.Channel,
	signalCh workflow.Channel,
) Pump

NewPump returns a new Pump

type PumpMock 

type PumpMock struct {
	mock.Mock
}

PumpMock is an autogenerated mock type for the Pump type

func (*PumpMock) Run 

func (_m *PumpMock) Run() PumpResult

Run provides a mock function with given fields:

type PumpResult 

type PumpResult struct {
	PumpedHashes          []uint64
	UnhandledCarryover    []ArchiveRequest
	TimeoutWithoutSignals bool
}

PumpResult is the result of pumping requests into request channel

type SizeEstimator added in v0.5.9 

type SizeEstimator interface {
	EstimateSize(v interface{}) (int, error)
}

SizeEstimator is used to estimate the size of any object

func NewJSONSizeEstimator added in v0.5.9 

func NewJSONSizeEstimator() SizeEstimator

NewJSONSizeEstimator returns a new SizeEstimator which uses json encoding to estimate size

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.