reconciler

README

Reconciler

The Reconciler package is used to ensure that balance changes derived from parsing Rosetta blocks are equivalent to the balance changes computed by the node. If you want to see an example of how to use this package, take a look at rosetta-cli.

Features

• Customizable Helper and Handler to define your own logic for retrieving calculated balance changes and handling successful/unsuccessful comparisons
• Perform balance lookup at either historical blocks or the current block (if historical balance query is not supported)
• Provide a list of accounts to compare at each block (for quick and easy debugging)

Installation

go get github.com/foundryservices/rosetta-sdk-go/reconciler

Reconciliation Strategies

Active Addresses

The reconciler checks that the balance of an account computed by its operations is equal to the balance of the account according to the node. If this balance is not identical, the reconciler will error.

Inactive Addresses

The reconciler randomly checks the balances of accounts that aren't involved in any transactions. The balances of accounts could change on the blockchain node without being included in an operation returned by the Rosetta Data API. Recall that all balance-changing operations must be returned by the Rosetta Data API.

Documentation

Index

• Constants
• Variables
• func ContainsAccountCurrency(m map[string]struct{}, change *types.AccountCurrency) bool
• func Err(err error) bool
• type Handler
• type Helper
• type InactiveEntry
• type Option
  • func WithActiveConcurrency(concurrency int) Option
  • func WithBacklogSize(size int) Option
  • func WithBalancePruning() Option
  • func WithDebugLogging() Option
  • func WithInactiveConcurrency(concurrency int) Option
  • func WithInactiveFrequency(blocks int64) Option
  • func WithInterestingAccounts(interesting []*types.AccountCurrency) Option
  • func WithLookupBalanceByBlock() Option
  • func WithSeenAccounts(seen []*types.AccountCurrency) Option
• type Reconciler
  • func New(helper Helper, handler Handler, p *parser.Parser, options ...Option) *Reconciler
  • func (r *Reconciler) CompareBalance(ctx context.Context, account *types.AccountIdentifier, ...) (string, string, int64, error)
  • func (r *Reconciler) LastIndexReconciled() int64
  • func (r *Reconciler) QueueChanges(ctx context.Context, block *types.BlockIdentifier, ...) error
  • func (r *Reconciler) QueueSize() int
  • func (r *Reconciler) Reconcile(ctx context.Context) error

Constants

const (
	// ActiveReconciliation is included in the reconciliation
	// error message if reconciliation failed during active
	// reconciliation.
	ActiveReconciliation = "ACTIVE"

	// InactiveReconciliation is included in the reconciliation
	// error message if reconciliation failed during inactive
	// reconciliation.
	InactiveReconciliation = "INACTIVE"
)

const (
	// BlockGone is when the block where a reconciliation
	// is supposed to happen is orphaned.
	BlockGone = "BLOCK_GONE"

	// HeadBehind is when the synced tip (where balances
	// were last computed) is behind the *types.BlockIdentifier
	// returned by the call to /account/balance.
	HeadBehind = "HEAD_BEHIND"

	// BacklogFull is when the reconciliation backlog is full.
	BacklogFull = "BACKLOG_FULL"

	// TipFailure is returned when looking up the live
	// balance fails but we are at tip. This usually occurs
	// when the node processes an re-org that we have yet
	// to process (so the index we are querying at may be
	// ahead of the nodes tip).
	TipFailure = "TIP_FAILURE"

	// AccountMissing is returned when looking up computed
	// balance fails because the account does not exist in
	// balance storage.
	// This can happen when interesting accounts
	// are specified. We try to reconcile balances for
	// each of these accounts at each block height.
	// But, until we encounter a block with an interesting account
	// in it, there is no entry for it in balance storage.
	// So, we can not reconcile.
	AccountMissing = "ACCOUNT_MISSING"
)

Variables

var (
	// ErrHeadBlockBehindLive is returned when the processed
	// head is behind the live head. Sometimes, it is
	// preferable to sleep and wait to catch up when
	// we are close to the live head (waitToCheckDiff).
	ErrHeadBlockBehindLive = errors.New("head block behind")

	// ErrBlockGone is returned when the processed block
	// head is greater than the live head but the block
	// does not exist in the store. This likely means
	// that the block was orphaned.
	ErrBlockGone = errors.New("block gone")

	ErrGetCurrentBlockFailed    = errors.New("unable to get current block for reconciliation")
	ErrBlockExistsFailed        = errors.New("unable to check if block exists")
	ErrGetComputedBalanceFailed = errors.New("unable to get computed balance")
	ErrLiveBalanceLookupFailed  = errors.New("unable to lookup live balance")
)

Named error types for Reconciler errors

Functions

func ContainsAccountCurrency

func ContainsAccountCurrency(
	m map[string]struct{},
	change *types.AccountCurrency,
) bool

ContainsAccountCurrency returns a boolean indicating if a AccountCurrency set already contains an Account and Currency combination.

func Err

func Err(err error) bool

Err takes an error as an argument and returns whether or not the error is one thrown by the reconciler package

Types

type Handler

type Handler interface {
	ReconciliationFailed(
		ctx context.Context,
		reconciliationType string,
		account *types.AccountIdentifier,
		currency *types.Currency,
		computedBalance string,
		liveBalance string,
		block *types.BlockIdentifier,
	) error

	ReconciliationSucceeded(
		ctx context.Context,
		reconciliationType string,
		account *types.AccountIdentifier,
		currency *types.Currency,
		balance string,
		block *types.BlockIdentifier,
	) error

	ReconciliationExempt(
		ctx context.Context,
		reconciliationType string,
		account *types.AccountIdentifier,
		currency *types.Currency,
		computedBalance string,
		liveBalance string,
		block *types.BlockIdentifier,
		exemption *types.BalanceExemption,
	) error

	ReconciliationSkipped(
		ctx context.Context,
		reconciliationType string,
		account *types.AccountIdentifier,
		currency *types.Currency,
		cause string,
	) error
}

Handler is called by Reconciler after a reconciliation is performed. When a reconciliation failure is observed, it is up to the client to trigger a halt (by returning an error) or to continue (by returning nil).

type Helper

type Helper interface {
	DatabaseTransaction(ctx context.Context) database.Transaction

	CurrentBlock(
		ctx context.Context,
		dbTx database.Transaction,
	) (*types.BlockIdentifier, error)

	IndexAtTip(
		ctx context.Context,
		index int64,
	) (bool, error)

	CanonicalBlock(
		ctx context.Context,
		dbTx database.Transaction,
		block *types.BlockIdentifier,
	) (bool, error)

	ComputedBalance(
		ctx context.Context,
		dbTx database.Transaction,
		account *types.AccountIdentifier,
		currency *types.Currency,
		index int64,
	) (*types.Amount, error)

	LiveBalance(
		ctx context.Context,
		account *types.AccountIdentifier,
		currency *types.Currency,
		index int64,
	) (*types.Amount, *types.BlockIdentifier, error)

	// PruneBalances is invoked by the reconciler
	// to indicate that all historical balance states
	// <= to index can be removed.
	PruneBalances(
		ctx context.Context,
		account *types.AccountIdentifier,
		currency *types.Currency,
		index int64,
	) error

	// ForceInactiveReconciliation is invoked by the
	// inactive reconciler when the next account to
	// reconcile has been checked within the configured
	// inactive reconciliation frequency. This allows
	// the helper to dynamically force inactive reconciliation
	// when desired (i.e. when at tip).
	ForceInactiveReconciliation(
		ctx context.Context,
		account *types.AccountIdentifier,
		currency *types.Currency,
		lastCheck *types.BlockIdentifier,
	) bool
}

Helper functions are used by Reconciler to compare computed balances from a block with the balance calculated by the node. Defining an interface allows the client to determine what sort of storage layer they want to use to provide the required information.

type InactiveEntry

type InactiveEntry struct {
	Entry     *types.AccountCurrency
	LastCheck *types.BlockIdentifier
}

InactiveEntry is used to track the last time that an *types.AccountCurrency was reconciled.

type Option

type Option func(r *Reconciler)

Option is used to overwrite default values in Reconciler construction. Any Option not provided falls back to the default value.

func WithActiveConcurrency

func WithActiveConcurrency(concurrency int) Option

WithActiveConcurrency overrides the default active concurrency.

func WithBacklogSize

func WithBacklogSize(size int) Option

WithBacklogSize overrides the defaultBacklogSize to some new size. This is often useful for blockchains that have high network activity.

func WithBalancePruning

func WithBalancePruning() Option

WithBalancePruning determines if historical balance states should be pruned after they are used. This can prevent storage blowup if historical states are only ever used once.

func WithDebugLogging

func WithDebugLogging() Option

WithDebugLogging determines if verbose logs should be printed.

func WithInactiveConcurrency

func WithInactiveConcurrency(concurrency int) Option

WithInactiveConcurrency overrides the default inactive concurrency.

func WithInactiveFrequency

func WithInactiveFrequency(blocks int64) Option

WithInactiveFrequency is how many blocks the reconciler should wait between inactive reconciliations on each account.

func WithInterestingAccounts

func WithInterestingAccounts(interesting []*types.AccountCurrency) Option

WithInterestingAccounts adds interesting accounts to check at each block.

func WithLookupBalanceByBlock

func WithLookupBalanceByBlock() Option

WithLookupBalanceByBlock sets lookupBlockByBalance to false.

func WithSeenAccounts

func WithSeenAccounts(seen []*types.AccountCurrency) Option

WithSeenAccounts adds accounts to the seenAccounts slice and inactiveQueue for inactive reconciliation.

type Reconciler

type Reconciler struct {

	// Reconciler concurrency is separated between
	// active and inactive concurrency to allow for
	// fine-grained tuning of reconciler behavior.
	// When there are many transactions in a block
	// on a resource-constrained machine (laptop),
	// it is useful to allocate more resources to
	// active reconciliation as it is synchronous
	// (when lookupBalanceByBlock is enabled).
	ActiveConcurrency   int
	InactiveConcurrency int
	// contains filtered or unexported fields
}

Reconciler contains all logic to reconcile balances of types.AccountIdentifiers returned in types.Operations by a Rosetta Server.

func New

func New(
	helper Helper,
	handler Handler,
	p *parser.Parser,
	options ...Option,
) *Reconciler

New creates a new Reconciler.

func (*Reconciler) CompareBalance

func (r *Reconciler) CompareBalance(
	ctx context.Context,
	account *types.AccountIdentifier,
	currency *types.Currency,
	liveBalance string,
	liveBlock *types.BlockIdentifier,
) (string, string, int64, error)

CompareBalance checks to see if the computed balance of an account is equal to the live balance of an account. This function ensures balance is checked correctly in the case of orphaned blocks.

We must transactionally fetch the last synced head, whether the liveBlock is canonical, and the computed balance of the *types.AccountIdentifier and *types.Currency.

func (*Reconciler) LastIndexReconciled

func (r *Reconciler) LastIndexReconciled() int64

LastIndexReconciled is the last block index reconciled. This is used to ensure all the enqueued accounts for a particular block have been reconciled.

func (*Reconciler) QueueChanges

func (r *Reconciler) QueueChanges(
	ctx context.Context,
	block *types.BlockIdentifier,
	balanceChanges []*parser.BalanceChange,
) error

QueueChanges enqueues a slice of *BalanceChanges for reconciliation.

func (*Reconciler) QueueSize

func (r *Reconciler) QueueSize() int

QueueSize is a helper that returns the total number of items currently enqueued for active reconciliation.

func (*Reconciler) Reconcile

func (r *Reconciler) Reconcile(ctx context.Context) error

Reconcile starts the active and inactive Reconciler goroutines. If any goroutine errors, the function will return an error.