dedupe

Documentation

Index

• type Deduplicator
• type EmbeddedDeduplicator
  • func NewEmbedded(dir string) (*EmbeddedDeduplicator, error)
  • func (d *EmbeddedDeduplicator) CheckAndMark(_ context.Context, eventID string) (bool, error)
  • func (d *EmbeddedDeduplicator) Close() error
  • func (d *EmbeddedDeduplicator) Stats() map[string]int64

Types

type Deduplicator

type Deduplicator interface {
	// CheckAndMark returns true if the event was already seen (duplicate).
	// If not seen, it atomically marks the event as seen.
	CheckAndMark(ctx context.Context, eventID string) (isDuplicate bool, err error)

	Stats() map[string]int64

	// Close releases resources held by the deduplicator.
	Close() error
}

Deduplicator checks whether an event has been seen before and marks it.

type EmbeddedDeduplicator

type EmbeddedDeduplicator struct {
	// contains filtered or unexported fields
}

EmbeddedDeduplicator uses Pebble for local deduplication.

func NewEmbedded

func NewEmbedded(dir string) (*EmbeddedDeduplicator, error)

NewEmbedded opens a Pebble database at the given path.

func (*EmbeddedDeduplicator) CheckAndMark

func (d *EmbeddedDeduplicator) CheckAndMark(_ context.Context, eventID string) (bool, error)

CheckAndMark returns true if the event was already seen.

func (*EmbeddedDeduplicator) Close

func (d *EmbeddedDeduplicator) Close() error

func (*EmbeddedDeduplicator) Stats

func (d *EmbeddedDeduplicator) Stats() map[string]int64