index

Documentation

Index

• func Normalize(s string) string
• func NormalizeCountry(s string) string
• func NormalizeURL(s string) string
• func ParseQuery(index string, query map[string]interface{}, norm Normalizer) ([]string, bool)
• type Container
  • func (c Container) Add(key, value string, norm Normalizer) bool
  • func (c Container) Contains(key string, value string, norm Normalizer) bool
  • func (c Container) Dump() (data []byte, err error)
  • func (c Container) Find(key string, norm Normalizer) (values []string, ok bool)
  • func (c Container) Load(data []byte) (err error)
  • func (c Container) Remove(key, value string, norm Normalizer) bool
  • func (c Container) Reverse(value string, norm Normalizer) ([]string, bool)
• type Index
• type MultiIndex
  • func NewCategoryIndex() MultiIndex
  • func NewCountryIndex() MultiIndex
• type Normalizer
• type Searcher
• type Sequence
  • func (c Sequence) Dump() (data []byte, err error)
  • func (c Sequence) Load(data []byte) (s Sequence, err error)
• type Serializer
• type SingleIndex
  • func NewNamesIndex() SingleIndex
  • func NewWebsiteIndex() SingleIndex
• type Unique
  • func (c Unique) Add(key, value string, norm Normalizer) bool
  • func (c Unique) Dump() (_ []byte, err error)
  • func (c Unique) Find(key string, norm Normalizer) (value string, ok bool)
  • func (c Unique) Load(data []byte) (err error)
  • func (c Unique) Overwrite(key, value string, norm Normalizer) bool
  • func (c Unique) Remove(key string, norm Normalizer) bool
  • func (c Unique) Reverse(value string, norm Normalizer) ([]string, bool)

Functions

func Normalize

func Normalize(s string) string

Normalize an index value in preparation for index storage and lookups.

func NormalizeCountry

func NormalizeCountry(s string) string

Normalize country attempts to return an ISO 3166-1 alpha-2 code for the specified country, then stores that as the index record. If the normalized record can't be found then standard normalization on the string is used.

func NormalizeURL

func NormalizeURL(s string) string

Normalize URL parses a URL and only returns the hostname, if an error occurs, then an empty string is returned so the URL doesn't get added to the index.

func ParseQuery

func ParseQuery(index string, query map[string]interface{}, norm Normalizer) ([]string, bool)

Queries are maps that hold an index name (e.g. "name" or "country") and map it to an indexable key in the index. The query can be either a single string or a list of strings; the parse function extracts the appropriate type and returns a list of strings if the query for the particular index name exists.

All values are normalized by the normalize function if provided.

Types

type Container

type Container map[string][]string

Container indices map a value to a set of values, most commonly an index to a set of record IDs that are related to that value. The container index is maintained in sorted order with no duplicates to save space and enhance ordered search.

func (Container) Add

func (c Container) Add(key, value string, norm Normalizer) bool

Add a record ID to the index for the specified value. This method ensures that the list of records is maintained in sorted order with no duplicates.

func (Container) Contains

func (c Container) Contains(key string, value string, norm Normalizer) bool

Contains determines if the value is contained by the key index

func (Container) Dump

func (c Container) Dump() (data []byte, err error)

Dump a container index to a byte representation for storage on disk.

func (Container) Find

func (c Container) Find(key string, norm Normalizer) (values []string, ok bool)

Find all values for the specified key in the index, returns nil if it doesn't exist.

func (Container) Load

func (c Container) Load(data []byte) (err error)

Load a container index from a byte representation on disk.

func (Container) Remove

func (c Container) Remove(key, value string, norm Normalizer) bool

Remove a value from the container specified by the key, normalizing the key if necessary. This method ensures that the containers are maintained in sorted order.

func (Container) Reverse

func (c Container) Reverse(value string, norm Normalizer) ([]string, bool)

Reverse find - find all keys that index the specified value.

type Index

type Index interface {
	Len() int
	Empty() bool
	Search(query map[string]interface{}) []string
	Add(key, value string) bool
	Reverse(value string) ([]string, bool)
}

Index types provide in-memory map index and helper methods for lookups and constraints to improve the performance of the store without disk access. The indices are intended to be checkpointed and synchronized to disk regularly but can also be rebuilt by stores that are Indexers.

The indices map normalized string values (lower case, extra whitespace removed) to base64 encoded []byte keys or to UUID strings depending on the context.

type MultiIndex

type MultiIndex interface {
	Index
	Serializer
	Find(key string) ([]string, bool)
	Remove(key, value string) bool
	Contains(key, value string) bool
}

MultiIndex is not the best term for this but we will refactor this during the Trtl indexing sprint. This index is used to done a mapping of indexed value to multiple record IDs.

func NewCategoryIndex

func NewCategoryIndex() MultiIndex

func NewCountryIndex

func NewCountryIndex() MultiIndex

type Normalizer

type Normalizer func(s string) string

Normalize specifies how to modify a string value in preparation for search.

type Searcher

type Searcher func(query map[string]interface{}) []string

Searcher is a function that takes a query, parses it and returns a list of the found values in the index. Searcher functions are produced by the ExactMatch, PrefixMatch, and ContainsRecord functions to be used with different Indices.

type Sequence

type Sequence uint64

An auto-increment primary key Sequence for generating monotonically increasing IDs

func (Sequence) Dump

func (c Sequence) Dump() (data []byte, err error)

func (Sequence) Load

func (c Sequence) Load(data []byte) (s Sequence, err error)

type Serializer

type Serializer interface {
	Load([]byte) error
	Dump() ([]byte, error)
}

Serializer allows indices to be loaded and dumped to disk.

type SingleIndex

type SingleIndex interface {
	Index
	Serializer
	Find(key string) (string, bool)
	Remove(key string) bool
	Overwrite(key, value string) bool
}

SingleIndex is not the best term for this but we will refactor this during the Trtl indexing sprint. This index is used to denote a mapping of indexed value to record ID.

func NewNamesIndex

func NewNamesIndex() SingleIndex

func NewWebsiteIndex

func NewWebsiteIndex() SingleIndex

type Unique

type Unique map[string]string

Unique indices map a key to a specified value, usually a search term or unique value to a record ID. This index prevents duplicates across records in the database and is also used for fast lookups matching values to documents.

func (Unique) Add

func (c Unique) Add(key, value string, norm Normalizer) bool

Add an entry to the unique index, normalizing if necessary. If the entry is already in the index, it will not be modified and false will be returned. See overwrite if the entry needs to be replaced.

func (Unique) Dump

func (c Unique) Dump() (_ []byte, err error)

Dump a unique index to a byte representation for storage on disk.

func (Unique) Find

func (c Unique) Find(key string, norm Normalizer) (value string, ok bool)

func (Unique) Load

func (c Unique) Load(data []byte) (err error)

Load a unique index from a byte representation on disk.

func (Unique) Overwrite

func (c Unique) Overwrite(key, value string, norm Normalizer) bool

Add or replace an entry that exists in the index.

func (Unique) Remove

func (c Unique) Remove(key string, norm Normalizer) bool

func (Unique) Reverse

func (c Unique) Reverse(value string, norm Normalizer) ([]string, bool)