db

package
Version: v0.0.0-...-89d8475 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 26, 2017 License: Apache-2.0, MIT Imports: 9 Imported by: 0

Documentation

Index

Constants

View Source
const DBLocation string = "poe.db"

DBLocation is the on-disk file containing our database

View Source
const GGGIDSize = 10

GGGIDSize is the size in bytes a derived ID can be

View Source
const IDSize = 8

IDSize is the size in bytes an internal identifier can be

View Source
const ItemModAverageScaleFactor = 10

ItemModAverageScaleFactor is the multiplier applied to the average of multi-mod items. This allows a fixed precision of averages.

Any more than 10 and we could risk overflowing our uint16

View Source
const LeagueHeapIDSize = 2

LeagueHeapIDSize is the number of bytes taken up by a LeagueHeapID when converted to bytes

View Source
const LookupItemsMultiModStrideLength = 32

LookupItemsMultiModStrideLength determines how many items is included in a stride of LookupItemsMultiMod.

Longer strides mean fewer intersections but more potentially useless item mods checked.

View Source
const ModIndexKeySuffixLength = TimestampSize

ModIndexKeySuffixLength allows us to fetch variable numbers of pre-pended values given their length.

View Source
const StringHeapIDSize = 4

StringHeapIDSize is the number of bytes a StringHeapID takes up when converted to bytes.

View Source
const TimestampBucketAccuracyReduction = 7 // 7 representing ~2.1 minute buckets

TimestampBucketAccuracyReduction is the number of rightward shifts applied to a Timestamp to discretely bucket it for Indexing.

View Source
const TimestampSize = 8 - 4

TimestampSize is the number of bytes used by Timestamp

This is sized to minimize waste while keeping accuracy

We discard the first 4 bytes as they're useless for our purposes.

Variables

This section is empty.

Functions

func AddItems

func AddItems(items []Item, db *bolt.DB) (int, error)

AddItems adds tbe given items to their correct paths in the database

Provided items CAN differ in their league.

This is simply a wrapper for addItems that also establishes the transaction as well as include the timestamp

func Boot

func Boot(path string) (*bolt.DB, error)

Boot gets the database from disk and performs necessary setup

If path is empty, it uses the default DBLocation

func DeindexItems

func DeindexItems(items []Item, tx *bolt.Tx) error

DeindexItems removes tbe given items from their correct indices

If an index entry cannot be removed, we return an error. This ensures all existing index entries must be alive

func GetTranslations

func GetTranslations(items []Item, db *bolt.DB) error

GetTranslations associates each provided item with an interal ID if it has not already been assigned one.

This modifies the provided items if they are assigned an ID

func IndexEntryCount

func IndexEntryCount(db *bolt.DB) (int, error)

IndexEntryCount returns the number of index entries across all leagues

func IndexItems

func IndexItems(items []Item, tx *bolt.Tx) (int, error)

IndexItems adds tbe given items to their correct indices for efficient lookup. Returns number of index entries added.

Provided items CAN differ in their league.

func InflateLeague

func InflateLeague(id LeagueHeapID, db *bolt.DB) string

InflateLeague returns the string represenation of the given LeagueHeapID

If you are providing a valid LeagueHeapID, this should never fail.

func InflateString

func InflateString(id StringHeapID, db *bolt.DB) string

InflateString returns the string represenation of the given StringHeapID

If you are providing a valid StringHeapID, this should never fail.

func ItemStoreCount

func ItemStoreCount(db *bolt.DB) (int, error)

ItemStoreCount returns the number of items across all leagues

func ListLeagues

func ListLeagues(db *bolt.DB) ([]string, error)

ListLeagues returns a list of all stored leagues

func RecordChangeIDWasEntered

func RecordChangeIDWasEntered(changeID string, db *bolt.DB) error

RecordChangeIDWasEntered adds a given changeID to the database with its value being the json time it was entered

func RemoveItems

func RemoveItems(ids []ID, league LeagueHeapID, db *bolt.DB) error

RemoveItems removes the provided IDs from the database

All provided IDs must be under the same League.

For higher performance, callers can sort the IDs to allow for more sequential access behaviors when handling the heap

func UpdateWasEntered

func UpdateWasEntered(changeID string, db *bolt.DB) (bool, error)

UpdateWasEntered determines if a given update has been entered into the database

Types

type GGGID

type GGGID [GGGIDSize]byte

GGGID is an Identifier derived from per-item/stash tab UID

A PID is 80 bits = 10 bytes, this allows 2^40 items to be represented taking into birthdays and represents significant savings relative to the GGG api provided id

func GGGIDFromUID

func GGGIDFromUID(uid string) GGGID

GGGIDFromUID generates an ID for internal use from a UID string

func (*GGGID) DecodeMsg

func (z *GGGID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*GGGID) EncodeMsg

func (z *GGGID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*GGGID) MarshalMsg

func (z *GGGID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*GGGID) Msgsize

func (z *GGGID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*GGGID) UnmarshalMsg

func (z *GGGID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type ID

type ID [IDSize]byte

ID is an Identifier calculated internally

This is effectively just a 64 bit uint

func GetGGGIDTranslations

func GetGGGIDTranslations(gggs []GGGID, league LeagueHeapID,
	tx *bolt.Tx) ([]ID, error)

GetGGGIDTranslations associates each provided item with an interal ID if it has not already been assigned one.

All IDs are in the context of the provided league, hence this is typically used for managing individual stash updates and that's why we take a transaction.

This returns the newly translated ID positionally mapping to the provided GGGIDs.

func IDFromSequence

func IDFromSequence(seq uint64) ID

IDFromSequence converts a sequence number into an identifier

func (*ID) DecodeMsg

func (z *ID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*ID) EncodeMsg

func (z *ID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*ID) MarshalMsg

func (z *ID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*ID) Msgsize

func (z *ID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*ID) UnmarshalMsg

func (z *ID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type IDMapPool

type IDMapPool struct {
	// contains filtered or unexported fields
}

IDMapPool represents a thread-safe pool of reusable sets for IndexQuery

func NewIDMapPool

func NewIDMapPool(maxPerSize int) IDMapPool

NewIDMapPool creates a properly initialized IDMapPool

func (*IDMapPool) Borrow

func (p *IDMapPool) Borrow(minLen int) map[ID]struct{}

Borrow returns a pre-allocated buffer in the pool or a fresh buffer if none are available

func (*IDMapPool) Give

func (p *IDMapPool) Give(buf map[ID]struct{})

Give returns a buffer to the set

If we have more buffers than we are configured to handle, further provided buffers are tossed out.

type IndexEntry

type IndexEntry []byte

IndexEntry represents bytes interpreted as an entry within the index

Whenever possible, we avoid allocations.

func IndexEntryAppend

func IndexEntryAppend(entry IndexEntry, id ID) IndexEntry

IndexEntryAppend adds another ID to the entry

If an id is already present in the id, we end up with a duplicate. Such is life.

func IndexEntryRemove

func IndexEntryRemove(entry IndexEntry, id ID) IndexEntry

IndexEntryRemove removes a given ID from the entry

If the ID is the last of the entry, the backing slice is set to nil. In that case, its the callers responsibility to ensure they Unwrap a valid byte slice.

func (IndexEntry) ForEachID

func (entry IndexEntry) ForEachID(cb func(id ID))

ForEachID calls the provided callback with each id contained within the IndexEntry. Order of returned IDs is in Append-descending

func (IndexEntry) GetIDs

func (entry IndexEntry) GetIDs(ids []ID) []ID

GetIDs returns all IDs in the entry.

Provided array slice will be resized if necessary or a new one will be created if passed nil. Updated slice will be returned.

type IndexQuery

type IndexQuery struct {
	// contains filtered or unexported fields
}

IndexQuery represents a query running over established indices

An IndexQuery can be rerun by reinitializing the ctx; this typically happens when the query is Run.

func NewIndexQuery

func NewIndexQuery(rootType, rootFlavor StringHeapID,
	mods []StringHeapID, minModValues []uint16,
	league LeagueHeapID,
	maxDesired int) IndexQuery

NewIndexQuery returns an IndexQuery with no context

func (*IndexQuery) Run

func (q *IndexQuery) Run(db *bolt.DB) ([]ID, error)

Run initialises transaction context for a query and attempts to find desired items.

type Item

type Item struct {
	ID         ID
	GGGID      GGGID        // Allows mapping from simple ID to UUID
	Stash      GGGID        // Allows access to stash and corresponding metadata
	Name       StringHeapID // On StringHeap
	TypeLine   StringHeapID // On StringHeap
	Note       StringHeapID // On StringHeap
	RootType   StringHeapID // On StringHeap
	RootFlavor StringHeapID // On StringHeap
	League     LeagueHeapID // On LeagueHeap
	Corrupted  bool
	Identified bool
	Mods       []ItemMod
	When       Timestamp // When this stash update was processed
}

Item represents a compact record of an item.

func GetItemByID

func GetItemByID(id ID, league LeagueHeapID, tx *bolt.Tx) (Item, error)

GetItemByID returns the item represented by provided ID in a specific league

func GetItemByIDGlobal

func GetItemByIDGlobal(id ID, db *bolt.DB) (Item, error)

GetItemByIDGlobal attempts to resolve an item ID across every available league.

func StashItemsToCompact

func StashItemsToCompact(items []stash.Item, when Timestamp,
	db *bolt.DB) ([]Item, error)

StashItemsToCompact converts fat Item records to their compact form

This also ensures all strings present on that item will be available on the StringHeap

func (*Item) DecodeMsg

func (z *Item) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*Item) EncodeMsg

func (z *Item) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (Item) Inflate

func (item Item) Inflate(db *bolt.DB) stash.Item

Inflate returns an inflated equivalent item fit for human use

func (*Item) MarshalMsg

func (z *Item) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*Item) Msgsize

func (z *Item) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*Item) UnmarshalMsg

func (z *Item) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type ItemMod

type ItemMod struct {
	Mod   StringHeapID
	Value uint16
}

ItemMod represents a compact explicit or implicit modifier on an item

func StashItemModToCompact

func StashItemModToCompact(mod stash.ItemMod,
	modStringID StringHeapID) ItemMod

StashItemModToCompact compacts the given source ItemMod and StringHeapID for the mod's text into our internal format

func (*ItemMod) DecodeMsg

func (z *ItemMod) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (ItemMod) EncodeMsg

func (z ItemMod) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (ItemMod) Inflate

func (mod ItemMod) Inflate(db *bolt.DB) stash.ItemMod

Inflate returns an inflated equivalent item modifier for human use

func (ItemMod) MarshalMsg

func (z ItemMod) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (ItemMod) Msgsize

func (z ItemMod) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*ItemMod) UnmarshalMsg

func (z *ItemMod) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type ItemStoreQuery

type ItemStoreQuery struct {
	// contains filtered or unexported fields
}

ItemStoreQuery represents a query running naively over the ItemStore

This is a naive and inefficient query. This is mostly used for ensuring the correctness of more efficient query methods.

An ItemStoreQuery can be rerun by reinitializing the ctx; this typically happens when the query is Run.

func NewItemStoreQuery

func NewItemStoreQuery(rootType, rootFlavor StringHeapID,
	mods []StringHeapID, minModValues []uint16,
	league LeagueHeapID,
	maxDesired int) ItemStoreQuery

NewItemStoreQuery returns an ItemStoreQuery with no context

If len(mods) != len(minModValues), we panic; so don't give us garbage

func (*ItemStoreQuery) Run

func (q *ItemStoreQuery) Run(db *bolt.DB) ([]ID, error)

Run initialises transaction context for a query and attempts to find desired items.

type ItemUpdateStats

type ItemUpdateStats struct {
	Added   int // Number of items added
	Removed int // Number of items removed
	Kept    int // Number of items kept
}

ItemUpdateStats is a subfield of StashUpdateStats broken out for easier instantiation.

type LeagueHeapID

type LeagueHeapID uint16

LeagueHeapID maps to a stored string identifier specific to league

This is basically StringHeapID but specialised for leagues

func GetLeagues

func GetLeagues(leagues []string, db *bolt.DB) ([]LeagueHeapID, error)

GetLeagues maps all indice league values to their corresponding LeagueHeapID

func LeagueHeapIDFromBytes

func LeagueHeapIDFromBytes(bytes []byte) LeagueHeapID

LeagueHeapIDFromBytes generats the corresponding heap id from the provided bytes

func LeagueHeapIDFromSequence

func LeagueHeapIDFromSequence(seq uint64) LeagueHeapID

LeagueHeapIDFromSequence transforms a 64 bit bucket sequence number into a LeagueHeapID

func SetLeagues

func SetLeagues(leagues []string, db *bolt.DB) ([]LeagueHeapID, error)

SetLeagues fills in any missing league heap index values and maps all indice league values to their corresponding LeagueHeapID

func (*LeagueHeapID) DecodeMsg

func (z *LeagueHeapID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (LeagueHeapID) EncodeMsg

func (z LeagueHeapID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (LeagueHeapID) Inflate

func (id LeagueHeapID) Inflate(db *bolt.DB) string

Inflate returns the string represented by the given LeagueHeapID

func (LeagueHeapID) MarshalMsg

func (z LeagueHeapID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (LeagueHeapID) Msgsize

func (z LeagueHeapID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (LeagueHeapID) ToBytes

func (id LeagueHeapID) ToBytes() []byte

ToBytes returns the byte-wise represenation of a LeagueHeapID

func (*LeagueHeapID) UnmarshalMsg

func (z *LeagueHeapID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type Stash

type Stash struct {
	ID          GGGID        // Reference value for this Stash
	AccountName string       // Account-wide name, we need nothing else to PM
	Items       []GGGID      // GGGIDs for all items stored in that Stash
	League      LeagueHeapID // LeagueHeapID as stashes are single-league
}

Stash represents a compact record of a stash.

func StashStashToCompact

func StashStashToCompact(stashes []stash.Stash, when time.Time,
	db *bolt.DB) ([]Stash, [][]Item, error)

StashStashToCompact converts fat Item records to their compact form while also stripping items out in their compact form.

func (*Stash) DecodeMsg

func (z *Stash) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (Stash) Diff

func (s Stash) Diff(old Stash) (add, remove []GGGID)

Diff takes an older version of a Stash and determines which items, in terms of GGGID, need to be added and which need to be removed.

func (*Stash) EncodeMsg

func (z *Stash) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*Stash) MarshalMsg

func (z *Stash) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*Stash) Msgsize

func (z *Stash) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*Stash) UnmarshalMsg

func (z *Stash) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type StashUpdateStats

type StashUpdateStats struct {
	Added   int // Number of stashes added
	Updated int // Number of stashes updated
	Intact  int // Number of stashes without any changes made
	// Items give item-wise stats
	Items ItemUpdateStats
}

StashUpdateStats represents the actual work done in an operation that is applied to Stashes.

All values are expected to be >= 0

func AddStashes

func AddStashes(stashes []Stash, items [][]Item,
	db *bolt.DB) (*StashUpdateStats, error)

AddStashes adds tbe given items to their correct paths in the database

Provided stashes CAN differ in their league.

func (*StashUpdateStats) Compare

func (s *StashUpdateStats) Compare(other *StashUpdateStats) error

Compare considers the receiver as the expected StashUpdateStats while the other is tested to see if it matches. Any difference between expected and other is reported as an error.

func (StashUpdateStats) String

func (s StashUpdateStats) String() string

type StringHeapID

type StringHeapID uint32

StringHeapID maps to a stored string identifier.

This creates a layer of indirection when rebuilding items but saves on space for ids

func GetStrings

func GetStrings(indices []string, db *bolt.DB) ([]StringHeapID, error)

GetStrings maps all indice string values to their corresponding StringHeapID

func SetStrings

func SetStrings(indices []string, db *bolt.DB) ([]StringHeapID, error)

SetStrings fills in any missing string heap index values and maps all indice string values to their corresponding StringHeapID

func StringHeapIDFromBytes

func StringHeapIDFromBytes(bytes []byte) StringHeapID

StringHeapIDFromBytes generats the corresponding heap id from the provided bytes

func (*StringHeapID) DecodeMsg

func (z *StringHeapID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (StringHeapID) EncodeMsg

func (z StringHeapID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (StringHeapID) Inflate

func (id StringHeapID) Inflate(db *bolt.DB) string

Inflate returns the string represented by the given StringHeapID

func (StringHeapID) MarshalMsg

func (z StringHeapID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (StringHeapID) Msgsize

func (z StringHeapID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (StringHeapID) ToBytes

func (id StringHeapID) ToBytes() []byte

ToBytes returns the byte-wise represenation of a StringHeapID

func (*StringHeapID) UnmarshalMsg

func (z *StringHeapID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type Timestamp

type Timestamp [TimestampSize]byte

Timestamp is a compact represenation of a unix timestamp

func NewTimestamp

func NewTimestamp() Timestamp

NewTimestamp returns a Timestamp at the current time

func TimeToTimestamp

func TimeToTimestamp(when time.Time) Timestamp

TimeToTimestamp returns a Timestamp representing the passed time.Time

func (*Timestamp) DecodeMsg

func (z *Timestamp) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*Timestamp) EncodeMsg

func (z *Timestamp) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*Timestamp) MarshalMsg

func (z *Timestamp) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*Timestamp) Msgsize

func (z *Timestamp) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (Timestamp) ToTime

func (ts Timestamp) ToTime() time.Time

ToTime converts a compact Timestamp to a time.Time

func (Timestamp) TruncateToIndexBucket

func (ts Timestamp) TruncateToIndexBucket() []byte

TruncateToIndexBucket returns its reduced accuracy form such that it can be used to bucket Timestamps into discrete index buckets.

func (*Timestamp) UnmarshalMsg

func (z *Timestamp) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL