db

package
Version: v0.0.0-...-89d8475 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 26, 2017 License: Apache-2.0, MIT Imports: 9 Imported by: 0

Documentation

Index

Constants

View Source 
const DBLocation string = "poe.db"

DBLocation is the on-disk file containing our database

View Source 
const GGGIDSize = 10

GGGIDSize is the size in bytes a derived ID can be

View Source 
const IDSize = 8

IDSize is the size in bytes an internal identifier can be

View Source 
const ItemModAverageScaleFactor = 10

ItemModAverageScaleFactor is the multiplier applied to the average of multi-mod items. This allows a fixed precision of averages.

Any more than 10 and we could risk overflowing our uint16

View Source 
const LeagueHeapIDSize = 2

LeagueHeapIDSize is the number of bytes taken up by a LeagueHeapID when converted to bytes

View Source 
const LookupItemsMultiModStrideLength = 32

LookupItemsMultiModStrideLength determines how many items is included in a stride of LookupItemsMultiMod.

Longer strides mean fewer intersections but more potentially useless item mods checked.

View Source 
const ModIndexKeySuffixLength = TimestampSize

ModIndexKeySuffixLength allows us to fetch variable numbers of pre-pended values given their length.

View Source 
const StringHeapIDSize = 4

StringHeapIDSize is the number of bytes a StringHeapID takes up when converted to bytes.

View Source 
const TimestampBucketAccuracyReduction = 7 // 7 representing ~2.1 minute buckets

TimestampBucketAccuracyReduction is the number of rightward shifts applied to a Timestamp to discretely bucket it for Indexing.

View Source 
const TimestampSize = 8 - 4

TimestampSize is the number of bytes used by Timestamp

This is sized to minimize waste while keeping accuracy

We discard the first 4 bytes as they're useless for our purposes.

Variables

This section is empty.

Functions

func AddItems 

func AddItems(items []Item, db *bolt.DB) (int, error)

AddItems adds tbe given items to their correct paths in the database

Provided items CAN differ in their league.

This is simply a wrapper for addItems that also establishes the transaction as well as include the timestamp

func Boot 

func Boot(path string) (*bolt.DB, error)

Boot gets the database from disk and performs necessary setup

If path is empty, it uses the default DBLocation

func DeindexItems 

func DeindexItems(items []Item, tx *bolt.Tx) error

DeindexItems removes tbe given items from their correct indices

If an index entry cannot be removed, we return an error. This ensures all existing index entries must be alive

func GetTranslations 

func GetTranslations(items []Item, db *bolt.DB) error

GetTranslations associates each provided item with an interal ID if it has not already been assigned one.

This modifies the provided items if they are assigned an ID

func IndexEntryCount 

func IndexEntryCount(db *bolt.DB) (int, error)

IndexEntryCount returns the number of index entries across all leagues

func IndexItems 

func IndexItems(items []Item, tx *bolt.Tx) (int, error)

IndexItems adds tbe given items to their correct indices for efficient lookup. Returns number of index entries added.

Provided items CAN differ in their league.

func InflateLeague 

func InflateLeague(id LeagueHeapID, db *bolt.DB) string

InflateLeague returns the string represenation of the given LeagueHeapID

If you are providing a valid LeagueHeapID, this should never fail.

func InflateString 

func InflateString(id StringHeapID, db *bolt.DB) string

InflateString returns the string represenation of the given StringHeapID

If you are providing a valid StringHeapID, this should never fail.

func ItemStoreCount 

func ItemStoreCount(db *bolt.DB) (int, error)

ItemStoreCount returns the number of items across all leagues

func ListLeagues 

func ListLeagues(db *bolt.DB) ([]string, error)

ListLeagues returns a list of all stored leagues

func RecordChangeIDWasEntered 

func RecordChangeIDWasEntered(changeID string, db *bolt.DB) error

RecordChangeIDWasEntered adds a given changeID to the database with its value being the json time it was entered

func RemoveItems 

func RemoveItems(ids []ID, league LeagueHeapID, db *bolt.DB) error

RemoveItems removes the provided IDs from the database

All provided IDs must be under the same League.

For higher performance, callers can sort the IDs to allow for more sequential access behaviors when handling the heap

func UpdateWasEntered 

func UpdateWasEntered(changeID string, db *bolt.DB) (bool, error)

UpdateWasEntered determines if a given update has been entered into the database

Types

type GGGID 

type GGGID [GGGIDSize]byte

GGGID is an Identifier derived from per-item/stash tab UID

A PID is 80 bits = 10 bytes, this allows 2^40 items to be represented taking into birthdays and represents significant savings relative to the GGG api provided id

func GGGIDFromUID 

func GGGIDFromUID(uid string) GGGID

GGGIDFromUID generates an ID for internal use from a UID string

func (*GGGID) DecodeMsg 

func (z *GGGID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*GGGID) EncodeMsg 

func (z *GGGID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*GGGID) MarshalMsg 

func (z *GGGID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*GGGID) Msgsize 

func (z *GGGID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*GGGID) UnmarshalMsg 

func (z *GGGID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type ID 

type ID [IDSize]byte

ID is an Identifier calculated internally

This is effectively just a 64 bit uint

func GetGGGIDTranslations 

func GetGGGIDTranslations(gggs []GGGID, league LeagueHeapID,
	tx *bolt.Tx) ([]ID, error)

GetGGGIDTranslations associates each provided item with an interal ID if it has not already been assigned one.

All IDs are in the context of the provided league, hence this is typically used for managing individual stash updates and that's why we take a transaction.

This returns the newly translated ID positionally mapping to the provided GGGIDs.

func IDFromSequence 

func IDFromSequence(seq uint64) ID

IDFromSequence converts a sequence number into an identifier

func (*ID) DecodeMsg 

func (z *ID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*ID) EncodeMsg 

func (z *ID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*ID) MarshalMsg 

func (z *ID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*ID) Msgsize 

func (z *ID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*ID) UnmarshalMsg 

func (z *ID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type IDMapPool 

type IDMapPool struct {
	// contains filtered or unexported fields
}

IDMapPool represents a thread-safe pool of reusable sets for IndexQuery

func NewIDMapPool 

func NewIDMapPool(maxPerSize int) IDMapPool

NewIDMapPool creates a properly initialized IDMapPool

func (*IDMapPool) Borrow 

func (p *IDMapPool) Borrow(minLen int) map[ID]struct{}

Borrow returns a pre-allocated buffer in the pool or a fresh buffer if none are available

func (*IDMapPool) Give 

func (p *IDMapPool) Give(buf map[ID]struct{})

Give returns a buffer to the set

If we have more buffers than we are configured to handle, further provided buffers are tossed out.

type IndexEntry 

type IndexEntry []byte

IndexEntry represents bytes interpreted as an entry within the index

Whenever possible, we avoid allocations.

func IndexEntryAppend 

func IndexEntryAppend(entry IndexEntry, id ID) IndexEntry

IndexEntryAppend adds another ID to the entry

If an id is already present in the id, we end up with a duplicate. Such is life.

func IndexEntryRemove 

func IndexEntryRemove(entry IndexEntry, id ID) IndexEntry

IndexEntryRemove removes a given ID from the entry

If the ID is the last of the entry, the backing slice is set to nil. In that case, its the callers responsibility to ensure they Unwrap a valid byte slice.

func (IndexEntry) ForEachID 

func (entry IndexEntry) ForEachID(cb func(id ID))

ForEachID calls the provided callback with each id contained within the IndexEntry. Order of returned IDs is in Append-descending

func (IndexEntry) GetIDs 

func (entry IndexEntry) GetIDs(ids []ID) []ID

GetIDs returns all IDs in the entry.

Provided array slice will be resized if necessary or a new one will be created if passed nil. Updated slice will be returned.

type IndexQuery 

type IndexQuery struct {
	// contains filtered or unexported fields
}

IndexQuery represents a query running over established indices

An IndexQuery can be rerun by reinitializing the ctx; this typically happens when the query is Run.

func NewIndexQuery 

func NewIndexQuery(rootType, rootFlavor StringHeapID,
	mods []StringHeapID, minModValues []uint16,
	league LeagueHeapID,
	maxDesired int) IndexQuery

NewIndexQuery returns an IndexQuery with no context

func (*IndexQuery) Run 

func (q *IndexQuery) Run(db *bolt.DB) ([]ID, error)

Run initialises transaction context for a query and attempts to find desired items.

type Item 

type Item struct {
	ID         ID
	GGGID      GGGID        // Allows mapping from simple ID to UUID
	Stash      GGGID        // Allows access to stash and corresponding metadata
	Name       StringHeapID // On StringHeap
	TypeLine   StringHeapID // On StringHeap
	Note       StringHeapID // On StringHeap
	RootType   StringHeapID // On StringHeap
	RootFlavor StringHeapID // On StringHeap
	League     LeagueHeapID // On LeagueHeap
	Corrupted  bool
	Identified bool
	Mods       []ItemMod
	When       Timestamp // When this stash update was processed
}

Item represents a compact record of an item.

func GetItemByID 

func GetItemByID(id ID, league LeagueHeapID, tx *bolt.Tx) (Item, error)

GetItemByID returns the item represented by provided ID in a specific league

func GetItemByIDGlobal 

func GetItemByIDGlobal(id ID, db *bolt.DB) (Item, error)

GetItemByIDGlobal attempts to resolve an item ID across every available league.

func StashItemsToCompact 

func StashItemsToCompact(items []stash.Item, when Timestamp,
	db *bolt.DB) ([]Item, error)

StashItemsToCompact converts fat Item records to their compact form

This also ensures all strings present on that item will be available on the StringHeap

func (*Item) DecodeMsg 

func (z *Item) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*Item) EncodeMsg 

func (z *Item) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (Item) Inflate 

func (item Item) Inflate(db *bolt.DB) stash.Item

Inflate returns an inflated equivalent item fit for human use

func (*Item) MarshalMsg 

func (z *Item) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*Item) Msgsize 

func (z *Item) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*Item) UnmarshalMsg 

func (z *Item) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type ItemMod 

type ItemMod struct {
	Mod   StringHeapID
	Value uint16
}

ItemMod represents a compact explicit or implicit modifier on an item

func StashItemModToCompact 

func StashItemModToCompact(mod stash.ItemMod,
	modStringID StringHeapID) ItemMod

StashItemModToCompact compacts the given source ItemMod and StringHeapID for the mod's text into our internal format

func (*ItemMod) DecodeMsg 

func (z *ItemMod) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (ItemMod) EncodeMsg 

func (z ItemMod) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (ItemMod) Inflate 

func (mod ItemMod) Inflate(db *bolt.DB) stash.ItemMod

Inflate returns an inflated equivalent item modifier for human use

func (ItemMod) MarshalMsg 

func (z ItemMod) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (ItemMod) Msgsize 

func (z ItemMod) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*ItemMod) UnmarshalMsg 

func (z *ItemMod) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type ItemStoreQuery 

type ItemStoreQuery struct {
	// contains filtered or unexported fields
}

ItemStoreQuery represents a query running naively over the ItemStore

This is a naive and inefficient query. This is mostly used for ensuring the correctness of more efficient query methods.

An ItemStoreQuery can be rerun by reinitializing the ctx; this typically happens when the query is Run.

func NewItemStoreQuery 

func NewItemStoreQuery(rootType, rootFlavor StringHeapID,
	mods []StringHeapID, minModValues []uint16,
	league LeagueHeapID,
	maxDesired int) ItemStoreQuery

NewItemStoreQuery returns an ItemStoreQuery with no context

If len(mods) != len(minModValues), we panic; so don't give us garbage

func (*ItemStoreQuery) Run 

func (q *ItemStoreQuery) Run(db *bolt.DB) ([]ID, error)

Run initialises transaction context for a query and attempts to find desired items.

type ItemUpdateStats 

type ItemUpdateStats struct {
	Added   int // Number of items added
	Removed int // Number of items removed
	Kept    int // Number of items kept
}

ItemUpdateStats is a subfield of StashUpdateStats broken out for easier instantiation.

type LeagueHeapID 

type LeagueHeapID uint16

LeagueHeapID maps to a stored string identifier specific to league

This is basically StringHeapID but specialised for leagues

func GetLeagues 

func GetLeagues(leagues []string, db *bolt.DB) ([]LeagueHeapID, error)

GetLeagues maps all indice league values to their corresponding LeagueHeapID

func LeagueHeapIDFromBytes 

func LeagueHeapIDFromBytes(bytes []byte) LeagueHeapID

LeagueHeapIDFromBytes generats the corresponding heap id from the provided bytes

func LeagueHeapIDFromSequence 

func LeagueHeapIDFromSequence(seq uint64) LeagueHeapID

LeagueHeapIDFromSequence transforms a 64 bit bucket sequence number into a LeagueHeapID

func SetLeagues 

func SetLeagues(leagues []string, db *bolt.DB) ([]LeagueHeapID, error)

SetLeagues fills in any missing league heap index values and maps all indice league values to their corresponding LeagueHeapID

func (*LeagueHeapID) DecodeMsg 

func (z *LeagueHeapID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (LeagueHeapID) EncodeMsg 

func (z LeagueHeapID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (LeagueHeapID) Inflate 

func (id LeagueHeapID) Inflate(db *bolt.DB) string

Inflate returns the string represented by the given LeagueHeapID

func (LeagueHeapID) MarshalMsg 

func (z LeagueHeapID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (LeagueHeapID) Msgsize 

func (z LeagueHeapID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (LeagueHeapID) ToBytes 

func (id LeagueHeapID) ToBytes() []byte

ToBytes returns the byte-wise represenation of a LeagueHeapID

func (*LeagueHeapID) UnmarshalMsg 

func (z *LeagueHeapID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type Stash 

type Stash struct {
	ID          GGGID        // Reference value for this Stash
	AccountName string       // Account-wide name, we need nothing else to PM
	Items       []GGGID      // GGGIDs for all items stored in that Stash
	League      LeagueHeapID // LeagueHeapID as stashes are single-league
}

Stash represents a compact record of a stash.

func StashStashToCompact 

func StashStashToCompact(stashes []stash.Stash, when time.Time,
	db *bolt.DB) ([]Stash, [][]Item, error)

StashStashToCompact converts fat Item records to their compact form while also stripping items out in their compact form.

func (*Stash) DecodeMsg 

func (z *Stash) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (Stash) Diff 

func (s Stash) Diff(old Stash) (add, remove []GGGID)

Diff takes an older version of a Stash and determines which items, in terms of GGGID, need to be added and which need to be removed.

func (*Stash) EncodeMsg 

func (z *Stash) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*Stash) MarshalMsg 

func (z *Stash) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*Stash) Msgsize 

func (z *Stash) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*Stash) UnmarshalMsg 

func (z *Stash) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type StashUpdateStats 

type StashUpdateStats struct {
	Added   int // Number of stashes added
	Updated int // Number of stashes updated
	Intact  int // Number of stashes without any changes made
	// Items give item-wise stats
	Items ItemUpdateStats
}

StashUpdateStats represents the actual work done in an operation that is applied to Stashes.

All values are expected to be >= 0

func AddStashes 

func AddStashes(stashes []Stash, items [][]Item,
	db *bolt.DB) (*StashUpdateStats, error)

AddStashes adds tbe given items to their correct paths in the database

Provided stashes CAN differ in their league.

func (*StashUpdateStats) Compare 

func (s *StashUpdateStats) Compare(other *StashUpdateStats) error

Compare considers the receiver as the expected StashUpdateStats while the other is tested to see if it matches. Any difference between expected and other is reported as an error.

func (StashUpdateStats) String 

func (s StashUpdateStats) String() string

type StringHeapID 

type StringHeapID uint32

StringHeapID maps to a stored string identifier.

This creates a layer of indirection when rebuilding items but saves on space for ids

func GetStrings 

func GetStrings(indices []string, db *bolt.DB) ([]StringHeapID, error)

GetStrings maps all indice string values to their corresponding StringHeapID

func SetStrings 

func SetStrings(indices []string, db *bolt.DB) ([]StringHeapID, error)

SetStrings fills in any missing string heap index values and maps all indice string values to their corresponding StringHeapID

func StringHeapIDFromBytes 

func StringHeapIDFromBytes(bytes []byte) StringHeapID

StringHeapIDFromBytes generats the corresponding heap id from the provided bytes

func (*StringHeapID) DecodeMsg 

func (z *StringHeapID) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (StringHeapID) EncodeMsg 

func (z StringHeapID) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (StringHeapID) Inflate 

func (id StringHeapID) Inflate(db *bolt.DB) string

Inflate returns the string represented by the given StringHeapID

func (StringHeapID) MarshalMsg 

func (z StringHeapID) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (StringHeapID) Msgsize 

func (z StringHeapID) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (StringHeapID) ToBytes 

func (id StringHeapID) ToBytes() []byte

ToBytes returns the byte-wise represenation of a StringHeapID

func (*StringHeapID) UnmarshalMsg 

func (z *StringHeapID) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type Timestamp 

type Timestamp [TimestampSize]byte

Timestamp is a compact represenation of a unix timestamp

func NewTimestamp 

func NewTimestamp() Timestamp

NewTimestamp returns a Timestamp at the current time

func TimeToTimestamp 

func TimeToTimestamp(when time.Time) Timestamp

TimeToTimestamp returns a Timestamp representing the passed time.Time

func (*Timestamp) DecodeMsg 

func (z *Timestamp) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*Timestamp) EncodeMsg 

func (z *Timestamp) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*Timestamp) MarshalMsg 

func (z *Timestamp) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*Timestamp) Msgsize 

func (z *Timestamp) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (Timestamp) ToTime 

func (ts Timestamp) ToTime() time.Time

ToTime converts a compact Timestamp to a time.Time

func (Timestamp) TruncateToIndexBucket 

func (ts Timestamp) TruncateToIndexBucket() []byte

TruncateToIndexBucket returns its reduced accuracy form such that it can be used to bucket Timestamps into discrete index buckets.

func (*Timestamp) UnmarshalMsg 

func (z *Timestamp) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

Source Files

View all

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.