README
¶
go-dsqueue
Buffered FIFO interface to the datastore
The dsqueue package provides a buffered FIFO queue backed by a Batching Datastore. Queued items are persisted in the datastore when the input buffer is full, after some amount of idle time, and when the queue is shuddown.
Documentation
https://pkg.go.dev/github.com/ipfs/go-dsqueue
Contributing
Contributions are welcome! This repository is part of the IPFS project and therefore governed by our contributing guidelines.
License
Documentation
¶
Overview ¶
Package dsqueue provides a buffered FIFO interface to the datastore for storing and retrieving items. Queued items are persisted across restarts.
Index ¶
Constants ¶
const ( DefaultBufferSize = 16 * 1024 DefaultDedupCacheSize = 2 * 1024 DefaultIdleWriteTime = time.Minute DefaultCloseTimeout = 10 * time.Second )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DSQueue ¶
type DSQueue struct {
// contains filtered or unexported fields
}
DSQueue provides a FIFO interface to the datastore for storing items.
Items in the process of being provided when a crash or shutdown occurs may be in the queue when the node is brought back online depending on whether they were fully written to the underlying datastore.
Input to the queue is buffered in memory. The contents of the buffer are written to the datastore when the input buffer is full (see WithBufferSize), or when the queue has been idle for some time (see WithIdleWriteTime) since the previous batch write or dequeue. Items to dequeue are read, in order, from the input buffer if there are none in the datastore. Otherwise they are read from the datastore.
If queued items are read from the input buffer before it reaches its limit, then queued items can remain in memory. When the queue is closed, any remaining items in memory are written to the datastore.
func (*DSQueue) Clear ¶
Clear clears all queued records from memory and the datastore. Returns the number of items removed from the queue.
type Option ¶
type Option func(*config)
Option is a function that sets a value in a config.
func WithBufferSize ¶
WithBufferSize sets the limit on number of items kept in input buffer memory, at which they are all written to the datastore. A value of 0 means the buffer size is unlimited, and items are only written to the datastore when the queue has been idle more then the idle write time or when the queue is closed.
func WithCloseTimeout ¶
WithCloseTimeout sets the duration that Close waits to finish writing items to the datastore. A value of 0 means wait until finished with no timeout.
func WithDedupCacheSize ¶
WithDedupCacheSize sets the size of the LRU cache used to deduplicate items in the queue. A value of 0 disables the dedup cache.
func WithIdleWriteTime ¶
WithIdleWriteTime sets the amout of time that the queue must be idle (no input or output) before all buffered input items are written to the datastore. A value of zero means that buffered input items are not automatically flushed to the datastore. A non-zero value must be greater than one second.
Source Files