Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type RedisConf ¶
type RedisConf struct {
Address string `json:"address"`
Database int `json:"database"`
QueueKey string `json:"queueKey"`
AppType string `json:"appType"`
Version string `json:"version"`
TZShift int `json:"tzShift"`
}
RedisConf is a structure containing information about Redis database containing logs to be processed.
type RedisQueue ¶
type RedisQueue struct {
// contains filtered or unexported fields
}
RedisQueue provides access to Redis database containing KonText log records.
func OpenRedisQueue ¶
func OpenRedisQueue(address string, database int, queueKey string, tzShift int) (*RedisQueue, error)
OpenRedisQueue creates a client for Redis
func (*RedisQueue) GetItems ¶
func (rc *RedisQueue) GetItems() []conversion.InputRecord
GetItems loads log data from a Redis queue (list type). The data is expected to be in JSON format.
Please note that invalid records are taken from queue too and then thrown away (with logged message containing the original item source).
func (*RedisQueue) GetRescuedChunksIterator ¶
func (rc *RedisQueue) GetRescuedChunksIterator() *RedisRescuedChunkIterator
GetRescuedChunksIterator returns an iterator object for rescued raw bulk insert records.
func (*RedisQueue) RescueFailedChunks ¶
func (rc *RedisQueue) RescueFailedChunks(data [][]byte) error
RescueFailedChunks puts records to the end of the Redis queue. This is mostly for handling ElasticSearch import errors.
type RedisRescuedChunkIterator ¶
type RedisRescuedChunkIterator struct {
// contains filtered or unexported fields
}
RedisRescuedChunkIterator provides stateful access to individual bulk insert chunks ([meta line, data line]+ "new line")
func (*RedisRescuedChunkIterator) GetNextChunk ¶
func (rrci *RedisRescuedChunkIterator) GetNextChunk() [][]byte
GetNextChunk provide next chunk of bulk insert data. If nothing is found then a slice of size 0 is returned.
func (*RedisRescuedChunkIterator) RemoveVisitedItems ¶
func (rrci *RedisRescuedChunkIterator) RemoveVisitedItems() (int, error)
RemoveVisitedItems removes from Redis all the items we iterated through so far
Source Files