ingest

Documentation

Index

• Constants
• Variables
• func HandleConnections(ln net.Listener, h ConnectionHandler, log Log, segmentFlushAge time.Duration, ...) error
• func HandleFastWriter(conn net.Conn, w *Writer, idGen IDGenerator, connectedClients prometheus.Gauge) (err error)
• type API
  • func NewAPI(peer ClusterPeer, log Log, pendingSegmentTimeout time.Duration, ...) *API
  • func (a *API) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (a *API) Stop()
• type ClusterPeer
• type ConnectionHandler
• type IDGenerator
• type Log
  • func NewFileLog(filesys fs.Filesystem, root string) (Log, error)
• type LogStats
• type ReadSegment
• type WriteSegment
• type Writer
  • func NewWriter(log Log, d time.Duration, sz int, bytes, records, syncs prometheus.Counter, ...) (*Writer, error)
  • func (w *Writer) Stop()
  • func (w *Writer) Sync() error
  • func (w *Writer) Write(p []byte) (int, error)

Constants

const (
	APIPathNext         = "/next"
	APIPathRead         = "/read"
	APIPathCommit       = "/commit"
	APIPathFailed       = "/failed"
	APIPathSegmentState = "/_segmentstate"
	APIPathClusterState = "/_clusterstate"
)

These are the ingest API URL paths.

Variables

var ErrNoSegmentsAvailable = errors.New("no segments available")

ErrNoSegmentsAvailable is returned by IngestLog Oldest, when no segments are available for reading.

Functions

func HandleConnections

func HandleConnections(
	ln net.Listener,
	h ConnectionHandler,
	log Log,
	segmentFlushAge time.Duration,
	segmentFlushSize int,
	connectedClients prometheus.Gauge,
	bytes, records, syncs prometheus.Counter,
	segmentAge, segmentSize prometheus.Histogram,
) error

HandleConnections passes each connection from the listener to the connection handler. Terminate the function by closing the listener.

func HandleFastWriter

func HandleFastWriter(conn net.Conn, w *Writer, idGen IDGenerator, connectedClients prometheus.Gauge) (err error)

HandleFastWriter is a ConnectionHandler that writes records to the IngestLog.

Types

type API

type API struct {
	// contains filtered or unexported fields
}

API serves the ingest API.

func NewAPI

func NewAPI(
	peer ClusterPeer,
	log Log,
	pendingSegmentTimeout time.Duration,
	failedSegments, committedSegments, committedBytes prometheus.Counter,
	duration *prometheus.HistogramVec,
) *API

NewAPI returns a usable ingest API.

func (*API) ServeHTTP

func (a *API) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*API) Stop

func (a *API) Stop()

Stop terminates the API.

type ClusterPeer

type ClusterPeer interface {
	State() map[string]interface{}
}

ClusterPeer models cluster.Peer.

type ConnectionHandler

type ConnectionHandler func(conn net.Conn, w *Writer, idGen IDGenerator, connectedClients prometheus.Gauge) error

ConnectionHandler forwards records from the net.Conn to the IngestLog.

type IDGenerator

type IDGenerator func() string

IDGenerator should return unique record identifiers, i.e. ULIDs.

type Log

type Log interface {
	Create() (WriteSegment, error)
	Oldest() (ReadSegment, error)
	Stats() (LogStats, error)
	Close() error
}

Log is an abstraction for segments on an ingest node. A new active segment may be created and written to. The oldest flushed segment may be selected and read from.

func NewFileLog

func NewFileLog(filesys fs.Filesystem, root string) (Log, error)

NewFileLog returns a Log implemented via the filesystem. All filesystem ops will be rooted at path root.

type LogStats

type LogStats struct {
	ActiveSegments  int64
	ActiveBytes     int64
	FlushedSegments int64
	FlushedBytes    int64
	PendingSegments int64
	PendingBytes    int64
}

LogStats describe the current state of the ingest log.

type ReadSegment

type ReadSegment interface {
	io.Reader
	Commit() error
	Failed() error
	Size() int64
}

ReadSegment is a segment that can be read from. Once read, it may be committed and thus deleted. Or it may be failed, and made available for selection again.

type WriteSegment

type WriteSegment interface {
	io.Writer
	Sync() error
	Close() error
	Delete() error
}

WriteSegment is a segment that can be written to. It may be optionally synced to disk manually. When writing is complete, it may be closed and flushed. If it would be closed with size 0, it may be deleted instead.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer implements io.Writer on top of a Log.

func NewWriter

func NewWriter(
	log Log,
	d time.Duration,
	sz int,
	bytes, records, syncs prometheus.Counter,
	age, size prometheus.Histogram,
) (*Writer, error)

NewWriter converts a Log to an io.Writer. Active segments are rotated once sz bytes are written, or every d if the segment is nonempty.

func (*Writer) Stop

func (w *Writer) Stop()

Stop terminates the Writer. No further writes are allowed.

func (*Writer) Sync

func (w *Writer) Sync() error

Sync the current segment to disk.

func (*Writer) Write

func (w *Writer) Write(p []byte) (int, error)

Write implements io.Writer.