butteredscones

package module
v0.0.0-...-eb37774 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 17, 2015 License: MIT Imports: 18 Imported by: 0

README

butteredscones Build Status

butteredscones watches log files on disk and forwards them to logstash via the lumberjack protocol. It is similar to logstash-forwarder.

I'm a lumberjack and I'm OK

I sleep all night and I work all day.

[...]

On Wednesdays I go shoppin'

And have buttered scones for tea.

Configuration

Like logstash-forwarder, butteredscones is configured via a JSON file.

{
  "state": "/var/lib/butteredscones/state.db",

  "network": {
    "servers": [
      {
        "addr": "192.168.0.1:5043",
        "name": "logstash.internal.example.com",
      }
    ],
    "certificate":  "/etc/butteredscones/forwarder.crt",
    "key":          "/etc/butteredscones/forwarder.key",
    "ca":           "/etc/butteredscones/ca.crt",
    "timeout":      15
  },

  "statistics": {
    "addr": "127.0.0.1:8088"
  },

  "files": [
    {
      "paths":  ["/var/log/messages", "/var/log/*.log"],
      "fields": {"type": "syslog"}
    }
  ]
}

state is where butteredscones keeps information about how far it has read into each file. The directory where it lives must be writable by the user that runs the butteredscones process.

network/servers can include one or more servers. If multiple servers are present, butteredscones will send to all servers concurrently. Specifying an name for a server is optional. If specified, the addr will be used to connect, but the name will be used to verify the certificate. This allows butteredscones to connect properly even if DNS is broken.

The SSL certificate presented by the remote logstash server must be signed by the specified CA, if the "ca" option is specified. Otherwise, butteredscones will not communicate with the remote server.

If given, statistics/addr specifies a socket address where an HTTP server will listen. Statistics about what butteredscones is doing will be written in JSON format. Use these statistics to debug problems or write automated monitoring tools. For example: curl -si http://localhost:8088

files supports glob patterns. butteredscones will periodically check for new files that match the glob pattern and tail them.

Currently, butteredscones does not support log files that are truncated or renamed. This is not a use case the original developers had. However, if it interests you, pull requests are welcomed.

Development & Packaging

To build the static binary, butteredscones:

script/build

To run the tests:

script/test

To package butteredscones into a debian package:

GOOS=linux GOARCH=amd64 VERSION=0.0.1 script/deb

Future Work

  • Support input from standard in
  • Support files which are truncated in place

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BoltSnapshotter 

type BoltSnapshotter struct {
	DB *bolt.DB
}

func (*BoltSnapshotter) HighWaterMark 

func (s *BoltSnapshotter) HighWaterMark(filePath string) (*HighWaterMark, error)

func (*BoltSnapshotter) SetHighWaterMarks 

func (s *BoltSnapshotter) SetHighWaterMarks(marks []*HighWaterMark) error

type ClientStatistics 

type ClientStatistics struct {
	Status string `json:"status"`

	// The number of lines sent successfully to the client
	LinesSent int `json:"lines_sent"`

	// The last time lines were successfully sent to this client
	LastSendTime time.Time `json:"last_send_time"`

	// The number of lines in the last chunk successfully sent to this client
	LastChunkSize int `json:"last_chunk_size"`
}

type Configuration 

type Configuration struct {
	State      string                  `json:"state"`
	Network    NetworkConfiguration    `json:"network"`
	Statistics StatisticsConfiguration `json:"statistics"`
	Files      []FileConfiguration     `json:"files"`
	MaxLength  int                     `json:"max_length"`
}

func LoadConfiguration 

func LoadConfiguration(configFile string) (*Configuration, error)

func (*Configuration) BuildTLSConfig 

func (c *Configuration) BuildTLSConfig() (*tls.Config, error)

type ExponentialBackoff 

type ExponentialBackoff struct {
	Minimum time.Duration
	Maximum time.Duration
	// contains filtered or unexported fields
}

func (*ExponentialBackoff) Current 

func (b *ExponentialBackoff) Current() time.Duration

func (*ExponentialBackoff) Next 

func (b *ExponentialBackoff) Next() time.Duration

func (*ExponentialBackoff) Reset 

func (b *ExponentialBackoff) Reset()

type FileConfiguration 

type FileConfiguration struct {
	Paths  []string          `json:"paths"`
	Fields map[string]string `json:"fields"`
}

type FileData 

type FileData struct {
	client.Data
	*HighWaterMark
}

type FileReader 

type FileReader struct {
	C         chan []*FileData
	ChunkSize int
	MaxLength int
	// contains filtered or unexported fields
}

func NewFileReader 

func NewFileReader(file *os.File, fields map[string]string, chunkSize, maxLength int) (*FileReader, error)

func (*FileReader) FilePath 

func (h *FileReader) FilePath() string

type FileReaderPool 

type FileReaderPool struct {
	// contains filtered or unexported fields
}

func NewFileReaderPool 

func NewFileReaderPool() *FileReaderPool

func (*FileReaderPool) Add 

func (p *FileReaderPool) Add(reader *FileReader)

func (*FileReaderPool) Counts 

func (p *FileReaderPool) Counts() (available int, locked int)

func (*FileReaderPool) IsPathInPool 

func (p *FileReaderPool) IsPathInPool(filePath string) bool

func (*FileReaderPool) LockNext 

func (p *FileReaderPool) LockNext() *FileReader

TODO: Figure out how to make this block, rather than return nil

func (*FileReaderPool) Remove 

func (p *FileReaderPool) Remove(reader *FileReader)

func (*FileReaderPool) Unlock 

func (p *FileReaderPool) Unlock(reader *FileReader)

func (*FileReaderPool) UnlockAll 

func (p *FileReaderPool) UnlockAll(readers []*FileReader)

type FileReaderPoolStatistics 

type FileReaderPoolStatistics struct {
	// The number of files in the pool that are available to be read
	Available int `json:"available"`

	// The number of files in the pool that are locked, ready to be sent, but
	// haven't been yet.
	Locked int `json:"locked"`
}

type FileStatistics 

type FileStatistics struct {
	// The current size of the file.
	Size int64 `json:"size"`

	// The current position (in bytes) that has been read into the file. This
	// might be greater than SnapshotPosition if there are lines buffered into
	// memory that haven't been acknowledged by the server
	Position int64 `json:"position"`

	// The last time the file was read from into the in-memory buffer.
	LastRead time.Time `json:"last_read"`

	// The current position (in bytes) that has been successfully sent and
	// acknowledged by the remote server.
	SnapshotPosition int64 `json:"snapshot_position"`

	// The last time a line from this file was successfully sent and acknowledged
	// by the remote server.
	LastSnapshot time.Time `json:"last_snapshot"`
}

type HighWaterMark 

type HighWaterMark struct {
	FilePath string

	// Position is the index in the file after a given line. Seeking to it would
	// read the next line.
	Position int64
}

type MemorySnapshotter 

type MemorySnapshotter struct {
	// contains filtered or unexported fields
}

func (*MemorySnapshotter) HighWaterMark 

func (s *MemorySnapshotter) HighWaterMark(filePath string) (*HighWaterMark, error)

func (*MemorySnapshotter) SetHighWaterMarks 

func (s *MemorySnapshotter) SetHighWaterMarks(marks []*HighWaterMark) error

type NetworkConfiguration 

type NetworkConfiguration struct {
	Servers     []ServerConfiguration `json:"servers"`
	Certificate string                `json:"certificate"`
	Key         string                `json:"key"`
	CA          string                `json:"ca"`
	Timeout     int                   `json:"timeout"`
	SpoolSize   int                   `json:"spool_size"`
}

type ServerConfiguration 

type ServerConfiguration struct {
	Addr string `json:"addr"`
	Name string `json:"name"`
}

type Snapshotter 

type Snapshotter interface {
	HighWaterMark(filePath string) (*HighWaterMark, error)
	SetHighWaterMarks(marks []*HighWaterMark) error
}

type Spooler 

type Spooler struct {
	In  chan *FileData
	Out chan []*FileData
	// contains filtered or unexported fields
}

Spooler accepts items on the In channel and chunks them into items on the Out channel.

func NewSpooler 

func NewSpooler(size int, timeout time.Duration) *Spooler

func (*Spooler) Spool 

func (s *Spooler) Spool()

Spool accepts items from the In channel and spools them into the Out channel. To stop the spooling, close the In channel.

type Statistics 

type Statistics struct {
	// contains filtered or unexported fields
}

Statistics keeps stats about the current operation of the program. It is meant to keep snapshot-in-time stats, as opposed to counters or timers that statsd offers.

Statistics may be exposed by APIs that allow human- or machine-readable monitoring. 

var GlobalStatistics *Statistics = NewStatistics()

func NewStatistics 

func NewStatistics() *Statistics

func (*Statistics) DeleteFileStatistics 

func (s *Statistics) DeleteFileStatistics(filePath string)

func (*Statistics) IncrementClientLinesSent 

func (s *Statistics) IncrementClientLinesSent(clientName string, linesSent int)

func (*Statistics) MarshalJSON 

func (s *Statistics) MarshalJSON() ([]byte, error)

func (*Statistics) SetClientStatus 

func (s *Statistics) SetClientStatus(clientName string, status string)

func (*Statistics) SetFilePosition 

func (s *Statistics) SetFilePosition(filePath string, position int64)

func (*Statistics) SetFileSnapshotPosition 

func (s *Statistics) SetFileSnapshotPosition(filePath string, snapshotPosition int64)

func (*Statistics) UpdateFileReaderPoolStatistics 

func (s *Statistics) UpdateFileReaderPoolStatistics(available int, locked int)

func (*Statistics) UpdateFileSizeStatistics 

func (s *Statistics) UpdateFileSizeStatistics()

UpdateFileSizeStatistics updates the Size attribute of each file, so it's easier to compare how much progress butteredscones has made through a file.

UpdateFileSizeStatistics should be called before displaying statistics to an end user.

type StatisticsConfiguration 

type StatisticsConfiguration struct {
	Addr string `json:"addr"`
}

type StatisticsServer 

type StatisticsServer struct {
	Statistics *Statistics
	Addr       string
}

StatisticsServer constructs an HTTP server that returns JSON formatted statistics. These statistics can be used for debugging or automated monitoring.

func (*StatisticsServer) ListenAndServe 

func (s *StatisticsServer) ListenAndServe() error

type Supervisor 

type Supervisor struct {

	// Optional settings
	SpoolSize int
	MaxLength int

	// How frequently to glob for new files that may have appeared
	GlobRefresh time.Duration
	// contains filtered or unexported fields
}

func NewSupervisor 

func NewSupervisor(files []FileConfiguration, clients []client.Client, snapshotter Snapshotter, maxLength int) *Supervisor

func (*Supervisor) Start 

func (s *Supervisor) Start()

Start pulls things together and plays match-maker.

func (*Supervisor) Stop 

func (s *Supervisor) Stop()

Stop stops the supervisor cleanly, making sure all progress has been snapshotted before exiting.

Source Files

View all Source files

Directories

Path Synopsis
cmd
butteredscones command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.