collector

Documentation

Overview

Package collector allows collecting logs from a k8s cluster brought up with bootkube semantics.

It creates following assets:

• fluentd-master deployment on one of the master nodes
• fluentd-worker daemonset to pushes all the logs to fluentd-master
• fluentd-master service for workers to talk to the master
• fluentd-master writes all container logs, docker and kubelet service logs to disk at /var/log/log-collector

For example:

Colecting and writting apiserver logs to local dir:

cr = collector.New(&collector.Config{
	K8sClient:     client,
	Namespace:     namespace,
})

 if err := cr.Start(); err != nil {
	...
 }

 if err := cr.OutputToLocal("/tmp/log-collector"); err != nil {
 	...
 }

 results, err := cr.CollectPodLogs("kube-apiserver")
 if err != nil {
 	...
 }

 if err := cr.Cleanup(); err != nil {
 	...
 }

Colecting and writting apiserver logs to s3:

cr = collector.New(&collector.Config{
	K8sClient:     client,
	Namespace:     namespace,
})

if err := cr.Start(); err != nil {
	...
}

if err := cr.OutputToS3(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), os.Getenv("AWS_REGION"), "log-collector", "prefix"); err != nil {
	...
}

results, err := cr.CollectPodLogs("kube-apiserver")
if err != nil {
	...
}

if err := cr.Cleanup(); err != nil {
	...
}

Index

• type Collector
  • func New(c *Config) *Collector
  • func (cr *Collector) Cleanup() error
  • func (cr *Collector) CollectPodLogs(pod string) ([]string, error)
  • func (cr *Collector) CollectServiceLogs(service string) ([]string, error)
  • func (cr *Collector) SetOutputToLocal(dstDir string) error
  • func (cr *Collector) SetOutputToS3(keyId, keySecret, region, bucketName, bucketPrefix string) error
  • func (cr *Collector) Start() error
• type Config
• type Output

Types

type Collector

type Collector struct {
	Output Output
	// contains filtered or unexported fields
}

Collector provides functions to collect logs.

func New

func New(c *Config) *Collector

New returns *Collector given a *Config.

func (*Collector) Cleanup

func (cr *Collector) Cleanup() error

Cleanup deletes the fluentd assets and removes all annotations that were created on nodes.

func (*Collector) CollectPodLogs

func (cr *Collector) CollectPodLogs(pod string) ([]string, error)

CollectPodLogs fetches the log file(s) for the pods name matching basic shell file name pattern and uploads all the file(s). It returns the list locations where the log(s) were uploaded. for example, pattern 'kube-*' returns kube-apiserver, kube-scheduler etc. and 'apiserver' returns kube-apiserver.

func (*Collector) CollectServiceLogs

func (cr *Collector) CollectServiceLogs(service string) ([]string, error)

CollectServiceLogs fetches the log file(s) for the services name matching basic shell file name pattern and uploads all the file(s). It returns the list locations where the log(s) were uploaded.

func (*Collector) SetOutputToLocal

func (cr *Collector) SetOutputToLocal(dstDir string) error

SetOutputToLocal sets the Collector output to local dir dstDir.

func (*Collector) SetOutputToS3

func (cr *Collector) SetOutputToS3(keyId, keySecret, region, bucketName, bucketPrefix string) error

SetOutputToS3 sets the Collector output to S3 bucket, given proper credentials.

func (*Collector) Start

func (cr *Collector) Start() error

Start creates the fluentd master and worker assets for log collection. It also registers the node that is running the fluentd master.

type Config

type Config struct {
	RemoteUser    string
	RemotePort    int32
	RemoteKeyFile string
	K8sClient     kubernetes.Interface
	Namespace     string
}

Config defines configuration options for the Collector.

Requires K8sClient and Namespce. If RemoteKeyFile is empty uses SSH_AUTH_SOCK to establish ssh connection. If RemoteUser is empty 'core' is used as default user for ssh connection. If RemotePort is empty '22' is used as default port for ssh connection.

type Output

type Output interface {
	Put(io.ReadSeeker, string) (string, error)
}

Output interface describes where the log file is uploaded.

Put accepts an 'io.ReadSeeker' of the log file and the 'filename' at the destination. Put returns the 'absolute (accessible) location' of the log file at the destination and an error if the file put was unsuccessful.