go-log-aggregator

README

Log aggregator

Features

---

• Logger
• Config
• Client
• Server
• Retry manager
• Store queue
• Buffering before storing
• Exponential backoff (I am not sure if we need it here.)
• HTTP API for querying logs from server
• Filter and Search on server side
• Filter wrapper for filtering logs
• HTTP/GRPC endpoint for getting logs from other service

I am not going to create a query algorithm or query api because this is unneccessary, complex and time consuming. So my focus is to make it more like Fluentd. There will be plugin in future for data visualizing tools/platforms. eg: Elasticsearch,datadog,logstash etc. (If you have a solution, create an issue and I am open to discussion.)

Roadmap

---

Make its architecture like flutend.

Installation

---

go get -u github.com/itzmanish/go-log-aggregator

Make sure $GOPATH/bin directory is in your path

Usages

---

$ go-log-aggregator -h

log-aggregator is a log aggregating tool which provides a server and agent.
Server command is used to start a server. Whereas agent command is used to start
an agent so that logs can be sent from host machine to server and stored in File system or S3.

Usage:
  log-aggregator [flags]
  log-aggregator [command]

Available Commands:
  agent       Log aggregator agent for collecting logs and sending to server.
  help        Help about any command
  server      Log aggregator server to collect logs from agent and process it.

Flags:
      --config string   config file (default is $HOME/.config/.log-aggregator.yaml)
  -h, --help            help for log-aggregator
  -v, --version         version for log-aggregator

Use "log-aggregator [command] --help" for more information about a command.

Architecture

---

Contributions

---

Note This is just a practice exercise and not production ready.

If anyone wants to make this project real or have some good features/roadmap in mind. Feel free to raise a issue. Every contributions to this project is ❤️ welcome.

License

---

This project is licence under Apache Licence 2.0

Documentation

Overview

Copyright © 2021 Manish <itzmanish108@gmail.com>

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.