jqtop

package module
Version: v0.0.0-...-08cbb52 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 15, 2021 License: MIT Imports: 23 Imported by: 0

README

GitHub release Built Status - CircleCI Build Status Actions Status codecov Go Report Card GoDoc

jqtop

jqtop is a tool to analyze json logs or equivalent stream of inputs in realtime.

Full manual is available here: https://github.com/shadyabhi/jqtop/wiki#manual

Installation

For now, we don't have rpm/deb packages. Download the latest release binary for your OS at https://github.com/shadyabhi/jqtop/releases.

go get github.com/shadyabhi/jqtop/cmd/jqtop

What does it do?

For eg, for a stream of logs like:-

{
  "time_local": "04/Oct/2018:06:30:27 +0000",
  "remote_addr": "69.162.124.230",
  "remote_user": "",
  "request": "HEAD /somepath.php HTTP/1.1",
  "status": "200",
  "body_bytes_sent": "0",
  "request_time": "0.000",
  "http_referrer": "https://abhi.host",
  "http_user_agent": "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
}

We can answer questions like:-

  • Stats for request.
  • Stats for derived fields like http_method using request field.
  • Filter lines that should be processed for stats

For the impatient folks, here's a sample usage:-

  • Count stats for a field that exists already, like request.
  • Count stats for a derived field, for ex, http_method. It's derived from request field by applying regex_capture function. more details
  • Filter only lines that have http_method == "GET". more details

Alt Text

Short tutorial

The CLI tool takes 3 basic parameters:-

  • --file: File that needs to be tailed.
  • --fields: Fields that need to be aggregated.
  • --filters: Filters used to filter lines that we want.
  • ... and others
$ ./jqtop -h
Usage: jqtop --file FILE [--interval INTERVAL] [--maxresult MAXRESULT] [--verbose] [--clearscreen] [--fields FIELDS] [--filters FILTERS]

Options:
  --file FILE            Path to file that will be read
  --interval INTERVAL, -i INTERVAL
                         Interval at which stats are calculated [default: 1]
  --maxresult MAXRESULT, -m MAXRESULT
                         Max results to show [default: 10]
  --verbose, -v
  --clearscreen, -c      Clear screen each time stats are shown
  --fields FIELDS        Fields that need to shown for stats
  --filters FILTERS      Filters to filter lines that'll be processed
  --help, -h
Existing Fields
Get stats for unique "requests" being made
$ ./jqtop -file ./logfile --fields request
2018/10/04 12:58:47 Seeked ./logfile - &{Offset:0 Whence:2}
✖ Parse error rate: 0
➤ request
└──    3: GET / HTTP/1.1
└──    2: GET /search/label/software%20access%20point?m=1 HTTP/1.1
└──    1: POST /1hou.php HTTP/1.1
└──    1: POST /miao.php HTTP/1.1
└──    1: POST /linuxse.php HTTP/1.1
└──    1: POST /tomcat.php HTTP/1.1
└──    1: POST /she.php HTTP/1.1
└──    1: POST /boots.php HTTP/1.1
└──    1: POST /qw.php HTTP/1.1
└──    1: POST /test.php HTTP/1.1

By default, stats are aggregated every second. You can change the internal using -i option.

Derived Fields

In the above json docs, we can see that there is no HTTP method field but a full string is available under "request" field.

Get stats on various HTTP methods being hit

We'll use a simple regex to derive the field "http_method" from the "request" field.

$ ./jqtop -file ./logfile --fields 'http_method = regex_capture(request, "(.*?) ")'
2018/10/05 06:45:23 Seeked ./logfile - &{Offset:0 Whence:2}
✖ Parse error rate: 0
➤ http_method
└──   18: GET
└──    4: HEAD

Show stats for multiple fields

We'll now show results for 2 fields, 1 being a regular field and another being a derived field. We can specify multiple fields by just deelimiting them with a semi-colon.

Let's show only top 2 results.

$ ./jqtop -file ./logfile --fields 'http_method = regex_capture(request, "(.*?) "); request' -m 2
2018/10/05 06:46:15 Seeked ./logfile - &{Offset:0 Whence:2}
✖ Parse error rate: 0
➤ request
└──    7: GET / HTTP/1.1
└──    2: GET /apple-touch-icon-precomposed.png HTTP/1.1

➤ http_method
└──   18: GET
└──    2: HEAD
Filtering

This is a complex example that uses both derived fields and filters.

Show top paths with http method as POST.
  • Create fields

There is no field which has context_path as it's value, so we'll have to create a new derived field.

paths = regex_capture(request, "[A-Z]+? (.*?) ")

As we need to filter logs that contain only POST, we'll need to create a new field http_method and filter based on that.

Our new field argument would be:-

paths = regex_capture(request, "[A-Z]+? (.*?) "); http_method = regex_capture(request, "(.*?) ");
  • Create filters

To filter based on http_method, out ---filters argument would look like:-

equals(http_method, "POST")
  • Final CLI options
$ ./jqtop -file ./logfile --fields 'paths = regex_capture(request, "[A-Z]+? (.*?) "); http_method = regex_capture(request, "(.*?) ");' ---filters 'equals(http_method, "POST")'
2018/10/05 07:07:38 Seeked ./logfile - &{Offset:0 Whence:2}
✖ Parse error rate: 0
➤ http_method
└──   18: POST

➤ paths
└──    1: /db__.init.php
└──    1: /wshell.php
└──    1: /xshell.php
└──    1: /qq.php
└──    1: /db_dataml.php
└──    1: /wc.php
└──    1: /xx.php
└──    1: /w.php
└──    1: /db_session.init.php
└──    1: /wp-admins.php

Benchmarks

I'm keeping a history of benchmarks as reference.

https://github.com/shadyabhi/jqtop/wiki/Benchmark-log

Documentation

Overview

Package jqtop is a tool to analyze json logs or equivalent stream of inputs in realtime.

Read: https://github.com/shadyabhi/jqtop/blob/master/README.md for more info

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DumpCounters

func DumpCounters(out io.Writer, totalIter int)

DumpCounters is a function to print stats to io.Writer nil io.Write is stdout

func ParseArgs

func ParseArgs() error

ParseArgs parses args and validates

func ProcessLines

func ProcessLines(lines chan *tail.Line)

ProcessLines reads from "lines" channel and processes them

func StartCLI

func StartCLI(outStream io.Writer)

StartCLI function is called from outside for default behavior

func TailFile

func TailFile(location string) (*tail.Tail, error)

TailFile tails the file that exists at path "location"

Types

type Arguments

type Arguments struct {
	File         string  `arg:"" help:"Path to file that will be read"`
	Interval     float64 `arg:"-i" help:"Interval at which stats are calculated"`
	MaxResult    int     `arg:"-m" help:"Max results to show"`
	Verbose      bool    `arg:"-v"`
	Clearscreen  bool    `arg:"-c" help:"Clear screen each time stats are shown"`
	ParallelProc int     `arg:"-p" help:"Number of parallel processes for processing"`
	Iter         int     `arg:"-s" help:"Automatically stop after n number of iterations "`

	Fields  string `arg:"required,separate" help:"Fields that need to shown for stats"`
	Filters string `arg:"separate" help:"Filters to filter lines that'll be processed"`
}

Arguments describes the argument that the program receives

var Config Arguments

Config stores all arguments passed via CLI

type Fields

type Fields struct {
	SimpleFields   []string
	DerivedFields  map[string]*derivedField
	FieldsInOrder  []string
	FieldsIndexMap map[string]int
}

Fields contains parsed fields that are sent

Directories

Path Synopsis
Package argparser is used to abstract the logic of parsing arguments in jqtop.
Package argparser is used to abstract the logic of parsing arguments in jqtop.
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL