sigbits

README

sigbits

sigbits is a simple tool to pull out bits from URLS provided on stdin - Inspired by tomnomnom's unfurl.

Resources

• Installation
  • From Binary
  • From source
  • From github
• Usage
• Adding sigbits to your workflow
  • Install
  • Use
• Contribution

Installation

From Binary

You can download the pre-built binary for your platform from this repository's Releases page, extract, then move it to your $PATHand you're ready to go.

From Source

sigbits requires go1.14+ to install successfully. Run the following command to get the repo

$ GO111MODULE=on go get -u -v github.com/Mr-51gn3d-Projects/sigbits/cmd/sigbits

From Github

$ git clone https://github.com/Mr-51gn3d-Projects/sigbits.git; cd sigbits/cmd/sigbits/; go build; mv sigbits /usr/local/bin/; sigbits -h

Usage

sigbits works with URLs provided on stdin; they might come from a file like urls.txt:

$ cat urls.txt

https://sub.example.com/users?id=123&name=Sam
https://sub.example.com/orgs?org=ExCo#about
http://example.net/about#contact

You can extract:

• Hostnames from the URLs with the hostnames mode:

  $ cat urls.txt | sigbits domains
  
  sub.example.com
  sub.example.com
  example.net
  

• Paths, with the paths mode:

  $ cat urls.txt | sigbits paths
  
  /users
  /orgs
  /about
  

• Query String Keys, with the keys mode:

  $ cat urls.txt | sigbits keys
  
  id
  name
  org
  

• Query String Values, with the values mode:

  $ cat urls.txt | sigbits values
  
  123
  Sam
  ExCo
  

• Query String Key/Value Pairs , with the keypairs mode:

  $ cat urls.txt | sigbits keypairs
  
  id=123
  name=Sam
  org=ExCo
  

• NOTE: You can use the format mode to specify a custom output format:

  $ cat urls.txt | sigbits format %d%p
  
  sub.example.com/users
  sub.example.com/orgs
  example.net/about
  

  For more format directives, checkout the help message sigbits -h under Format Directives.

  Any characters that don't match a format directive remain untouched:

  $ cat urls.txt | sigbits -u format "%d (%s)"
  
  sub.example.com (https)
  example.net (http)
  

Note that if a URL does not include the data requested, there will be no output for that URL:

$ echo http://example.com | sigbits format "%P"

$ echo http://example.com:8080 | sigbits format "%P"
8080

To display help message for sigbits use the -h flag:

$ sigbits -h

help message:

Usage:
  sigbits [OPTIONS] [MODE] [FORMATSTRING]

Options:
  -u, --unique      only output unique values
  -v, --verbose     verbose mode: output URL parse errors

Modes:
  keys              keys from the query string (one per line)
  values            values from the query string (one per line)
  keypairs          `key=value` pairs from the query string (one per line)
  hostnames         the hostname (e.g. sub.example.com)
  paths             the request path (e.g. /users)
  format            specify a custom format (see below)

Format Directives:
  %%                a literal percent character
  %s                the request scheme (e.g. https)
  %u                tThe user info (e.g. user:pass)
  %h                the hostname (e.g. sub.example.com)
  %S                the subdomain (e.g. sub)
  %r                the root of domain (e.g. example)
  %t                the TLD (e.g. com)
  %P                the port (e.g. 8080)
  %p                the path (e.g. /users)
  %q                the raw query string (e.g. a=1&b=2)
  %f                the page fragment (e.g. page-section)
  %@                inserts an @ if user info is specified
  %:                inserts a colon if a port is specified
  %?                inserts a question mark if a query string exists
  %#                inserts a hash if a fragment exists
  %a                authority (alias for %u%@%d%:%P)

Examples:
  cat urls.txt | sigbits keys
  cat urls.txt | sigbits format %s://%d%p?%q

Adding sigbits to your workflow

The sigbits package has the same API as net/url except sigbits.URL contains extra fields.

Install

$ go get github.com/Mr-51gn3d-Projects/sigbits/pkg/sigbits

Use

package main

import (
	"fmt"

	"github.com/Mr-51gn3d-Projects/sigbits/pkg/sigbits"
)

func main() {
    u, _ := sigbits.Parse("postgres://user:pass@sub.host.com:5432/path?k=v#f")
    
		fmt.Printf("%50s = [ %s ] [ %s ] [ %s ] [ %s ] [ %s ]\n",
			u, u.Subdomain, u.Domain, u.TLD, u.Port, u.Path)
}

Contibution

Issues and Pull Requests are welcome!