mink

README

mink

About

mink is a command line SEO tool that allows you to crawl URLs and get their basic metrics including, but not limited to: HTTP status code, indexibility, emails, meta description, size of the page, number of internal and external links and others.

It is a simple command-line alternative to tools like Screaming Frog SEO Spider, Netspeak Spider and other. It is useful to create plain-text or CSV report that can be used in spreadsheet software for further analysis.

Install

go install gitlab.com/ribtoks/mink@latest

Usage

Usage of ./mink:
  -depth int
    	Maximum depth for crawling (default 1)
  -external
    	Include external links
  -format string
    	Format of the output (table|csv|tsv) (default "table")
  -ignore-cert
    	Ignore expired certificates
  -index string
    	Indexability (ok | non)
  -log string
    	Path to the logfile (default "mink.log")
  -short
    	Print less output per URL
  -verbose
    	Write verbose logs

mink reads URLs from STDIN and writes reports to STDOUT. Report can be written in a form of a table, comma-separated values and tab-separated values.

Examples

Crawl all pages of a single website:

echo "https://your-website.com" | mink -depth 1000 -format csv > report.csv

Crawl a file with a list of URLs (1 per each line):

cat urls.txt | mink -format csv > report.csv

Verify if a list of URLs is indexable

cat urls.txt | mink -short -index non

Limitations

Currently mink does not handle javascript-based pages well.