webanalyze

README

webanalyze

This is a port of Wappalyzer in Go. This tool is designed to be performant and allows to test huge lists of hosts.

Installation and usage

$ go get -u github.com/rverton/webanalyze/...
$ webanalyze -update # loads new apps.json file from wappalyzer project
$ webanalyze -h
Usage of webanalyze:
  -apps string
        app definition file. (default "apps.json")
  -host string
        single host to test
  -hosts string
        filename with hosts, one host per line.
  -output string
        output format (stdout|csv|json) (default "stdout")
  -update
        update apps file
  -worker int
        number of worker (default 4)

The -update flags downloads a current version of apps.json from the wappalyzer repository to the current folder.

Display

Run cmd/webanalyze/index.html (on sth. like SimpleHTTPServer) to display results in a searchable dashboard.

Development / Usage as a lib

See cmd/webanalyze/main.go for an example.

Example

$ webanalyze -host stackshare.io
2017/06/19 10:22:23 Scanning with 4 workers.
2017/06/19 10:22:24 [+] http://stackshare.io (556.835509ms):
2017/06/19 10:22:24 	- Express,  (Web Frameworks, Web Servers)
2017/06/19 10:22:24 	- Nginx, 1.8.1 (Web Servers)
2017/06/19 10:22:24 	- Ruby on Rails,  (Web Frameworks)
2017/06/19 10:22:24 	- Google Font API,  (Font Scripts)

$ webanalyze -host stackshare.io -output csv
2017/06/19 10:22:50 Scanning with 4 workers.
Host,Category,App,Version
http://stackshare.io,"Web Frameworks,Web Servers",Express,
http://stackshare.io,Web Servers,Nginx,1.8.1
http://stackshare.io,Font Scripts,Google Font API,
http://stackshare.io,Web Frameworks,Ruby on Rails,

Documentation

Index

• Constants
• Variables
• func DownloadFile(from, to string) error
• func Init(workers int, hosts io.Reader, appsFile string) (chan Result, error)
• type App
  • func (app *App) FindInHeaders(headers http.Header) (matches [][]string, version string)
• type AppRegexp
• type AppsDefinition
• type Category
• type Job
  • func NewOfflineJob(url, body string, headers map[string][]string) *Job
  • func NewOnlineJob(url, body string, headers map[string][]string) *Job
• type Match
• type Result
• type StringArray
  • func (t *StringArray) UnmarshalJSON(data []byte) error
• type WebAnalyzer
  • func NewWebAnalyzer(workers int, appsFile string) (*WebAnalyzer, error)
  • func (wa *WebAnalyzer) Close()
  • func (wa *WebAnalyzer) Schedule(job *Job)

Constants

const WappalyzerURL = "https://raw.githubusercontent.com/AliasIO/Wappalyzer/master/src/apps.json"

WappalyzerURL is the link to the latest apps.json file in the Wappalyzer repo

Variables

var Timeout = 8 * time.Second

Functions

func DownloadFile

func DownloadFile(from, to string) error

DownloadFile pulls the latest apps.json file from the Wappalyzer github

func Init

func Init(workers int, hosts io.Reader, appsFile string) (chan Result, error)

Init sets up all the workders, reads in the host data and returns the results channel or an error

Types

type App

type App struct {
	Cats     StringArray       `json:"cats"`
	CatNames []string          `json:"category_names"`
	Headers  map[string]string `json:"headers"`
	Meta     map[string]string `json:"meta"`
	HTML     StringArray       `json:"html"`
	Script   StringArray       `json:"script"`
	URL      StringArray       `json:"url"`
	Website  string            `json:"website"`

	HTMLRegex   []AppRegexp `json:"-"`
	ScriptRegex []AppRegexp `json:"-"`
	URLRegex    []AppRegexp `json:"-"`
	HeaderRegex []AppRegexp `json:"-"`
	MetaRegex   []AppRegexp `json:"-"`
}

App type encapsulates all the data about an App from apps.json

func (*App) FindInHeaders

func (app *App) FindInHeaders(headers http.Header) (matches [][]string, version string)

type AppRegexp

type AppRegexp struct {
	Name    string
	Regexp  *regexp.Regexp
	Version string
}

type AppsDefinition

type AppsDefinition struct {
	Apps map[string]App      `json:"apps"`
	Cats map[string]Category `json:"categories"`
}

AppsDefinition type encapsulates the json encoding of the whole apps.json file

var (

	// AppDefs provides access to the unmarshalled apps.json file
	AppDefs *AppsDefinition
)

type Category

type Category struct {
	Name string `json:"name"`
}

type Job

type Job struct {
	URL     string
	Body    []byte
	Headers http.Header //map[string][]string
	// contains filtered or unexported fields
}

Job may consist only of a URL, in which case webanalyse will proceed to download from that URL, or it may consist of the Body and Headers of a request to a URL and the URL itself, in which case these fields will be trusted and used for analysis without further network traffic. If a Job is constructed using the OfflineJob constructor then a flag will be set to prevent downloading regardless of the contents (or absence) of the Body or Headers fields.

func NewOfflineJob

func NewOfflineJob(url, body string, headers map[string][]string) *Job

NewOfflineJob constructs a job out of the constituents of a webanalyzer analysis; a URL, a body, and response headers. This constructor also sets a flag to explicitly prevent fetching from the URL even if the body and headers are nil or empty. Use this for...offline jobs.

func NewOnlineJob

func NewOnlineJob(url, body string, headers map[string][]string) *Job

NewOnlineJob constructs a job that may either have a URL only, or a URL, Body and Headers. If it contains at least a URL and Body, then webanalyzer will not re-download the data, but if a Body is absent then downloading will be attempted.

type Match

type Match struct {
	App     `json:"app"`
	AppName string     `json:"app_name"`
	Matches [][]string `json:"matches"`
	Version string     `json:"version"`
}

Match type encapsulates the App information from a match on a document

type Result

type Result struct {
	Host     string        `json:"host"`
	Matches  []Match       `json:"matches"`
	Duration time.Duration `json:"duration"`
	Error    error         `json:"error"`
}

Result type encapsulates the result information from a given host

type StringArray

type StringArray []string

StringArray type is a wrapper for []string for use in unmarshalling the apps.json

func (*StringArray) UnmarshalJSON

func (t *StringArray) UnmarshalJSON(data []byte) error

UnmarshalJSON is a custom unmarshaler for handling bogus apps.json types from wappalyzer

type WebAnalyzer

type WebAnalyzer struct {
	Results chan Result
	// contains filtered or unexported fields
}

func NewWebAnalyzer

func NewWebAnalyzer(workers int, appsFile string) (*WebAnalyzer, error)

NewWebanalyzer returns an analyzer struct for an ongoing job, which may be "fed" jobs via a method and returns them via a channel when complete.

func (*WebAnalyzer) Close

func (wa *WebAnalyzer) Close()

func (*WebAnalyzer) Schedule

func (wa *WebAnalyzer) Schedule(job *Job)