urlinsane

package module
v0.0.0-...-3c04461 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 24, 2019 License: MIT Imports: 20 Imported by: 0

README

URLInsane

Build Status

Multilingual domain typo permutation engine used to perform or detect typosquatting, brandjacking, URL hijacking, fraud, phishing attacks, corporate espionage and threat intelligence.

Documentation: URLInsane Docs

Downloads: URLInsane Downloads

URLInsane Demo1 URLInsane Demo2

Features

  • Binary executable, written in Go with no dependencies.
  • Will have all the functionally of URLCrazy and DNSTwist.
  • Contains 19 typosquatting algorithms and 10 extra functions to retrieve additional data such as ip to geographic location, dns lookups and more
  • Modular architecture for language, keyboard, typo algorithm, and functions extensibility.
  • Supports multiple keyboard layouts found in English, Spanish, Russian, Finish, and Arabic.
  • Supports multiple languages with the ability to add more languages with ease.
  • Concurrent function (-x --funcs) workers to retrieve additional info on each record.
  • Concurrent typo squatting workers.

Cli Tool

Multilingual domain typo permutation engine used to perform or detect typosquatting, brandjacking, URL hijacking, fraud, phishing attacks, corporate espionage and threat intelligence.

Usage:
  urlinsane [command]

Available Commands:
  help        Help about any command
  server      Start an API server to use this tool programmatically
  typo        Generates domain typos and variations

Flags:
  -h, --help   help for urlinsane

Use "urlinsane [command] --help" for more information about a command.
Squatting Options
urlinsane typo -h


Multilingual domain typo permutation engine used to perform or detect typosquatting, brandjacking, URL hijacking, fraud, phishing attacks, corporate espionage and threat intelligence.

USAGE:
  urlinsane [domains] [flags]

OPTIONS:
  -c, --concurrency int         Number of concurrent workers (default 50)
  -f, --file string             Output filename
  -o, --format string           Output format (csv, text) (default "text")
  -x, --funcs stringArray       Extra functions for data or filtering (default [idna])
  -h, --help                    help for urlinsane
  -k, --keyboards stringArray   Keyboards/layouts ID to use (default [en1])
  -t, --typos stringArray       Types of typos to perform (default [all])
  -v, --verbose                 Output additional details

KEYBOARDS:
  AR2	Arabic PC keyboard layout
  RU2	Phonetic Russian keybaord layout
  RU3	PC Russian keyboard layout
  ES1	Spanish keyboard layout
  AR1	Arabic keyboard layout
  EN2	English AZERTY keyboard layout
  ES2	Spanish ISO keyboard layout
  AR3	Arabic North african keyboard layout
  HY2	Armenian, Western QWERTY keyboard layout
  EN3	English QWERTZ keyboard layout
  EN4	English DVORAK keyboard layout
  FI1	Finnish QWERTY keybaord layout
  HY1	Armenian QWERTY keyboard layout
  EN1	English QWERTY keyboard layout
  IW1	Hebrew standard layout
  FA1	Persian standard layout
  RU1	Russian keyboard layout
  AR4	Arabic keyboard layout
  ALL	Use all keyboards

TYPOS: These are the types of typo/error algorithms that generate the domain variants
  MD	Missing Dot is created by omitting a dot from the domain.
  MDS	Missing Dashes is created by stripping all dashes from the domain.
  CO	Character Omission Omitting a character from the domain.
  CS	Character Swap Swapping two consecutive characters in a domain
  ACS	Adjacent Character Substitution replaces adjacent characters
  ACI	Adjacent Character Insertion inserts adjacent character 
  CR	Character Repeat Repeats a character of the domain name twice
  DCR	Double Character Replacement repeats a character twice.
  SD	Strip Dashes is created by omitting a dash from the domain
  SP	Singular Pluralise creates a singular domain plural and vice versa
  CM	Common Misspellings are created from a dictionary of commonly misspelled words
  VS	Vowel Swapping is created by swaps vowels
  HG	Homoglyphs replaces characters with characters that look similar
  WTLD	Wrong Top Level Domain
  W2TLD	Wrong Second Level Domain
  W3TLD	Wrong Third Level Domain
  HP	Homophones Typos are created from sets of words that sound the same
  BF	Bitsquatting relies on random bit-errors to redirect connections
  NS	Numeral Swap numbers, words and vice versa
  ALL   Apply all typosquatting algorithms

FUNCTIONS: Post processig functions that retieve aditional information on each domain variant.
  MX	Checking for DNS's MX records
  IP	Checking for IP address
  IDNA	Show international domain name
  TXT	Checking for DNS's TXT records
  NS	Checks DNS NS records
  CNAME	Checks DNS CNAME records
  SIM	Show domain content similarity
  LIVE	Show domains with ip addresses only
  301	Show domains redirects
  GEO	Show country location of ip address
  ALL  	Apply all post typosquating functions

EXAMPLE:

    urlinsane google.com
    urlinsane google.com -t co
    urlinsane google.com -t co -x ip -x idna -x ns

AUTHOR:
  Written by Rangertaha <rangertaha@gmail.com>

Server Options

urlinsane server -h

This command starts up a REST API server to use this tool programmatically.

Usage:
  urlinsane server [flags]

Flags:
  -a, --addr.host string   IP address for API server (default "127.0.0.1")
  -p, --addr.port string   Port to use (default "8888")
  -c, --concurrency int    Number of concurrent workers (default 50)
  -h, --help               help for server
      --log.file string    Filename to send logs to (default "urlinsane.log")
      --log.level string   Logging level (default "DEBUG")
  -s, --stream             Stream results via http2

Usage

Generates variations for google.com using the character omission (CO) algorithm.

urlinsane typo google.com -t co

 _   _  ____   _      ___
| | | ||  _ \ | |    |_ _| _ __   ___   __ _  _ __    ___
| | | || |_) || |     | | | '_ \ / __| / _' || '_ \  / _ \
| |_| ||  _ < | |___  | | | | | |\__ \| (_| || | | ||  __/
 \___/ |_| \_\|_____||___||_| |_||___/ \__,_||_| |_| \___|

 Version: 0.5.2

  LIVE | TYPE |   TYPO    | SUFFIX |   IDNA     
-------+------+-----------+--------+------------
       | CO   | oogle.com | com    | oogle.com  
       | CO   | gogle.com | com    | gogle.com  
       | CO   | goole.com | com    | goole.com  
       | CO   | gogle.com | com    | gogle.com  
       | CO   | googl.com | com    | googl.com  
       | CO   | googe.com | com    | googe.com

Additional extra functions can be selected with the -x, --funcs options. These functions can add columns to the output. For example the following generates variations for google.com using the character omission (CO) algorithm then checks for ip addresses.

urlinsane typo google.com -t co  -x geo

Generates variations for google.com with the following parameters:

  • -t hg lets us use the Homoglyphs(HG) algorithm only
  • -v Verbose mode shows us the full name 'Homoglyphs' of the algorithm not just the short name 'HG'
  • -x ip Check or ip address
  • -x idna Shows the IDNA format
  • -x ns Checks for DNS NS records
urlinsane typo google.com -t hg -v -x ip -x idna -x ns

Languages

English
  • Over 8000 common misspellings
  • Over 500 common homophones
  • English alphabet, vowels, homoglyphs, and numerals
  • Common keyboard layouts (qwerty, azerty, qwertz, dvorak)
Finnish, Russian, Persian, Hebrew, Arabic, Spanish

See Languages for details on other languages.

Algorithms

The modular architecture for code extensibility allows developers to add new typosquatting algorithms with ease. Currently we have implements 19 typosquatting algorithms. See Typo Algorithms for details.

Extra Functions

  • IDNA Show international domain name (Default)
  • MX Checking for DNS's MX records
  • TXT Checking for DNS's TXT records
  • IP Checking for IP address
  • NS Checks DNS NS records
  • CNAME Checks DNS CNAME records
  • SIM Show domain similarity % using fuzzy hashing with ssdeep
  • LIVE Show domains with ip addresses only
  • 301 Show domains redirects
  • GEO Show country location of ip address

Tools Comparisons

Language & Keyboard Comparison

This table shows which tools have support for common misspellings, homophones, numerals, vowels, homoglyphs, and the number of keyboards that support each language's character set.

Lang (# Keyboards) URLInsane URLCrazy DNSTwist DomainFuzz
Arabic (4) X
Armenian (3) X
English (4) X X X X
Finnish (1) X
Russian (3) X
Spanish (2) X
Hebrew (1) X
Persian (1) X
Algorithms

This table shows the list of algorithms supported for each tool.

Algorithms URLInsane URLCrazy DNSTwist DomainFuzz (TODO)
Missing Dot X X X
Missing Dashes X
Strip Dashes X X
Character Omission X X X
Character Swap X X
Adjacent Character Substitution X X
Adjacent Character Insertion X X X
Homoglyphs X X P
Singular Pluralise X X
Character Repeat X X X
Double Character Replacement X X
Common Misspellings X X
Homophones X X P
Vowel Swapping X X
Bitsquatting X X X
Wrong Top Level Domain X X
Wrong Second Level Domain X X
Wrong Third Level Domain X
Ordinal Number Swap X
Cardinal Number Swap X
Hyphenation X
Combosquatting(Keywords)
Multithreaded Algorithms X ? X

Post Typo Functions

Extra Functions URLInsane URLCrazy DNSTwist DomainFuzz
Live/Online Check X X X
DNS A Records X X X X
DNS MX Records X X X
DNS txt Records X X
DNS AAAA Records X X X
DNS CName Records X
DNS NS Records X X X
GeoIP Info X X X
Domain Similarity X X X
Domain Redirects X
IDNA Format X X
CSV output X X X X
JSON output X X X
Human Readable output X X X X
Google Popularity Estimate X
HTTP/SMTP Banner X
WHOIS Info X
Test MX email intercepts X
Multithreaded Extra Functions X X X
TODO
  • Extract keywords from domains. Keywords will be used for additional algorithms
  • Estimate popularity of a domain variant via google search
  • Lookup whois record
  • Emoji domains
  • Grabs HTTP and SMTP service banners

Authors

License

This project is licensed under the MIT License - see the LICENSE file for details

Documentation

Index

Constants

View Source 
const (
	VERSION = "0.5.2"
	DEBUG   = false
 _   _  ____   _      ___
| | | ||  _ \ | |    |_ _| _ __   ___   __ _  _ __    ___
| | | || |_) || |     | | | '_ \ / __| / _' || '_ \  / _ \
| |_| ||  _ < | |___  | | | | | |\__ \| (_| || | | ||  __/
 \___/ |_| \_\|_____||___||_| |_||___/ \__,_||_| |_| \___|

 Version: ` + VERSION + "\n"
)

Variables

View Source 
var FREGISTRY = make(map[string][]Extra)

The registry for extra functions

View Source 
var TLD = []string{}/* 8621 elements not displayed */
View Source 
var TREGISTRY = make(map[string][]Typo)

The registry for typos functions

Functions

func Asset 

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDir 

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy: 

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

func AssetInfo 

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames 

func AssetNames() []string

AssetNames returns the names of the assets.

func FRegister 

func FRegister(name string, efunc ...Extra)

FRegister

func MustAsset 

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func RestoreAsset 

func RestoreAsset(dir, name string) error

RestoreAsset restores an asset under the given directory

func RestoreAssets 

func RestoreAssets(dir, name string) error

RestoreAssets restores an asset under the given directory recursively

func TRegister 

func TRegister(name string, typo ...Typo)

TRegister

Types

type BasicConfig 

type BasicConfig struct {
	Domains     []string `json:"domains,omitempty"`
	Keyboards   []string `json:"keyboards,omitempty"`
	Typos       []string `json:"typos,omitempty"`
	Funcs       []string `json:"funcs,omitempty"`
	Concurrency int      `json:"concurrency,omitempty"`
	Format      string   `json:"format,omitempty"`
	File        string   `json:"file,omitempty"`
	Verbose     bool     `json:"verbose,omitempty"`
}

func (*BasicConfig) Config 

func (b *BasicConfig) Config() (c Config)

Config creates a Config

type Config 

type Config struct {
	// contains filtered or unexported fields
}

func CobraConfig 

func CobraConfig(cmd *cobra.Command, args []string) (c Config)

CobraConfig creates a configuration from a cobra command and arguments

func (*Config) GetConcurrency 

func (c *Config) GetConcurrency(concurrency int)

GetConcurrency

func (*Config) GetDomains 

func (c *Config) GetDomains(args []string)

GetDomains

func (*Config) GetFile 

func (c *Config) GetFile(file string)

GetFile

func (*Config) GetFormat 

func (c *Config) GetFormat(format string)

GetFormat

func (*Config) GetFuncs 

func (c *Config) GetFuncs(funcs []string)

GetFuncs

func (*Config) GetHeaders 

func (c *Config) GetHeaders(funcs []Extra)

GetHeaders

func (*Config) GetKeyboards 

func (c *Config) GetKeyboards(keyboards []string)

GetKeyboards retrieves a list of keyboards

func (*Config) GetTypos 

func (c *Config) GetTypos(typos []string)

GetTypos

func (*Config) GetVerbose 

func (c *Config) GetVerbose(verbose bool)

GetVerbose

type Domain 

type Domain struct {
	Subdomain string `json:"subdomain,omitempty"`
	Domain    string `json:"domain,omitempty"`
	Suffix    string `json:"suffix,omitempty"`
}

func (*Domain) Idna 

func (d *Domain) Idna() (punycode string)

Idna

func (*Domain) String 

func (d *Domain) String() (domain string)

String

type Extra 

type Extra struct {
	Code        string    `json:"code,omitempty"`
	Name        string    `json:"name,omitempty"`
	Description string    `json:"description,omitempty"`
	Headers     []string  `json:"code,omitempty"`
	Exec        ExtraFunc `json:"-"`
}

func FRetrieve 

func FRetrieve(strs ...string) (results []Extra)

FRetrieve

type ExtraFunc 

type ExtraFunc func(TypoResult) []TypoResult

ExtraFunc defines a function to register typos.

type OutputResult 

type OutputResult map[string]interface{}

type Typo 

type Typo struct {
	Code        string   `json:"code,omitempty"`
	Name        string   `json:"name,omitempty"`
	Description string   `json:"description,omitempty"`
	Exec        TypoFunc `json:"-"`
}

func TRetrieve 

func TRetrieve(strs ...string) (results []Typo)

TRetrieve

type TypoConfig 

type TypoConfig struct {
	Original  Domain               `json:"original,omitempty"`
	Variant   Domain               `json:"variant,omitempty"`
	Keyboards []languages.Keyboard `json:"keyboards,omitempty"`
	Languages []languages.Language `json:"languages,omitempty"`
	Typo      Typo                 `json:"typo,omitempty"`
}

type TypoFunc 

type TypoFunc func(TypoConfig) []TypoConfig

TypoFunc defines a function to register typos.

type TypoResult 

type TypoResult struct {
	Original Domain            `json:"original,omitempty"`
	Variant  Domain            `json:"variant,omitempty"`
	Typo     Typo              `json:"typo,omitempty"`
	Live     bool              `json:"live,omitempty"`
	Data     map[string]string `json:"data,omitempty"`
}

type URLInsane 

type URLInsane struct {
	// contains filtered or unexported fields
}

func New 

func New(c Config) (i URLInsane)

New

func (*URLInsane) Dedup 

func (urli *URLInsane) Dedup(in <-chan TypoResult) <-chan TypoResult

Dedup filters the results for unique variations of domains

func (*URLInsane) DistChain 

func (urli *URLInsane) DistChain(in <-chan TypoResult) <-chan TypoResult

DistChain creates workers of chained functions

func (*URLInsane) Execute 

func (urli *URLInsane) Execute() (res []TypoResult)

Execute program returning results

func (*URLInsane) FuncChain 

func (urli *URLInsane) FuncChain(funcs []Extra, in <-chan TypoResult) <-chan TypoResult

FuncChain creates a chain of extra functions

func (*URLInsane) GenTypoConfig 

func (urli *URLInsane) GenTypoConfig() <-chan TypoConfig

GenTypoConfig

func (*URLInsane) Output 

func (urli *URLInsane) Output(in <-chan TypoResult)

func (*URLInsane) Results 

func (urli *URLInsane) Results(in <-chan TypoConfig) <-chan TypoResult

Results

func (*URLInsane) Start 

func (urli *URLInsane) Start()

Start executes the program and outputs results. Primarily used for CLI tools

func (*URLInsane) Stream 

func (urli *URLInsane) Stream() <-chan TypoResult

Stream results via channels

func (*URLInsane) Typos 

func (urli *URLInsane) Typos(in <-chan TypoConfig) <-chan TypoConfig

Typos gives typo options to a pool of workers

Source Files

View all Source files

Directories

Path Synopsis
cmd
urlinsane

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.