parsers

Documentation

Index

• Variables
• func ExtractHostsFromResponse(needles []*regexp.Regexp, haystack string) map[string]struct{}
• func ExtractHostsFromResponses(needles []*regexp.Regexp, haystacks []string) map[string]struct{}
• func FQDNTrim(name string) string
• func GetDepth(hostAddress string) (int, error)
• func GetETLD(hostAddress string) (string, error)
• func GetSubDomain(hostAddress string) (string, error)
• func GetSubDomainAndDomain(hostAddress string) (string, string, error)
• func IsBannedIP(ip string) bool
• func IsETLD(hostAddress string) bool
• func ParseArpa(arpa string) (string, bool)
• func ParseCookies(s string) []*http.Cookie
• func ParseHTML(dom string, fn HTMLParserFn) error
• func ParseIPv4Arpa(ipv4arpa string) (string, bool)
• func ParseIPv6Arpa(ipv4arpa string) (string, bool)
• func SpecialETLD(hostAddress string) string
• func SplitAddresses(hostAddress string) ([]string, error)
• func ToIPv6(in []byte) string
• type HTMLParserFn

Variables

var BannedURLs = []string{
	"https://169.254.*",
	"https://127.*",
	"https://[::1]*",
	"https://[fe80::*]*",
	"https://10.*",
	"https://192.168.*",
	"http://169.254.*",
	"http://127.*",
	"http://[::1]*",
	"http://[fe80::*]*",
	"http://10.*",
	"http://192.168.*",
}

BannedURLs so we don't initiate bad requests

var (
	ErrHostIsIPAddress = errors.New("provided host is an ip address")
)

var SpecialCaseTLDs = map[string]struct{}{}/* 333 elements not displayed */

Functions

func ExtractHostsFromResponse

func ExtractHostsFromResponse(needles []*regexp.Regexp, haystack string) map[string]struct{}

ExtractHostsFromResponse returns potential hosts from a response TODO: make this a bit more robust.

func ExtractHostsFromResponses

func ExtractHostsFromResponses(needles []*regexp.Regexp, haystacks []string) map[string]struct{}

ExtractHostsFromResponses returns potential hosts from a response

func FQDNTrim

func FQDNTrim(name string) string

FQDNTrim trims the trailing .

func GetDepth

func GetDepth(hostAddress string) (int, error)

GetDepth returns how many subdomains the host address has: ex: test1.test2.example.com would return 3. ex2: test2.example.com would return 2 ex3: test1.amazon.co.uk would return 2

func GetETLD

func GetETLD(hostAddress string) (string, error)

GetETLD returns just the eltd of the supplied host address

func GetSubDomain

func GetSubDomain(hostAddress string) (string, error)

GetSubDomain returns the last sub domain part of a host address

func GetSubDomainAndDomain

func GetSubDomainAndDomain(hostAddress string) (string, string, error)

GetSubDomainAndDomain returns the subdomain + the rest of the domain, or error

func IsBannedIP

func IsBannedIP(ip string) bool

func IsETLD

func IsETLD(hostAddress string) bool

IsETLD returns true iff hostAddress is an etld amazon.co.uk == true sub.amazon.co.uk == false

func ParseArpa

func ParseArpa(arpa string) (string, bool)

ParseArpa parses an in-addr.arpa or ip6.arpa name to IP address.

func ParseCookies

func ParseCookies(s string) []*http.Cookie

Parse cookies by creating a fake response

func ParseHTML

func ParseHTML(dom string, fn HTMLParserFn) error

func ParseIPv4Arpa

func ParseIPv4Arpa(ipv4arpa string) (string, bool)

ParseIPv4Arpa uses sscanf to ensure we only get integer values for the in-addr.arpa string.

func ParseIPv6Arpa

func ParseIPv6Arpa(ipv4arpa string) (string, bool)

ParseIPv6Arpa uses sscanf to ensure we only get integer values for the in-addr.arpa string.

func SpecialETLD

func SpecialETLD(hostAddress string) string

SpecialETLD case where publicsuffix doesn't fit what we want to test for etlds.

func SplitAddresses

func SplitAddresses(hostAddress string) ([]string, error)

SplitAddresses preserving etld. sub1.sub2.test.co.uk would become []string{"test.co.uk", "sub2.test.co.uk"} returns nil if hostAddress = eltd and does not return the original address that was supplied.

func ToIPv6

func ToIPv6(in []byte) string

ToIPv6 takes an ipv6 address of bytes and returns a string representation

Types

type HTMLParserFn

type HTMLParserFn func(node *html.Node) error