linkutils

Documentation

Index

• func HappyResponse(resp *http.Response) bool
• func Hash(u *url.URL) string
• func IsNiceFileType(u *url.URL) bool
• func ParseURL(s string) (*url.URL, error)
• func ScrapeDaTing(u *url.URL) bool

Functions

func HappyResponse

func HappyResponse(resp *http.Response) bool

HappyResponse returns true if we want to continue scraping this thing.

func Hash

func Hash(u *url.URL) string

Hash returns a SHA1 hash of the host and path

func IsNiceFileType

func IsNiceFileType(u *url.URL) bool

IsNiceFileType returns true if the file extension is of type html (or unknown)

func ParseURL

func ParseURL(s string) (*url.URL, error)

ParseURL is a helper function that takes a string url, trims whitespace, parses into a url.URL and finally checks whether it fits our url requirements.

func ScrapeDaTing

func ScrapeDaTing(u *url.URL) bool

ScrapeDaTing gives us a yes/no on whether or not we should scrape the following URL, based on our opinionated filters