synonyms

package
Version: v0.0.0-...-b28cec7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2020 License: MIT Imports: 7 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddEdgesIfDoNotExist

func AddEdgesIfDoNotExist(
	currentNode string,
	neighborNodes []string,
) (
	neighborsAdded []string,
	err error,
)

adds edge to DB, returns new neighbors added (to crawl on)

func CleanUrl

func CleanUrl(link string) string

decodes and standaridizes URL

func FilterPage

func FilterPage(e *colly.HTMLElement) (*colly.HTMLElement, error)

filters down full page body to elements we want to focus on

func GetRandomNode

func GetRandomNode() (string, error)
func IsValidCrawlLink(link string) bool

determines if is good link to crawl on

Types

This section is empty.

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to