crawler

package

v1.1.4 Latest Latest Go to latest Published: Sep 17, 2020 License: MIT Imports: 7 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/imthaghost/goclone

Links

Open Source Insights

Documentation ¶

Index ¶

func Collector(url string, projectPath string)
func Crawl(site string, projectPath string)
func Extractor(link string, projectPath string)
func HTMLExtractor(link string, projectPath string)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func Collector ¶

func Collector(url string, projectPath string)

Collector searches for css, js, and images within a given link TODO improve for better performance

func Crawl ¶

func Crawl(site string, projectPath string)

Crawl asks the necessary crawlers for collecting links for building the web page

func Extractor ¶

func Extractor(link string, projectPath string)

Extractor visits a link determines if its a page or sublink downloads the contents to a correct directory in project folder TODO add functionality for determining if page or sublink

func HTMLExtractor ¶

func HTMLExtractor(link string, projectPath string)

HTMLExtractor ...

Types ¶

This section is empty.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL