gocrawl

package module

v1.1.0 Latest Latest Go to latest Published: Dec 22, 2021 License: MIT Imports: 6 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/fnkr/gocrawl

Links

Open Source Insights

README ¶

gocrawl

Simple http crawler, written in Go.

Usage example

$ gocrawl -url https://www.example.com -threads 10

Run gocrawl -help for help.

Documentation ¶

Index ¶

type Crawler
- func (crawler *Crawler) Crawl(startURL, post string)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Crawler ¶

type Crawler struct {
	AllowedURLPatterns    []*regexp.Regexp
	DisallowedURLPatterns []*regexp.Regexp
	NoParent              bool
	Threads               int
	InMemoryQueueStorage  *collyqueue.InMemoryQueueStorage
}

func (*Crawler) Crawl ¶

func (crawler *Crawler) Crawl(startURL, post string)

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
cmd

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL