crawler

command

v0.2.1 Latest Latest Go to latest Published: Sep 21, 2020 License: MIT Imports: 11 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This package contains a basic crawler built on top of Bokchoy.

go run main.go -run producer -url https://golang.org/

By default the depth is 1, you can override it with -depth flag.

go run main.go -run worker

By default the concurrency is 1, you can override it with -concurrency flag.

There is no documentation for this package.