README
¶
Crawler
Toy crawler for whatever you want to use it.
Requirements
go1.15
Installation
$ go install
Running
$ crawler -h
Usage:
crawler [OPTIONS]
Application Options:
-d, --domain= domain to crawl
-n, --workers= number of concurrent workers (default: 1)
-t, --timeout= Maximum dial timeout duration in seconds (default: 15)
--loglevel= log level {trace, debug, info, error, critical} (default: info)
Help Options:
-h, --help Show this help message
MIT License, 2020. Yarik Bratashchuk
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package data holds primitives directly related to the data processing
|
Package data holds primitives directly related to the data processing |
|
Package linkstorage holds everything related to storing links during the crawling process
|
Package linkstorage holds everything related to storing links during the crawling process |
Click to show internal directories.
Click to hide internal directories.