README
¶
Zeno
State-of-the-art web crawler 🔱
Introduction
Zeno is a web crawler designed to operate wide crawls or to simply archive one web page. Zeno's key concepts are: portability, performance, simplicity. With an emphasis on performance.
It heavily relies on the gowarc module for traffic recording into WARC files.
The name Zeno comes from Zenodotus (Ζηνόδοτος), a Greek grammarian, literary critic, Homeric scholar, and the first librarian of the Library of Alexandria.
Requirements for Building
- Go 1.25+ - As specified in go.mod
- If CGO_ENABLED=1 (enabled by default):
GCC 12+ - Required for building C++ dependencies with C++20 constexpr support for the WHATWG URL parser (github.com/ada-url/goada).
- If CGO_ENABLED=0:
No additional requirements, as the CGO-free WebAssembly wrapper of goada (goada-wasm) will be used. (1x slower than CGO version on amd64 and arm64, and 10x or more slower on other CPU architectures! Check https://wazero.io/docs/#compiler for details)
Note: GCC 11 and earlier versions do not support the C++20 constexpr features required by the ada-url/goada dependency. On Ubuntu 22 LTS and earlier, you may need to install a newer GCC version or disable CGO.
Installation
go install github.com/internetarchive/Zeno/v2@latest
or utilize our pre-built release binaries here, but do note that we are mainly focused on linux/amd64 support at this time.
Quick Start
To archive a single web page:
Zeno get url https://www.france.fr
Zeno is highly configurable with many parameters that can be customized. To see all available configuration options, use Zeno -h and/or Zeno get -h.
Contributing
Contributions are welcome! Please feel free to submit a Pull Request & open issues!
Zeno is being developed and maintained by the Internet Archive and awesome contributors. The project has evolved into what it is today thanks to the invaluable contributions from the community. While we can't list everyone, special thanks to:
- Corentin Barreau former Wayback Machine Software Engineer at the Internet Archive for his initial work on the project.
- Jake LaFountain, Wayback Machine Software Engineer at the Internet Archive.
- Thomas Foubert, former Wayback Machine Platform Engineer at the Internet Archive.
- yzqzss, Lead Developer of the Save The Web Project.
- Will Howes, Wayback Machine Software Engineer at the Internet Archive.
- Vangelis Banos, Wayback Machine Software Engineer at the Internet Archive.
License
This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See the LICENSE file for details.
Documentation
¶
Overview ¶
Authors:
Corentin Barreau <corentin@archive.org> Jake LaFountain <jakelf@archive.org> Thomas Foubert <thomas@archive.org>
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
pkg/api
Package api defines the web API for Zeno.
|
Package api defines the web API for Zeno. |
|
pkg/consul
Package consul provides an interface to Consul for Zeno's use.
|
Package consul provides an interface to Consul for Zeno's use. |
|
pkg/controler
Package controler provides a way to start and stop the pipeline.
|
Package controler provides a way to start and stop the pipeline. |
|
pkg/log
config.go
|
config.go |
|
pkg/postprocessor/domainscrawl
Package domainscrawl is a postprocessing component that parse domains from a given input and stores them for later matching.
|
Package domainscrawl is a postprocessing component that parse domains from a given input and stores them for later matching. |
|
pkg/preprocessor
Package preprocessor is the stage of the pipeline that :
|
Package preprocessor is the stage of the pipeline that : |
|
pkg/reactor
Package reactor provides functionality to manage and control the processing of seeds.
|
Package reactor provides functionality to manage and control the processing of seeds. |
|
pkg/source
Package source defines the interface for a data source that can produce items to crawl and accept items that have been crawled.
|
Package source defines the interface for a data source that can produce items to crawl and accept items that have been crawled. |
|
pkg/source/hq
Package hq provides a way to interact with the HQv3 API and consumes, produces and mark items as finished asynchronusly.
|
Package hq provides a way to interact with the HQv3 API and consumes, produces and mark items as finished asynchronusly. |
|
pkg/ui
Package ui provides the terminal user interface for Zeno.
|
Package ui provides the terminal user interface for Zeno. |
|
pkg
|
|