crawler

README

Streetcode Crawler

Crawl various websites in search of articles for Streetcode

Authors

• @jonesrussell

License

MIT

Environment Variables

To run this project, you will need to add the following environment variables to your .env file

REDIS_HOST

REDIS_PORT

REDIS_STREAM

Run Locally

Clone the project

git clone https://github.com/jonesrussell/streetcode-crawler.git

Go to the project directory

cd streetcode-crawler

crawler

cd cmd/crawler

Install dependencies

go install

Start the crawler

go run main.go https://www.sudbury.com

Usage/Examples

./crawler https://www.sudbury.com

Feedback

If you have any feedback, please reach out to me at https://jonesrussell42.xyz/contact