README
¶
Chaos Downloader
A simple tool to download, organize, and consolidate data from Project Discovery's Chaos dataset.
Getting Started
Ensure Go is installed on your system to run this script.
Steps
- Clone this repository:
git clone https://github.com/tonytsep/chaos_downloader.git
- Navigate to the repository folder:
cd chaos_downloader
- Execute the script:
go run chaos_downloader.go
The script downloads ZIP files listed in https://chaos-data.projectdiscovery.io/index.json, extracts them into named directories, and compiles text from those directories into a single file named everything.txt.
Usage
Running the script processes the data and generates everything.txt in the script's execution directory, alongside a folder AllChaosData containing the organized data.
Contributing
Feel free to fork, modify, or suggest improvements to this script. Any contribution is welcome.
License
This project is open-source, licensed under the MIT License.
Documentation
¶
Overview ¶
Package main implements a tool for downloading and organizing security data from Project Discovery's Chaos dataset. The tool fetches a list of targets from a JSON index, downloads ZIP archives associated with each target, extracts them into dedicated directories, and compiles all text file contents into a single file named 'everything.txt'. This file, 'everything.txt', is saved in the directory from which the script is executed, providing a consolidated view of the textual data gathered from the downloaded archives.
Source Files