linktadoru

README

LinkTadoru

A high-performance web crawler and link analysis tool built in Go.

Features

• Fast Concurrent Crawling: Configurable worker pool for parallel processing
• Link Analysis: Maps internal and external link relationships
• Robots.txt Compliance: Respects robots.txt rules and crawl delays
• SQLite Storage: All data stored in a queryable SQLite database
• Resumable: Persistent queue for interrupted sessions
• Flexible Configuration: CLI flags, environment variables, or config file

Installation

Download Binary

Download pre-built binaries from the releases page.

Build from Source

git clone https://github.com/masahif/linktadoru.git
cd linktadoru
make build

Requirements: Go 1.23+

Quick Start

# Crawl a website
./linktadoru https://httpbin.org

# With options
./linktadoru --limit 100 --concurrency 5 https://httpbin.org

# Using config file
./linktadoru --config mysite.yaml https://httpbin.org

Documentation

• 📖 Basic Usage - Command-line usage and examples
• 🔧 Configuration - All configuration options
• 🏗️ Technical Details - Architecture and internals
• 🚀 Development - Building and contributing

Configuration

# config.yaml
concurrency: 10
request_delay: 1s
user_agent: "MyBot/1.0"
respect_robots: true
database_path: "./crawl.db"

Or use environment variables:

export LT_CONCURRENCY=5
export LT_REQUEST_DELAY=2s
./linktadoru https://httpbin.org

Contributing

See CONTRIBUTING.md for guidelines.

License

Apache License 2.0 - see LICENSE file.