web-scrapper

README

Distributed Web Scraper

A production-ready, scalable web scraper built in Go with clean gRPC API, featuring multiple queue backends, circuit breakers, and comprehensive observability.

Features

• Clean gRPC API: Simple, focused gRPC service for web scraping operations
• Multiple Queue Backends: Memory, Redis, and Kafka support
• Resilience: Circuit breakers, retries, dead letter queues, proxy rotation
• Rate Limiting: Per-host rate limiting with configurable policies
• Event Sourcing: Complete audit trail of scraping operations (Kafka mode)
• Observability: Prometheus metrics, structured logging, health checks
• Storage: SQLite for development with extensible storage interface
• Priority Processing: High, normal, and low priority queues

Quick Start

Prerequisites

• Go 1.19+
• Redis (optional, for Redis queue mode)
• Docker & Docker Compose (optional, for Kafka mode)

Installation

git clone <repository>
cd web-scrapper
go mod download
go build -o scraper

Basic Usage

# Memory queue (development)
./scraper -grpc-port 9090 -mem-queue -seed="https://example.com"

# Redis queue (production)
./scraper -grpc-port 9090 -seed="https://example.com"

# Kafka queue (high-scale)
docker-compose -f docker-compose.kafka.yml up -d
./scraper -grpc-port 9090 -kafka-queue -seed="https://example.com"

Using the API

# Submit URL for scraping
grpcurl -plaintext -d '{"url": "https://example.com", "priority": "NORMAL"}' \
  localhost:9090 scraper.WebScraperService/SubmitURL

# List scraped content
grpcurl -plaintext -d '{"page": 1, "page_size": 10}' \
  localhost:9090 scraper.WebScraperService/ListContent

# Health check
grpcurl -plaintext -d '{}' \
  localhost:9090 scraper.WebScraperService/HealthCheck

Architecture

graph TD
    Client[gRPC Clients] --> API[gRPC API Server]
    API --> Crawler[Crawler Service]
    
    Crawler --> Queue[Queue System]
    Queue --> Redis[Redis Queue]
    Queue --> Kafka[Kafka Queue]
    Queue --> Memory[Memory Queue]
    
    Crawler --> Storage[Storage Service]
    Crawler --> Circuit[Circuit Breaker]
    Crawler --> Proxy[Proxy Manager]
    
    Storage --> DB[(SQLite Database)]
    
    Crawler --> Metrics[Prometheus Metrics]

Component Documentation

Each component has detailed documentation in its respective directory:

Core Components

• api/ - gRPC API service and HTTP endpoints
• server/ - gRPC server implementation and service methods
• crawler/ - Core crawling logic, circuit breakers, rate limiting
• queue/ - Queue backends (Memory, Redis, Kafka)
• database/ - Storage systems and data models

Supporting Components

• config/ - Configuration management and settings
• metrics/ - Prometheus metrics and monitoring
• proxy/ - Proxy rotation and management

Command Line Options

-grpc-port int     # gRPC server port (default 9090)
-mem-queue         # Use in-memory queue
-kafka-queue       # Use Kafka queue  
-seed string       # Seed URL to start crawling

Configuration

Basic configuration example:

server:
  port: 8080
  timeout: 30s

crawler:
  workers: 10
  timeout: 30s
  userAgent: "WebScraper/1.0"

storage:
  type: "sqlite"
  path: "./data/scraper.db"

For detailed configuration options, see config/README.md.

Deployment

Docker

docker build -t web-scraper .
docker run -p 9090:9090 web-scraper

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-scraper
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: web-scraper
        image: web-scraper:latest
        ports:
        - containerPort: 9090

Monitoring

• Metrics: Available at http://localhost:2112/metrics
• Health Check: grpcurl -plaintext localhost:9090 grpc.health.v1.Health/Check
• Dashboard: Access Kafka UI at http://localhost:8090 (Kafka mode)

For detailed monitoring setup, see metrics/README.md.

Development

Run Tests

go test ./...

Generate Proto Files

protoc --go_out=. --go_opt=paths=source_relative \
       --go-grpc_out=. --go-grpc_opt=paths=source_relative \
       api/proto/scraper.proto

Performance & Scaling

• Memory Queue: Development and testing
• Redis Queue: Small to medium production deployments
• Kafka Queue: High-scale production with event sourcing

See component documentation for detailed performance tuning options.

Security

• gRPC TLS encryption ready
• Rate limiting per IP and domain
• Robots.txt compliance
• Proxy support for IP rotation
• Input validation and sanitization

License

MIT License - see LICENSE file for details.