scraper

README

Scraper Service

A Go-based web scraping service that extracts content, images, and metadata from web pages using Ollama AI models. Available as both a command-line tool and a REST API server with persistent storage.

Features

• AI-powered content extraction using Ollama
• Image analysis with vision models
• Link and metadata extraction
• SQLite storage with caching
• Batch URL processing
• REST API with CORS support
• UUID-based resource identification

Requirements

• Go 1.24 or higher
• Ollama running locally
• GCC (for SQLite CGO compilation)

Ollama Models

# Install required models
ollama pull llama3.2         # Text model
ollama pull llama3.2-vision  # Vision model for image analysis

Installation

# Build CLI tool
go build -o scraper-bin

# Build API server
go build -o scraper-api ./cmd/api

# Using Make
make build        # CLI only
make build-api    # API server only
make build-both   # Both CLI and API

Usage

Command-Line Tool

# Basic usage
./scraper-bin -url "https://example.com" -pretty

# Custom configuration
./scraper-bin -url "https://example.com" \
  -timeout 60s \
  -ollama-url "http://localhost:11434" \
  -ollama-model "llama3.2"

# Save output to file
./scraper-bin -url "https://example.com" -pretty > output.json

# Using Make
make run URL=https://example.com

API Server

# Start server (default port 8080)
./scraper-api

# Custom configuration
./scraper-api -addr :3000 -db ./data/scraper.db

# Using Make
make run-api
make run-api PORT=3000 DB=./data/scraper.db

Command-Line Options

CLI Tool:

• -url (required) - URL to scrape
• -timeout - Request timeout (default: 120s)
• -ollama-url - Ollama base URL (default: http://localhost:11434)
• -ollama-model - Ollama model (default: llama3.2)
• -pretty - Pretty print JSON output

API Server:

• -addr - Server address (default: :8080)
• -db - Database file path (default: scraper.db)
• -ollama-url - Ollama base URL (default: http://localhost:11434)
• -ollama-model - Ollama model (default: llama3.2)
• -disable-cors - Disable CORS support

Output Format

The scraper returns structured JSON data:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "url": "https://example.com",
  "title": "Page Title",
  "content": "AI-cleaned main content...",
  "images": [
    {
      "url": "https://example.com/image.jpg",
      "alt_text": "Image description",
      "summary": "AI-generated description",
      "tags": ["tag1", "tag2"]
    }
  ],
  "links": ["https://example.com/page1"],
  "fetched_at": "2024-01-01T12:00:00Z",
  "created_at": "2024-01-01T12:00:00Z",
  "processing_time_seconds": 3.45,
  "cached": false,
  "metadata": {
    "description": "Page meta description",
    "keywords": ["keyword1", "keyword2"],
    "author": "Author Name",
    "published_date": "2024-01-01"
  }
}

Architecture

Package Structure

• models/ - Data structures and types
• ollama/ - Ollama API client implementation
• scraper/ - Core scraping logic
• db/ - Database layer with migrations
• api/ - REST API server implementation
• cmd/ - Application entry points

Processing Pipeline

1. Fetch HTML content from target URL
2. Parse HTML structure
3. Extract title, text, images, links, and metadata
4. Clean content using Ollama AI
5. Analyze images with Ollama vision
6. Return structured JSON data

Error Handling

The scraper handles various error conditions:

• Invalid URLs
• Network timeouts
• HTTP errors (404, 500, etc.)
• Ollama connection issues
• Malformed HTML
• Image download failures

Image processing errors are isolated and do not fail the entire operation. If AI content extraction fails, the scraper falls back to raw text extraction.

Development

Make Commands

make help           # Show all available commands
make build          # Build CLI application
make build-api      # Build API server
make build-both     # Build both
make test           # Run tests
make test-coverage  # Generate coverage report
make coverage-html  # Generate HTML coverage
make clean          # Remove build artifacts
make fmt            # Format code
make vet            # Run go vet
make check          # Format, vet, and test
make build-cross    # Cross-compile for multiple platforms

Running Tests

# Run all tests
make test

# Generate coverage report
make test-coverage

# Generate HTML coverage report
make coverage-html

# Using Go directly
go test ./...
go test -v ./...
go test -cover ./...

Database

The API server uses SQLite with automatic migrations. The database schema includes:

• scraped_data - Stores scraped content with UUID-based IDs
• schema_migrations - Tracks applied database migrations

URLs are deduplicated using a unique constraint. Cached results are returned for previously scraped URLs unless the force parameter is used.

Switching to PostgreSQL

To use PostgreSQL instead of SQLite:

1. Add PostgreSQL driver:

   go get github.com/lib/pq
   

2. Update db/db.go import:

   import _ "github.com/lib/pq"
   

3. Update SQL syntax in migrations (AUTOINCREMENT → SERIAL, DATETIME → TIMESTAMP)

4. Use PostgreSQL connection string:

   ./scraper-api -db "postgres://user:pass@localhost/scraper?sslmode=disable"
   

Performance Considerations

• Default HTTP timeout: 120 seconds
• Concurrent batch processing (up to 50 URLs)
• Database connection pooling (25 max open, 5 idle)
• Image processing is sequential (can be parallelized)
• Processing time depends on Ollama hardware and model size

API Documentation

See API.md for complete API reference including:

• Endpoint specifications
• Request/response formats
• Error handling
• Code examples
• Integration patterns

License

This project is licensed under the MIT License - see the LICENSE file for details.

Documentation

Index

• Constants
• type Config
  • func DefaultConfig() Config
• type DB
• type Scraper
  • func New(config Config, db DB, storage StorageBackend) *Scraper
  • func (s *Scraper) Config() Config
  • func (s *Scraper) ExtractLinks(ctx context.Context, targetURL string) ([]string, error)
  • func (s *Scraper) OllamaClient() *ollama.Client
  • func (s *Scraper) ScoreLinkContent(ctx context.Context, targetURL string) (*models.LinkScore, error)
  • func (s *Scraper) Scrape(ctx context.Context, targetURL string) (*models.ScrapedData, error)
• type StorageBackend

Constants

const (
	// UserAgent mimics a recent Chrome browser to avoid being blocked
	UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
)

Types

type Config

type Config struct {
	HTTPTimeout         time.Duration
	OllamaBaseURL       string
	OllamaModel         string
	OllamaVisionModel   string        // Separate model for vision tasks (can be same as OllamaModel)
	EnableImageAnalysis bool          // Enable AI-powered image analysis
	MaxImageSizeBytes   int64         // Maximum image size to download (bytes)
	ImageTimeout        time.Duration // Timeout for downloading individual images
	LinkScoreThreshold  float64       // Minimum score for link to be recommended (0.0-1.0)
	StoragePath         string        // Base path for filesystem storage
	MaxImages           int           // Maximum number of images to download per scrape (0 = unlimited)
}

Config contains scraper configuration

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns default scraper configuration

type DB

type DB interface {
	GetImageByURL(url string) (*models.ImageInfo, error)
}

DB interface defines the database operations needed by the scraper

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper handles web scraping operations

func New

func New(config Config, db DB, storage StorageBackend) *Scraper

New creates a new Scraper instance db parameter can be nil if image deduplication is not needed storage parameter can be nil if storage is not needed

func (*Scraper) Config

func (s *Scraper) Config() Config

Config returns the scraper configuration

func (*Scraper) ExtractLinks

func (s *Scraper) ExtractLinks(ctx context.Context, targetURL string) ([]string, error)

ExtractLinks fetches a URL and returns links using Ollama with fallback to basic extraction

func (*Scraper) OllamaClient

func (s *Scraper) OllamaClient() *ollama.Client

OllamaClient returns the Ollama client for external use

func (*Scraper) ScoreLinkContent

func (s *Scraper) ScoreLinkContent(ctx context.Context, targetURL string) (*models.LinkScore, error)

ScoreLinkContent fetches and scores a URL to determine if it should be ingested

func (*Scraper) Scrape

func (s *Scraper) Scrape(ctx context.Context, targetURL string) (*models.ScrapedData, error)

Scrape fetches and processes a URL

type StorageBackend

type StorageBackend interface {
	SaveImage(imageData []byte, slug, contentType string) (string, error)
	SaveContent(content, slug string) (string, error)
	ReadImage(relPath string) ([]byte, error)
	ReadContent(relPath string) (string, error)
	DeleteImage(relPath string) error
	DeleteContent(relPath string) error
}

StorageBackend interface defines the storage operations needed by the scraper