emailscraper

README

emailscraper

Minimalistic library to scrape emails from websites.

Requires chromium or google-chrome available in environment for JS render utilization.

Installation

go get github.com/lawzava/emailscraper

Usage

package main

import (
	"fmt"
	
	"github.com/lawzava/emailscraper"
)

func main() {
	s := emailscraper.New(emailscraper.DefaultConfig())

	extractedEmails, err := s.Scrape("https://lawzava.com")
	if err != nil {
		panic(err)
	}
	
	fmt.Println(extractedEmails)
}

Documentation

Index

• type Config
  • func DefaultConfig() Config
• type Scraper
  • func New(cfg Config) *Scraper
  • func (s *Scraper) Scrape(url string) ([]string, error)
  • func (s *Scraper) ScrapeWithContext(ctx context.Context, url string) ([]string, error)

Types

type Config

type Config struct {
	MaxDepth int
	Timeout  int

	// Rate limiting
	RateLimitDelay time.Duration
	Parallelism    int

	// Retry configuration
	MaxRetries int
	RetryDelay time.Duration

	// Behavior flags
	Recursively         bool
	Async               bool
	EnableJavascript    bool
	FollowExternalLinks bool
	RespectRobotsTxt    bool
	Debug               bool
}

Config for the scraper.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig defines default config with sane defaults for most use cases.

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper config.

func New

func New(cfg Config) *Scraper

New initiates new scraper entity.

func (*Scraper) Scrape

func (s *Scraper) Scrape(url string) ([]string, error)

Scrape is responsible for main scraping logic.

func (*Scraper) ScrapeWithContext

func (s *Scraper) ScrapeWithContext(ctx context.Context, url string) ([]string, error)

ScrapeWithContext is responsible for main scraping logic with context support.