config

Documentation

Index

• type Config
  • func LoadDefaultConfig() (*Config, error)
  • func LoadFromFile(filePath string) (*Config, error)

Types

type Config

type Config struct {
	MaxDepth        int      `json:"max_depth"`
	MinSleep        int      `json:"min_sleep"`
	MaxSleep        int      `json:"max_sleep"`
	Timeout         int      `json:"timeout"`
	RootURLs        []string `json:"root_urls"`
	BlacklistedURLs []string `json:"blacklisted_urls"`
	UserAgents      []string `json:"user_agents"`
}

Config represents the configuration for the crawler

func LoadDefaultConfig

func LoadDefaultConfig() (*Config, error)

LoadDefaultConfig loads the default configuration embedded in the binary

func LoadFromFile

func LoadFromFile(filePath string) (*Config, error)

LoadFromFile loads configuration from a JSON file