fetch

Documentation

Index

• func IsBlockedAddr(addr netip.Addr) bool
• func NewClient(cfg config.HTTPClientConfig, log *slog.Logger) *http.Client
• func SafeDialContext(base *net.Dialer) func(ctx context.Context, network, addr string) (net.Conn, error)
• type Fetcher
  • func NewFetcher(client *http.Client, cfg *config.AppConfig, log *slog.Logger) *Fetcher
  • func (f *Fetcher) FetchWithRetry(req *http.Request, ctx context.Context) (*http.Response, error)
• type HTTPFetcher
• type HostSemaphorePool
  • func NewHostSemaphorePool(maxPerHost int, log *slog.Logger) *HostSemaphorePool
  • func (p *HostSemaphorePool) Acquire(ctx context.Context, host string) error
  • func (p *HostSemaphorePool) Len() int
  • func (p *HostSemaphorePool) Release(host string)
  • func (p *HostSemaphorePool) RunEviction(ctx context.Context, interval time.Duration)
• type RateLimiter
  • func NewRateLimiter(defaultDelay time.Duration, log *slog.Logger) *RateLimiter
  • func (rl *RateLimiter) ApplyDelay(ctx context.Context, host string, minDelay time.Duration)
• type RobotsHandler
  • func NewRobotsHandler(fetcher HTTPFetcher, rateLimiter *RateLimiter, ...) *RobotsHandler
  • func (rh *RobotsHandler) GetRobotsData(targetURL *url.URL, signalChan chan<- bool, ctx context.Context) *robotstxt.RobotsData
  • func (rh *RobotsHandler) TestAgent(targetURL *url.URL, userAgent string, ctx context.Context) bool
• type SitemapDiscoverer

Functions

func IsBlockedAddr

func IsBlockedAddr(addr netip.Addr) bool

IsBlockedAddr reports whether the given IP address should be rejected from outbound HTTP dials to prevent SSRF against internal infrastructure. It blocks loopback, unspecified, link-local, private (RFC 1918 / RFC 4193), CGNAT, and multicast addresses.

func NewClient

func NewClient(cfg config.HTTPClientConfig, log *slog.Logger) *http.Client

NewClient creates an HTTP client with an SSRF-guarding dialer unless allow_private_networks is set.

func SafeDialContext

func SafeDialContext(base *net.Dialer) func(ctx context.Context, network, addr string) (net.Conn, error)

SafeDialContext wraps the given base dialer with a DialContext that resolves the destination hostname, rejects connections to private/loopback/link-local/ CGNAT/multicast addresses, and dials each pre-validated IP directly to prevent DNS-rebinding races between the check and the connect.

HTTPS still works because http.Transport sets TLS ServerName from the request URL, not from the dial target.

Types

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher makes HTTP requests with exponential-backoff retry logic.

func NewFetcher

func NewFetcher(client *http.Client, cfg *config.AppConfig, log *slog.Logger) *Fetcher

func (*Fetcher) FetchWithRetry

func (f *Fetcher) FetchWithRetry(req *http.Request, ctx context.Context) (*http.Response, error)

FetchWithRetry performs the request with exponential backoff and jitter for 5xx and 429 responses.

type HTTPFetcher

type HTTPFetcher interface {
	FetchWithRetry(req *http.Request, ctx context.Context) (*http.Response, error)
}

HTTPFetcher is the interface for performing HTTP requests with retry logic.

type HostSemaphorePool

type HostSemaphorePool struct {
	// contains filtered or unexported fields
}

HostSemaphorePool manages per-host semaphores for rate limiting concurrent requests to each host. A single pool should be shared across all components (crawler, image processor) so the per-host limit is enforced globally.

func NewHostSemaphorePool

func NewHostSemaphorePool(maxPerHost int, log *slog.Logger) *HostSemaphorePool

NewHostSemaphorePool creates a new pool with the given per-host concurrency limit.

func (*HostSemaphorePool) Acquire

func (p *HostSemaphorePool) Acquire(ctx context.Context, host string) error

Acquire gets or creates a host semaphore and acquires one permit, blocking until available or ctx is cancelled.

func (*HostSemaphorePool) Len

func (p *HostSemaphorePool) Len() int

Len returns the number of currently tracked hosts.

func (*HostSemaphorePool) Release

func (p *HostSemaphorePool) Release(host string)

Release releases one permit for the given host. Must be paired with a successful Acquire.

func (*HostSemaphorePool) RunEviction

func (p *HostSemaphorePool) RunEviction(ctx context.Context, interval time.Duration)

RunEviction periodically removes idle host entries; run in a goroutine.

type RateLimiter

type RateLimiter struct {
	// contains filtered or unexported fields
}

RateLimiter enforces a minimum spacing between successive requests to the same host.

func NewRateLimiter

func NewRateLimiter(defaultDelay time.Duration, log *slog.Logger) *RateLimiter

NewRateLimiter creates a RateLimiter whose per-host spacing defaults to defaultDelay when a caller does not supply a positive one.

func (*RateLimiter) ApplyDelay

func (rl *RateLimiter) ApplyDelay(ctx context.Context, host string, minDelay time.Duration)

ApplyDelay reserves the next politeness slot for host under the lock, then blocks until it is due (or ctx is cancelled). Reserving before sleeping is what makes spacing hold under concurrency: without it, several goroutines racing in for the same host would all read one stale timestamp and fire together. The reservation advances the clock immediately, so there is no separate post-request update to remember.

type RobotsHandler

type RobotsHandler struct {
	// contains filtered or unexported fields
}

RobotsHandler manages fetching, caching, and querying robots.txt data.

func NewRobotsHandler

func NewRobotsHandler(
	fetcher HTTPFetcher,
	rateLimiter *RateLimiter,
	globalSemaphore *semaphore.Weighted,
	sitemapNotifier SitemapDiscoverer,
	cfg *config.AppConfig,
	log *slog.Logger,
) *RobotsHandler

func (*RobotsHandler) GetRobotsData

func (rh *RobotsHandler) GetRobotsData(targetURL *url.URL, signalChan chan<- bool, ctx context.Context) *robotstxt.RobotsData

GetRobotsData returns parsed robots.txt for the host, fetching and caching on first call. Returns nil on any fetch/parse error (callers should treat nil as "allow all"). signalChan, if non-nil, receives true when the fetch completes (used for startup coordination).

func (*RobotsHandler) TestAgent

func (rh *RobotsHandler) TestAgent(targetURL *url.URL, userAgent string, ctx context.Context) bool

TestAgent reports whether the given user agent is allowed to access targetURL. Returns true if no robots data could be obtained (fail-open).

type SitemapDiscoverer

type SitemapDiscoverer interface {
	FoundSitemap(sitemapURL string)
}

SitemapDiscoverer is the callback interface for handling sitemap URLs found in robots.txt.