safefetch

package module
v0.0.0-...-ae279d0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2026 License: MIT Imports: 18 Imported by: 0

README

safefetch

Go Reference

A production-grade safe HTTP fetch library for Go with comprehensive SSRF protection, DNS Rebinding defense, redirect chain validation, response safety controls, and character encoding auto-detection/conversion.

Designed as a secure drop-in wrapper around the standard http.Client — ideal for any scenario where you need to fetch user-provided URLs.

Features

  • SSRF Protection — Blocks connections to private, loopback, link-local, and all reserved IP ranges (IPv4 & IPv6), including IPv4-mapped IPv6, NAT64, 6to4, and Teredo tunnel addresses
  • DNS Rebinding Defense — IP validation at DialContext, Dialer.Control, and post-connect remote-IP verification ensures the actually-connected IP is safe, eliminating TOCTOU vulnerabilities
  • Redirect Chain Validation — Every hop in a redirect chain is re-validated through all security layers; HTTPS→HTTP downgrade is blocked by default
  • Response Safety Controls — Body size limiting, Content-Type whitelisting (empty Content-Type header is permitted), response header size limiting, and multi-layer timeouts
  • Encoding Detection & Conversion — Auto-detects character encoding (BOM → Content-Type → HTML meta → heuristics) and converts to UTF-8
  • Proxy Support — Routes requests through HTTP, HTTPS, or SOCKS5 proxies while preserving URL pre-validation and redirect chain validation

Security Architecture

safefetch implements a 4-layer defense-in-depth architecture:

Layer Component Purpose
1 URL Pre-validation Scheme, port, hostname, credential, and backslash checks reject malicious URLs before any network I/O
2 DialContext IP Verification Every resolved IP is checked against built-in reserved/private ranges and custom allow/block lists at connection time
3 Dialer.Control + Remote IP Re-validation Re-checks the dial target before connect(), then validates the established connection's remote IP as an additional safety net
4 Redirect Chain Validation Each redirect target is re-validated through all prior layers

Installation

go get github.com/simp-lee/safefetch

Requires Go 1.25 or later.

Quick Start

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/simp-lee/safefetch"
)

func main() {
	client, err := safefetch.NewClient()
	if err != nil {
		log.Fatal(err)
	}

	result, err := client.Fetch(context.Background(), "https://example.com")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Status: %d\n", result.StatusCode)
	fmt.Printf("Content-Type: %s\n", result.ContentType)
	fmt.Printf("Encoding: %s (converted: %v)\n", result.OriginalEncoding, result.EncodingConverted)
	fmt.Printf("Body length: %d bytes\n", len(result.Body))
}

Configuration

safefetch uses the Functional Options pattern. All defaults are secure out of the box.

client, err := safefetch.NewClient(
	safefetch.WithMaxBodySize(5 << 20),                  // 5 MB (default: 10 MB)
	safefetch.WithTimeout(20 * time.Second),              // Overall timeout (default: 30s)
	safefetch.WithConnectTimeout(5 * time.Second),        // TCP connect timeout (default: 10s)
	safefetch.WithTLSHandshakeTimeout(5 * time.Second),   // TLS handshake timeout (default: 10s)
	safefetch.WithResponseHeaderTimeout(10 * time.Second),// Response header timeout (default: 15s)
	safefetch.WithAllowedPorts(80, 443, 8080),            // Port whitelist (default: 80, 443)
	safefetch.WithContentTypes("text/html", "application/json"), // Content-Type whitelist
	safefetch.WithUserAgent("MyBot/1.0"),                 // Custom User-Agent
	safefetch.WithHeaders(map[string]string{              // Custom request headers
		"Accept-Language": "en-US",
	}),
	safefetch.WithMaxRedirects(5),                        // Max redirects (default: 10)
	safefetch.WithMaxResponseHeaderBytes(512 << 10),      // Max response header size (default: 1 MB)
	safefetch.WithAllowedCIDR("10.0.1.0/24"),             // Trust specific internal CIDR
	safefetch.WithBlockedCIDR("203.0.113.0/24"),          // Block additional CIDR
	safefetch.WithAllowHTTPSDowngrade(),                  // Allow HTTPS→HTTP redirect
	safefetch.WithAllowCredentials(),                     // Allow embedded credentials in URLs
	safefetch.WithoutEncodingConversion(),                // Return raw bytes without UTF-8 conversion
	safefetch.WithResolver(customResolver),               // Custom DNS resolver (for testing)
	safefetch.WithProxy("http://proxy.corp.example:3128"), // HTTP/HTTPS/SOCKS5 proxy
)
Defaults
Setting Default
Max body size 10 MB
Overall timeout 30s
Connect timeout 10s
TLS handshake timeout 10s
Response header timeout 15s
Allowed ports 80, 443
Content-Types text/html, application/xhtml+xml, application/json, text/plain, application/xml, text/xml, application/rss+xml, application/atom+xml
Max redirects 10
Max response header size 1 MB
HTTPS downgrade Blocked
Embedded credentials Blocked
Encoding conversion Enabled
Proxy None (direct connection)

Proxy Support

Use WithProxy to route requests through an HTTP, HTTPS, or SOCKS5 proxy:

// HTTP proxy
client, err := safefetch.NewClient(
	safefetch.WithProxy("http://proxy.corp.example:3128"),
)

// SOCKS5 proxy
client, err := safefetch.NewClient(
	safefetch.WithProxy("socks5://127.0.0.1:1080"),
	safefetch.WithConnectTimeout(5 * time.Second), // controls proxy connect timeout
)

// Proxy with authentication
client, err := safefetch.NewClient(
	safefetch.WithProxy("http://user:pass@proxy.corp.example:3128"),
)

WithProxy returns ErrInvalidOption from NewClient if:

  • The proxy URL cannot be parsed
  • The scheme is not http, https, or socks5
  • The host part is empty
  • The port (if specified) is outside the 1–65535 range

Proxy configuration is explicit-only:

  • safefetch does not auto-read HTTP_PROXY / HTTPS_PROXY environment variables
  • safefetch does not implement no_proxy domain routing logic
  • Configure proxy behavior explicitly via WithProxy(...) in application code
Security Considerations in Proxy Mode

When a proxy is configured, the security posture changes as follows:

Security Layer Proxy Mode Behavior
URL Pre-validation (Layer 1) Fully enforced — scheme, port, hostname, credential, backslash checks all apply to the target URL
DialContext IP Verification (Layer 2) Checks the proxy server's IP, not the destination's — DNS resolution is delegated to the proxy
Dialer.Control + Remote IP Re-validation (Layer 3) Verifies the proxy server's IP
Redirect Chain Validation (Layer 4) Fully enforced — every redirect target is re-validated

Note: In proxy mode, the client cannot verify the destination IP address because DNS is resolved by the proxy server. Only configure trusted proxy servers. The WithConnectTimeout option controls the proxy connection timeout.

When no proxy is configured, all behavior is 100% backward-compatible.

Result

The Fetch method returns a *Result containing:

type Result struct {
	StatusCode        int         // HTTP status code
	Headers           http.Header // Response headers
	Body              []byte      // Response body (UTF-8, unless conversion is disabled)
	ContentType       string      // Parsed MIME type (e.g. "text/html")
	OriginalEncoding  string      // Detected original encoding (e.g. "gbk")
	EncodingConverted bool        // Whether encoding conversion was performed
	FinalURL          string      // Final URL after redirects
	RedirectCount     int         // Number of redirects followed
}

Error Handling

safefetch provides sentinel errors for precise error handling:

result, err := client.Fetch(ctx, url)
if err != nil {
	switch {
	case errors.Is(err, safefetch.ErrBlockedIP):
		// Connection to private/reserved IP blocked
	case errors.Is(err, safefetch.ErrBlockedScheme):
		// Non-HTTP(S) scheme rejected
	case errors.Is(err, safefetch.ErrBlockedPort):
		// Port not in whitelist
	case errors.Is(err, safefetch.ErrBlockedHost):
		// Known dangerous hostname blocked
	case errors.Is(err, safefetch.ErrBlockedRedirect):
		// Redirect target failed security checks
	case errors.Is(err, safefetch.ErrTooManyRedirects):
		// Redirect limit exceeded
	case errors.Is(err, safefetch.ErrBodyTooLarge):
		// Response body exceeds size limit
	case errors.Is(err, safefetch.ErrContentType):
		// Content-Type not in whitelist
	case errors.Is(err, safefetch.ErrTimeout):
		// Request timed out
	case errors.Is(err, safefetch.ErrCanceled):
		// Request canceled via context
	case errors.Is(err, safefetch.ErrBackslashInURL):
		// URL contains backslash
	case errors.Is(err, safefetch.ErrEmbeddedCredentials):
		// URL contains embedded credentials
	case errors.Is(err, safefetch.ErrInvalidOption):
		// Invalid client configuration
	}
}

What Gets Blocked

URL Validation
  • Non-HTTP(S) schemes (file://, gopher://, ftp://, data:, etc.)
  • Ports outside the whitelist (default: only 80 and 443)
  • URLs containing backslashes (parser inconsistency exploit)
  • Embedded credentials (http://user:pass@host/)
  • Known dangerous hostnames (metadata.google.internal, kubernetes.default.svc, etc.)
  • Encoded IP hostnames in decimal/octal/hex forms (e.g. 2130706433, 0177.0.0.1, 0x7f000001)
IP Ranges (IPv4)
Range Description
10.0.0.0/8 RFC 1918 Private
172.16.0.0/12 RFC 1918 Private
192.168.0.0/16 RFC 1918 Private
127.0.0.0/8 Loopback
169.254.0.0/16 Link-Local (cloud metadata 169.254.169.254)
100.64.0.0/10 CGN / RFC 6598 (Alibaba Cloud metadata 100.100.100.200)
0.0.0.0/8 This Network
192.0.0.0/24 IETF Protocol Assignments
192.0.2.0/24 TEST-NET-1
192.88.99.0/24 6to4 Relay Anycast (deprecated; blocked conservatively)
198.51.100.0/24 TEST-NET-2
203.0.113.0/24 TEST-NET-3
198.18.0.0/15 Benchmarking
224.0.0.0/4 Multicast
240.0.0.0/4 Reserved
255.255.255.255/32 Broadcast
IP Ranges (IPv6)
Range Description
::/128 Unspecified
::1/128 Loopback
::ffff:0:0/96 IPv4-mapped IPv6
64:ff9b:1::/48 NAT64 Local-Use
fc00::/7 Unique Local Address
fe80::/10 Link-Local
ff00::/8 Multicast
2001::/23 IETF Protocol Assignments
2001:db8::/32 Documentation
100::/64 Discard
Tunnel / Mapping Addresses
  • IPv4-mapped IPv6 (::ffff:x.x.x.x) — unpacked and re-checked
  • NAT64 (64:ff9b::/96, 64:ff9b:1::/48) — unpacked and re-checked
  • 6to4 (2002::/16) — unpacked and re-checked
  • Teredo (2001::/32) — unpacked and re-checked

Thread Safety

Client instances are safe for concurrent use. Create once, share freely across goroutines.

Dependencies

Minimal external dependencies:

License

MIT

Documentation

Overview

Package safefetch provides a production-grade safe HTTP fetch library with comprehensive SSRF protection, DNS Rebinding defense, redirect chain validation, response safety controls, and character encoding auto-detection and conversion.

Security Architecture

safefetch implements a 4-layer defense-in-depth architecture:

  1. URL Pre-validation — scheme, port, hostname, credential, and backslash checks reject obviously malicious URLs before any network I/O.
  2. DialContext IP Verification — every resolved IP address is checked against built-in reserved/private ranges (RFC 1918, RFC 4193, CGN, link-local, loopback, TEST-NET, NAT64, 6to4, Teredo, etc.) and custom allow/block lists. This runs at connection time, defeating DNS Rebinding attacks.
  3. Dialer Address Re-validation — Dialer.Control re-checks the dial target address parameters before connect, and the established connection's remote IP is validated after connect as an additional safety net.
  4. Redirect Chain Validation — each redirect target is re-validated through all prior layers, preventing redirect-based SSRF bypasses.

Security Features

  • SSRF Protection: blocks connections to private, loopback, link-local, and other reserved IP ranges for both IPv4 and IPv6.
  • DNS Rebinding Defense: IP checks at DialContext, Dialer.Control, and post-connect remote IP validation ensure the actually-connected IP is safe, not just the initially-resolved one.
  • Redirect Chain Validation: every hop in a redirect chain is validated against the same security policies.
  • Response Body Size Limiting: prevents memory exhaustion from oversized responses (default 10 MB).
  • Content-Type Whitelisting: restricts acceptable MIME types (configurable via WithContentTypes; default allows text/html, application/xhtml+xml, application/json, text/plain, application/xml, text/xml, application/rss+xml, application/atom+xml). Responses without a Content-Type header are permitted.
  • Encoding Detection & Conversion: auto-detects response character encoding and converts to UTF-8 when needed. If conversion fails, the original bytes are returned unchanged and EncodingConverted remains false.

Proxy Support

Use WithProxy to route requests through an HTTP, HTTPS, or SOCKS5 proxy: 

client, _ := safefetch.NewClient(
	safefetch.WithProxy("http://proxy.corp.example:3128"),
)

Security considerations in proxy mode:

  • Degraded layer: DNS resolution is performed by the proxy server, so the client cannot verify the destination site's actual IP address. The DialContext IP verification (layer 2) checks the proxy server's IP instead of the target's.
  • Still enforced: URL pre-validation (scheme, port, dangerous hostnames), redirect chain validation, and IP verification of the proxy server itself.
  • Proxy connection timeout is controlled by WithConnectTimeout.
  • Only configure trusted proxy servers. The proxy effectively becomes part of your trust boundary.

When no proxy is configured, behavior is 100% backward-compatible.

Usage

Create a Client with functional options, then call Fetch: 

client, err := safefetch.NewClient(
	safefetch.WithMaxBodySize(5 << 20),          // 5 MB limit
	safefetch.WithTimeout(15 * time.Second),
	safefetch.WithContentTypes("text/html", "application/json"),
	safefetch.WithAllowedPorts(80, 443),
)
if err != nil {
	log.Fatal(err)
}

result, err := client.Fetch(ctx, "https://example.com/api/data")
if err != nil {
	// Handle error — may be a sentinel error such as
	// safefetch.ErrBlockedIP or safefetch.ErrContentType.
	log.Fatal(err)
}

fmt.Println(result.StatusCode)
fmt.Println(string(result.Body))

Fetching through a proxy: 

proxyClient, err := safefetch.NewClient(
	safefetch.WithProxy("socks5://127.0.0.1:1080"),
	safefetch.WithTimeout(30 * time.Second),
	safefetch.WithConnectTimeout(5 * time.Second),
)
if err != nil {
	log.Fatal(err)
}

result, err = proxyClient.Fetch(ctx, "https://example.com/api/data")

All security checks are enabled by default with safe defaults. Use the With* option functions to customize behavior for your use case.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrBlockedIP           = errors.New("safefetch: IP address is blocked")
	ErrBlockedScheme       = errors.New("safefetch: URL scheme is blocked")
	ErrBlockedPort         = errors.New("safefetch: port is blocked")
	ErrBlockedRedirect     = errors.New("safefetch: redirect target is blocked")
	ErrTooManyRedirects    = errors.New("safefetch: too many redirects")
	ErrBodyTooLarge        = errors.New("safefetch: response body too large")
	ErrContentType         = errors.New("safefetch: content type is not allowed")
	ErrTimeout             = errors.New("safefetch: request timed out")
	ErrCanceled            = errors.New("safefetch: request canceled")
	ErrBlockedHost         = errors.New("safefetch: host is blocked")
	ErrBackslashInURL      = errors.New("safefetch: backslash in URL is not allowed")
	ErrEmbeddedCredentials = errors.New("safefetch: embedded credentials in URL are not allowed")
	ErrInvalidOption       = errors.New("safefetch: invalid option")
)

Sentinel errors returned by safefetch when a request is denied or fails.

Functions

This section is empty.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the main safefetch client. It holds resolved configuration, a pre-configured *http.Client, and precomputed CIDR prefix lists.

func NewClient 

func NewClient(opts ...Option) (*Client, error)

NewClient creates a new Client with the given options. It parses CIDR strings, builds the transport, and wires up the redirect policy.

func (*Client) Fetch 

func (c *Client) Fetch(ctx context.Context, rawURL string) (*Result, error)

Fetch retrieves rawURL safely, applying all security checks (URL validation, IP filtering, redirect policy, Content-Type whitelist, body size limit) and optional encoding conversion.

type Option 

type Option func(*options)

Option configures an options struct.

func WithAllowCredentials 

func WithAllowCredentials() Option

WithAllowCredentials permits URLs that contain embedded credentials.

func WithAllowHTTPSDowngrade 

func WithAllowHTTPSDowngrade() Option

WithAllowHTTPSDowngrade permits HTTPS to HTTP downgrade during redirects.

func WithAllowedCIDR 

func WithAllowedCIDR(cidrs ...string) Option

WithAllowedCIDR sets custom allowed CIDR ranges (parsed later in NewClient).

func WithAllowedPorts 

func WithAllowedPorts(ports ...int) Option

WithAllowedPorts sets the whitelisted ports for outgoing requests.

func WithBlockedCIDR 

func WithBlockedCIDR(cidrs ...string) Option

WithBlockedCIDR sets custom blocked CIDR ranges (parsed later in NewClient).

func WithConnectTimeout 

func WithConnectTimeout(d time.Duration) Option

WithConnectTimeout sets the TCP connection timeout. NewClient returns ErrInvalidOption when d <= 0.

func WithContentTypes 

func WithContentTypes(types ...string) Option

WithContentTypes sets the whitelisted Content-Type values. If called with one or more types, only responses matching those types are accepted. If called with no arguments (nil variadic), Content-Type filtering is disabled entirely and all response types are accepted. Expanding an explicit empty non-nil slice causes NewClient to return ErrInvalidOption.

func WithHeaders 

func WithHeaders(headers map[string]string) Option

WithHeaders sets custom request headers.

func WithMaxBodySize 

func WithMaxBodySize(n int64) Option

WithMaxBodySize sets the maximum response body size in bytes. NewClient returns ErrInvalidOption when n <= 0.

func WithMaxRedirects 

func WithMaxRedirects(n int) Option

WithMaxRedirects sets the maximum number of redirects to follow. NewClient returns ErrInvalidOption when n <= 0.

func WithMaxResponseHeaderBytes 

func WithMaxResponseHeaderBytes(n int64) Option

WithMaxResponseHeaderBytes sets the maximum size of response headers in bytes. NewClient returns ErrInvalidOption when n <= 0.

func WithProxy 

func WithProxy(proxyURL string) Option

WithProxy sets the proxy URL for outgoing requests. Supported schemes are http, https, and socks5. The raw string is always stored; if url.Parse fails, the parsed URL is left nil and NewClient will return ErrInvalidOption during validation.

func WithResolver 

func WithResolver(r Resolver) Option

WithResolver sets a custom DNS resolver.

func WithResponseHeaderTimeout 

func WithResponseHeaderTimeout(d time.Duration) Option

WithResponseHeaderTimeout sets the timeout for reading response headers. NewClient returns ErrInvalidOption when d <= 0.

func WithTLSHandshakeTimeout 

func WithTLSHandshakeTimeout(d time.Duration) Option

WithTLSHandshakeTimeout sets the TLS handshake timeout. NewClient returns ErrInvalidOption when d <= 0.

func WithTimeout 

func WithTimeout(d time.Duration) Option

WithTimeout sets the overall request timeout. NewClient returns ErrInvalidOption when d <= 0.

func WithUserAgent 

func WithUserAgent(ua string) Option

WithUserAgent sets the User-Agent header for requests.

func WithoutEncodingConversion 

func WithoutEncodingConversion() Option

WithoutEncodingConversion disables automatic encoding conversion to UTF-8.

type Resolver 

type Resolver interface {
	LookupNetIP(ctx context.Context, network, host string) ([]netip.Addr, error)
}

Resolver is the interface used for DNS resolution. *net.Resolver satisfies this interface.

type Result 

type Result struct {
	// StatusCode is the HTTP status code returned by the server.
	StatusCode int

	// Headers contains the HTTP response headers.
	Headers http.Header

	// Body is the raw response body as bytes.
	Body []byte

	// ContentType is the MIME type of the response (e.g. "text/html", "application/json").
	ContentType string

	// OriginalEncoding is the character encoding of the original response body
	// before any conversion (e.g. "gbk", "shift_jis").
	OriginalEncoding string

	// EncodingConverted indicates whether the response body was converted
	// from its original encoding to UTF-8.
	EncodingConverted bool

	// FinalURL is the URL after following all redirects.
	FinalURL string

	// RedirectCount is the number of redirects that were followed to reach the final URL.
	RedirectCount int
}

Result holds the outcome of a safe URL fetch operation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.