httpclient

README

HTTP Client Module

This module provides a configurable HTTP client service that can be used by other modules in the modular framework. It supports configurable connection pooling, timeouts, and optional verbose logging of HTTP requests and responses.

Features

• Configurable HTTP client settings including connection pooling and timeouts
• Optional verbose logging of HTTP requests and responses
• Support for logging to files or application logger
• Request modifier support for customizing requests before they are sent
• Easy integration with other modules through service dependencies

Configuration

The module can be configured using YAML, JSON, or environment variables:

httpclient:
  max_idle_conns: 100             # Maximum idle connections across all hosts
  max_idle_conns_per_host: 10     # Maximum idle connections per host
  idle_conn_timeout: 90           # Maximum time an idle connection is kept alive (seconds)
  request_timeout: 30             # Default timeout for HTTP requests (seconds)
  tls_timeout: 10                 # TLS handshake timeout (seconds)
  disable_compression: false      # Whether to disable response body compression
  disable_keep_alives: false      # Whether to disable HTTP keep-alives
  verbose: false                  # Enable verbose logging of HTTP requests and responses
  verbose_options:                # Options for verbose logging (when verbose is true)
    log_headers: true             # Log request and response headers
    log_body: true                # Log request and response bodies
    max_body_log_size: 10000      # Maximum size of logged bodies (bytes)
    log_to_file: false            # Whether to log to files instead of application logger
    log_file_path: "/tmp/logs"    # Directory path for log files (required when log_to_file is true)

Integration with Other Modules

The HTTP client module provides a ClientService that can be used by other modules through service dependency injection. For example, to use this client in the reverseproxy module:

// In reverseproxy module:
func (m *ReverseProxyModule) RequiresServices() []modular.ServiceDependency {
	return []modular.ServiceDependency{
		{
			Name: "router", 
			Required: true, 
			MatchByInterface: true, 
			SatisfiesInterface: reflect.TypeOf((*handleFuncService)(nil)).Elem(),
		},
		{
			Name: "httpclient",
			Required: false, // Optional dependency
			MatchByInterface: true,
			SatisfiesInterface: reflect.TypeOf((*httpclient.ClientService)(nil)).Elem(),
		},
	}
}

Then in the constructor:

func (m *ReverseProxyModule) Constructor() modular.ModuleConstructor {
	return func(app modular.Application, services map[string]any) (modular.Module, error) {
		// Get router service
		handleFuncSvc, ok := services["router"].(handleFuncService)
		if !ok {
			return nil, fmt.Errorf("service %s does not implement HandleFunc interface", "router")
		}
		m.router = handleFuncSvc

		// Get optional HTTP client service
		if clientService, ok := services["httpclient"].(httpclient.ClientService); ok {
			// Use the provided HTTP client
			m.httpClient = clientService.Client()
		} else {
			// Create a default HTTP client
			m.httpClient = &http.Client{
				// Default settings...
			}
		}

		return m, nil
	}
}

Usage Example

package main

import (
	"github.com/CrisisTextLine/modular"
	"github.com/CrisisTextLine/modular/modules/httpclient"
	"github.com/CrisisTextLine/modular/modules/reverseproxy"
)

func main() {
	app := modular.NewApplication()
	
	// Register modules
	app.RegisterModule(httpclient.NewHTTPClientModule())
	app.RegisterModule(reverseproxy.NewModule())
	
	// The reverseproxy module will automatically use the httpclient service if available
	
	// Run the application
	if err := app.Run(); err != nil {
		panic(err)
	}
}

Documentation

Overview

Package httpclient provides a configurable HTTP client module for the modular framework.

Package httpclient provides a configurable HTTP client module for the modular framework.

Package httpclient provides a configurable HTTP client module for the modular framework.

This module offers a production-ready HTTP client with comprehensive configuration options, request/response logging, connection pooling, timeout management, and request modification capabilities. It's designed for reliable HTTP communication in microservices and web applications.

Features

The httpclient module provides the following capabilities:

• Configurable connection pooling and keep-alive settings
• Request and response timeout management
• TLS handshake timeout configuration
• Comprehensive request/response logging with file output
• Request modification pipeline for adding headers, authentication, etc.
• Performance-optimized transport settings
• Support for compression and keep-alive control
• Service interface for dependency injection

Configuration

The module can be configured through the Config structure:

config := &Config{
    MaxIdleConns:        100,        // total idle connections
    MaxIdleConnsPerHost: 10,         // idle connections per host
    IdleConnTimeout:     90,         // idle connection timeout (seconds)
    RequestTimeout:      30,         // request timeout (seconds)
    TLSTimeout:          10,         // TLS handshake timeout (seconds)
    DisableCompression:  false,      // enable gzip compression
    DisableKeepAlives:   false,      // enable connection reuse
    Verbose:             true,       // enable request/response logging
    VerboseOptions: &VerboseOptions{
        LogToFile:    true,
        LogFilePath:  "/var/log/httpclient",
    },
}

Service Registration

The module registers itself as a service for dependency injection:

// Get the HTTP client service
client := app.GetService("httpclient").(httpclient.ClientService)

// Use the client
resp, err := client.Client().Get("https://api.example.com/users")

// Create a client with custom timeout
timeoutClient := client.WithTimeout(60)
resp, err := timeoutClient.Post("https://api.example.com/upload", "application/json", data)

Usage Examples

Basic HTTP requests:

// GET request
resp, err := client.Client().Get("https://api.example.com/health")
if err != nil {
    return err
}
defer resp.Body.Close()

// POST request with JSON
jsonData := bytes.NewBuffer([]byte(`{"name": "test"}`))
resp, err := client.Client().Post(
    "https://api.example.com/users",
    "application/json",
    jsonData,
)

Request modification for authentication:

// Set up request modifier for API key authentication
modifier := func(req *http.Request) *http.Request {
    req.Header.Set("Authorization", "Bearer "+apiToken)
    req.Header.Set("User-Agent", "MyApp/1.0")
    return req
}
client.SetRequestModifier(modifier)

// All subsequent requests will include the headers
resp, err := client.Client().Get("https://api.example.com/protected")

Custom timeout scenarios:

// Short timeout for health checks
healthClient := client.WithTimeout(5)
resp, err := healthClient.Get("https://service.example.com/health")

// Long timeout for file uploads
uploadClient := client.WithTimeout(300)
resp, err := uploadClient.Post("https://api.example.com/upload", contentType, fileData)

Logging and Debugging

When verbose logging is enabled, the module logs detailed request and response information including headers, bodies, and timing data. This is invaluable for debugging API integrations and monitoring HTTP performance.

Log output includes:

• Request method, URL, and headers
• Request body (configurable)
• Response status, headers, and body
• Request duration and timing breakdown
• Error details and retry information

Performance Considerations

The module is optimized for production use with:

• Connection pooling to reduce connection overhead
• Keep-alive connections for better performance
• Configurable timeouts to prevent resource leaks
• Optional compression to reduce bandwidth usage
• Efficient request modification pipeline

Index

• Constants
• func NewHTTPClientModule() modular.Module
• type ClientService
• type Config
  • func (c *Config) GetTimeout(seconds int) time.Duration
  • func (c *Config) Validate() error
• type FileLogger
  • func NewFileLogger(baseDir string, logger modular.Logger) (*FileLogger, error)
  • func (f *FileLogger) Close() error
  • func (f *FileLogger) LogRequest(id string, data []byte) error
  • func (f *FileLogger) LogResponse(id string, data []byte) error
  • func (f *FileLogger) LogTransactionToFile(id string, reqData, respData []byte, duration time.Duration, url string) error
• type HTTPClientModule
  • func (m *HTTPClientModule) Client() *http.Client
  • func (m *HTTPClientModule) Dependencies() []string
  • func (m *HTTPClientModule) Init(app modular.Application) error
  • func (m *HTTPClientModule) Name() string
  • func (m *HTTPClientModule) ProvidesServices() []modular.ServiceProvider
  • func (m *HTTPClientModule) RegisterConfig(app modular.Application) error
  • func (m *HTTPClientModule) RequestModifier() RequestModifierFunc
  • func (m *HTTPClientModule) RequiresServices() []modular.ServiceDependency
  • func (m *HTTPClientModule) SetRequestModifier(modifier RequestModifierFunc)
  • func (m *HTTPClientModule) Start(context.Context) error
  • func (m *HTTPClientModule) Stop(context.Context) error
  • func (m *HTTPClientModule) WithTimeout(timeoutSeconds int) *http.Client
• type RequestModifierFunc
• type VerboseOptions

Constants

const ModuleName = "httpclient"

ModuleName is the unique identifier for the httpclient module.

const ServiceName = "httpclient"

ServiceName is the name of the service provided by this module. Other modules can use this name to request the HTTP client service through dependency injection.

Functions

func NewHTTPClientModule

func NewHTTPClientModule() modular.Module

NewHTTPClientModule creates a new instance of the HTTP client module. This is the primary constructor for the httpclient module and should be used when registering the module with the application.

Example:

app.RegisterModule(httpclient.NewHTTPClientModule())

Types

type ClientService

type ClientService interface {
	// Client returns the configured http.Client instance.
	// This client uses the module's configuration for timeouts, connection
	// pooling, compression, and other transport settings. The client is
	// thread-safe and can be used concurrently.
	//
	// The returned client includes any configured request modification
	// pipeline and verbose logging if enabled.
	Client() *http.Client

	// RequestModifier returns a modifier function that can modify a request before it's sent.
	// This function applies any configured request modifications such as
	// authentication headers, user agents, or custom headers.
	//
	// The modifier can be used manually when creating custom requests:
	//	req, _ := http.NewRequest("POST", url, body)
	//	req = modifier(req)
	//	resp, err := client.Do(req)
	RequestModifier() RequestModifierFunc

	// WithTimeout creates a new client with the specified timeout in seconds.
	// This is useful for creating clients with different timeout requirements
	// without affecting the default client configuration.
	//
	// The new client inherits all other configuration from the module
	// (connection pooling, compression, etc.) but uses the specified timeout.
	//
	// Common timeout scenarios:
	//   - Health checks: 5-10 seconds
	//   - API calls: 30-60 seconds
	//   - File uploads: 300+ seconds
	WithTimeout(timeoutSeconds int) *http.Client
}

ClientService defines the interface for the HTTP client service. This interface provides access to configured HTTP clients and request modification capabilities. Any module that needs to make HTTP requests can use this service through dependency injection.

The service provides multiple ways to access HTTP clients:

• Default client with module configuration
• Timeout-specific clients for different use cases
• Request modification pipeline for common headers/auth

Example usage:

// Basic usage
client := httpClientService.Client()
resp, err := client.Get("https://api.example.com/data")

// Custom timeout
shortTimeoutClient := httpClientService.WithTimeout(5)
resp, err := shortTimeoutClient.Get("https://api.example.com/health")

// Request modification
modifier := httpClientService.RequestModifier()
req, _ := http.NewRequest("GET", "https://api.example.com/data", nil)
modifiedReq := modifier(req)

type Config

type Config struct {
	// MaxIdleConns controls the maximum number of idle (keep-alive) connections across all hosts.
	// This setting affects the total connection pool size and memory usage.
	// Higher values allow more concurrent connections but use more memory.
	// Default: 100
	MaxIdleConns int `yaml:"max_idle_conns" json:"max_idle_conns" env:"MAX_IDLE_CONNS"`

	// MaxIdleConnsPerHost controls the maximum idle (keep-alive) connections to keep per-host.
	// This prevents a single host from monopolizing the connection pool.
	// Should be tuned based on expected traffic patterns to specific hosts.
	// Default: 10
	MaxIdleConnsPerHost int `yaml:"max_idle_conns_per_host" json:"max_idle_conns_per_host" env:"MAX_IDLE_CONNS_PER_HOST"`

	// IdleConnTimeout is the maximum amount of time an idle connection will remain idle
	// before closing itself, in seconds. This helps prevent stale connections and
	// reduces server-side resource usage.
	// Default: 90 seconds
	IdleConnTimeout int `yaml:"idle_conn_timeout" json:"idle_conn_timeout" env:"IDLE_CONN_TIMEOUT"`

	// RequestTimeout is the maximum time for a request to complete, in seconds.
	// This includes connection time, any redirects, and reading the response body.
	// Use WithTimeout() method for per-request timeout overrides.
	// Default: 30 seconds
	RequestTimeout int `yaml:"request_timeout" json:"request_timeout" env:"REQUEST_TIMEOUT"`

	// TLSTimeout is the maximum time waiting for TLS handshake, in seconds.
	// This only affects HTTPS connections and should be set based on expected
	// network latency and certificate chain complexity.
	// Default: 10 seconds
	TLSTimeout int `yaml:"tls_timeout" json:"tls_timeout" env:"TLS_TIMEOUT"`

	// DisableCompression disables decompressing response bodies.
	// When false (default), the client automatically handles gzip compression.
	// Set to true if you need to handle compression manually or want raw responses.
	// Default: false (compression enabled)
	DisableCompression bool `yaml:"disable_compression" json:"disable_compression" env:"DISABLE_COMPRESSION"`

	// DisableKeepAlives disables HTTP keep-alive and will only use connections for a single request.
	// This can be useful for debugging or when connecting to servers that don't handle
	// keep-alives properly, but significantly impacts performance.
	// Default: false (keep-alives enabled)
	DisableKeepAlives bool `yaml:"disable_keep_alives" json:"disable_keep_alives" env:"DISABLE_KEEP_ALIVES"`

	// Verbose enables detailed logging of HTTP requests and responses.
	// When enabled, logs include request/response headers, bodies, timing information,
	// and error details. Very useful for debugging but can impact performance.
	// Default: false
	Verbose bool `yaml:"verbose" json:"verbose" env:"VERBOSE"`

	// VerboseOptions configures the behavior when Verbose is enabled.
	// This allows fine-grained control over what gets logged and where.
	VerboseOptions *VerboseOptions `yaml:"verbose_options" json:"verbose_options" env:"VERBOSE_OPTIONS"`
}

Config defines the configuration for the HTTP client module. This structure contains all the settings needed to configure HTTP client behavior, connection pooling, timeouts, and logging.

Configuration can be provided through JSON, YAML, or environment variables. The struct tags define the mapping for each configuration source.

Example YAML configuration:

max_idle_conns: 200
max_idle_conns_per_host: 20
idle_conn_timeout: 120
request_timeout: 60
tls_timeout: 15
disable_compression: false
disable_keep_alives: false
verbose: true
verbose_options:
  log_headers: true
  log_body: true
  max_body_log_size: 1024
  log_to_file: true
  log_file_path: "/var/log/httpclient"

Example environment variables:

HTTPCLIENT_MAX_IDLE_CONNS=200
HTTPCLIENT_REQUEST_TIMEOUT=60
HTTPCLIENT_VERBOSE=true

func (*Config) GetTimeout

func (c *Config) GetTimeout(seconds int) time.Duration

GetTimeout converts a timeout value from seconds to time.Duration.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks the configuration values and sets sensible defaults.

type FileLogger

type FileLogger struct {
	// contains filtered or unexported fields
}

FileLogger handles logging HTTP request and response data to files.

func NewFileLogger

func NewFileLogger(baseDir string, logger modular.Logger) (*FileLogger, error)

NewFileLogger creates a new file logger that writes HTTP data to files.

func (*FileLogger) Close

func (f *FileLogger) Close() error

Close closes any open files and cleans up resources.

func (*FileLogger) LogRequest

func (f *FileLogger) LogRequest(id string, data []byte) error

LogRequest writes request data to a file.

func (*FileLogger) LogResponse

func (f *FileLogger) LogResponse(id string, data []byte) error

LogResponse writes response data to a file.

func (*FileLogger) LogTransactionToFile

func (f *FileLogger) LogTransactionToFile(id string, reqData, respData []byte, duration time.Duration, url string) error

LogTransactionToFile logs both request and response data to a single file for easier analysis.

type HTTPClientModule

type HTTPClientModule struct {
	// contains filtered or unexported fields
}

HTTPClientModule implements a configurable HTTP client module. It provides a production-ready HTTP client with comprehensive configuration options, logging capabilities, and request modification features.

The module implements the following interfaces:

• modular.Module: Basic module lifecycle
• modular.Configurable: Configuration management
• modular.ServiceAware: Service dependency management
• ClientService: HTTP client service interface

The HTTP client is thread-safe and can be used concurrently from multiple goroutines.

func (*HTTPClientModule) Client

func (m *HTTPClientModule) Client() *http.Client

Client returns the configured http.Client instance.

func (*HTTPClientModule) Dependencies

func (m *HTTPClientModule) Dependencies() []string

Dependencies returns the names of modules this module depends on.

func (*HTTPClientModule) Init

func (m *HTTPClientModule) Init(app modular.Application) error

Init initializes the httpclient module with the application context. This method is called after all modules have been registered and their configurations loaded. It sets up the HTTP client, transport, and logging.

The initialization process:

1. Retrieves the module's configuration
2. Sets up logging
3. Creates and configures the HTTP transport with connection pooling
4. Sets up request/response logging if verbose mode is enabled
5. Creates the HTTP client with configured transport and middleware
6. Initializes request modification pipeline

Transport configuration includes:

• Connection pooling settings for optimal performance
• Timeout configurations for reliability
• Compression and keep-alive settings
• TLS handshake timeout for secure connections

func (*HTTPClientModule) Name

func (m *HTTPClientModule) Name() string

Name returns the unique identifier for this module. This name is used for service registration, dependency resolution, and configuration section identification.

func (*HTTPClientModule) ProvidesServices

func (m *HTTPClientModule) ProvidesServices() []modular.ServiceProvider

ProvidesServices returns services provided by this module.

func (*HTTPClientModule) RegisterConfig

func (m *HTTPClientModule) RegisterConfig(app modular.Application) error

RegisterConfig registers the module's configuration structure. This method is called during application initialization to register the default configuration values for the httpclient module.

Default configuration:

• MaxIdleConns: 100 (total idle connections)
• MaxIdleConnsPerHost: 10 (idle connections per host)
• IdleConnTimeout: 90 seconds
• RequestTimeout: 30 seconds
• TLSTimeout: 10 seconds
• DisableCompression: false (compression enabled)
• DisableKeepAlives: false (keep-alives enabled)
• Verbose: false (logging disabled)

func (*HTTPClientModule) RequestModifier

func (m *HTTPClientModule) RequestModifier() RequestModifierFunc

RequestModifier returns a modifier function that can modify a request before it's sent.

func (*HTTPClientModule) RequiresServices

func (m *HTTPClientModule) RequiresServices() []modular.ServiceDependency

RequiresServices returns services required by this module.

func (*HTTPClientModule) SetRequestModifier

func (m *HTTPClientModule) SetRequestModifier(modifier RequestModifierFunc)

SetRequestModifier sets the request modifier function.

func (*HTTPClientModule) Start

func (m *HTTPClientModule) Start(context.Context) error

Start performs startup logic for the module.

func (*HTTPClientModule) Stop

func (m *HTTPClientModule) Stop(context.Context) error

Stop performs shutdown logic for the module.

func (*HTTPClientModule) WithTimeout

func (m *HTTPClientModule) WithTimeout(timeoutSeconds int) *http.Client

WithTimeout creates a new client with the specified timeout in seconds.

type RequestModifierFunc

type RequestModifierFunc func(*http.Request) *http.Request

RequestModifierFunc is a function type that can be used to modify an HTTP request before it is sent by the client.

Request modifiers are useful for:

• Adding authentication headers (Bearer tokens, API keys)
• Setting common headers (User-Agent, Content-Type)
• Adding request tracking (correlation IDs, request IDs)
• Request logging and debugging
• Request validation and sanitization

Example modifier implementations:

// API key authentication
func apiKeyModifier(apiKey string) RequestModifierFunc {
    return func(req *http.Request) *http.Request {
        req.Header.Set("Authorization", "Bearer "+apiKey)
        return req
    }
}

// Request tracing
func tracingModifier(req *http.Request) *http.Request {
    req.Header.Set("X-Request-ID", generateRequestID())
    req.Header.Set("X-Trace-ID", getTraceID(req.Context()))
    return req
}

// User agent setting
func userAgentModifier(userAgent string) RequestModifierFunc {
    return func(req *http.Request) *http.Request {
        req.Header.Set("User-Agent", userAgent)
        return req
    }
}

type VerboseOptions

type VerboseOptions struct {
	// LogHeaders enables logging of request and response headers.
	// This includes all HTTP headers sent and received, which can contain
	// sensitive information like authorization tokens.
	// Default: false
	LogHeaders bool `yaml:"log_headers" json:"log_headers" env:"LOG_HEADERS"`

	// LogBody enables logging of request and response bodies.
	// This can generate large amounts of log data and may contain sensitive
	// information. Consider using MaxBodyLogSize to limit logged content.
	// Default: false
	LogBody bool `yaml:"log_body" json:"log_body" env:"LOG_BODY"`

	// MaxBodyLogSize limits the size of logged request and response bodies.
	// Bodies larger than this size will be truncated in logs. Set to 0 for no limit.
	// Helps prevent log spam from large file uploads or downloads.
	// Default: 0 (no limit)
	MaxBodyLogSize int `yaml:"max_body_log_size" json:"max_body_log_size" env:"MAX_BODY_LOG_SIZE"`

	// LogToFile enables logging to files instead of just the application logger.
	// When enabled, HTTP logs are written to separate files for easier analysis.
	// Requires LogFilePath to be set.
	// Default: false
	LogToFile bool `yaml:"log_to_file" json:"log_to_file" env:"LOG_TO_FILE"`

	// LogFilePath is the directory where log files will be written.
	// Log files are organized by date and include request/response details.
	// The directory must be writable by the application.
	// Default: "" (current directory)
	LogFilePath string `yaml:"log_file_path" json:"log_file_path" env:"LOG_FILE_PATH"`
}

VerboseOptions configures the behavior of verbose logging. These options provide fine-grained control over HTTP request/response logging to balance debugging needs with performance and security considerations.