httptines

README

httptines

httptines is a powerful Go package for sending HTTP requests using proxy servers. It provides an efficient and flexible way to handle web scraping tasks with automatic proxy management, load balancing, and real-time monitoring. The solution is highly concurrent and can handle thousands of simultaneous requests.

Features

• Automatic proxy management and validation
• Smart load balancing with minimal and auto strategies
• Real-time monitoring via web interface
• User agent rotation and retry mechanism

Automatic Proxy Management

The package automatically fetches and validates proxy servers from multiple sources. It continuously monitors proxy health and performance, automatically removing failing proxies and adjusting load based on their capabilities.

Load Balancing

Two strategies are available for proxy utilization:

• Minimal Strategy: Single-threaded mode, ideal for proxies with limited concurrent connections
• Auto Strategy: Automatically determines optimal concurrent connections per proxy

Real-time Monitoring

A built-in web interface provides real-time insights into:

• Proxy performance and health
• Success/failure rates
• Request latency
• Current throughput

Installation

go get github.com/grishkovelli/httptines

Example

package main

import (
	"fmt"
	"strconv"

	"github.com/grishkovelli/httptines"
)

// handleResponse processes the HTTP response data received from the worker.
//
// Parameters:
//   - data: A byte slice containing the HTTP response body.
func handleResponse(data []byte) {
	fmt.Printf("got %d bytes\n", len(data))
}

func main() {
	// Create a slice of target URLs for testing.
	// Each target is a test endpoint with a unique ID parameter.
	targets := []string{}
	for i := range 100 {
		targets = append(targets, "https://httpstat.us/200?id="+strconv.Itoa(i+1))
	}

	// Initialize a worker with proxy sources and settings.
	worker := httptines.Worker{
		Timeout:    5,
		TestTarget: "https://httpstat.us",
		Sources: map[string][]string{
			"http": {
                "https://vakhov.github.io/fresh-proxy-list/http.txt",
                "https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt",
			},
		},
	}

	// Start processing the targets using the worker.
	// Each response is passed to the handleResponse function for processing.
	worker.Run(targets, handleResponse)
}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Documentation

Index

• type Payload
• type Server
• type Stat
  • func (s *Stat) MarshalJSON() ([]byte, error)
• type Worker
  • func (w *Worker) Run(targets []string, handler func([]byte))

Types

type Payload

type Payload struct {
	Kind string `json:"kind"` // Type of the message
	Body any    `json:"body"` // Content of the message
}

Payload represents the structure of WebSocket messages.

type Server

type Server struct {
	// URL is the proxy server's URL
	URL *url.URL `json:"url"`
	// Disabled indicates whether the server is disabled (1) or enabled (0)
	Disabled uint32 `json:"disabled"`
	// Capacity represents the maximum number of concurrent requests this server can handle
	Capacity int `json:"capacity"`
	// Latency is the last measured response time in milliseconds
	Latency int `json:"latency"`
	// Requests is the current number of active requests being processed
	Requests int `json:"requests"`
	// Positive is the count of successful requests processed by this server
	Positive int `json:"positive"`
	// Negative is the count of failed requests processed by this server
	Negative int `json:"negative"`
	// contains filtered or unexported fields
}

Server represents a proxy server with its current state and performance metrics.

type Stat

type Stat struct {
	// Targets is the total number of URLs to process
	Targets int `json:"targets"`
	// RPM represents the current requests per minute
	RPM int `json:"rpm"`
	// Servers contains a map of active proxy servers and their statistics
	Servers map[string]srvMap `json:"servers"`
	// contains filtered or unexported fields
}

Stat represents the global statistics for the application.

func (*Stat) MarshalJSON

func (s *Stat) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for Stat Returns:

• []byte: JSON representation of the statistics
• error: Any error that occurred during marshaling

type Worker

type Worker struct {
	// Interval defines the time (in seconds) between proxy downloads and health checks.
	Interval int `default:"300"`
	// Port specifies the HTTP server port for the web interface
	Port int `default:"8080"`
	// Workers determines the number of parent workers.
	// - In "minimal" strategy, it represents the maximum number of concurrent connections.
	// - In "auto" strategy, it defines the number of parent workers, while child workers
	//   are dynamically allocated based on proxy server capacity.
	//
	// Example (minimal):
	//   If Workers == 100, then max concurrent requests == 100.
	//
	// Example (auto):
	//   If Workers == 50 and each proxy server supports 100 concurrent connections,
	//   then max concurrent requests == 5000.
	Workers int `default:"100"`
	// Sources contains a map of proxy source URLs grouped by schema (http/https/socks4/socks5)
	Sources proxySrc `validate:"required"`
	// StatInterval defines the interval (in seconds) for updating statistics.
	StatInterval int `default:"2"`
	// Strategy determines the load balancing approach: "minimal" or "auto".
	//
	// - "minimal" Single-threaded mode, suitable for proxies with limited concurrency.
	// - "auto" Dynamically adjusts concurrency based on proxy capabilities.
	Strategy string `default:"minimal"`
	// Timeout specifies the request timeout in seconds
	Timeout int `default:"10"`
	// URL used for testing the connection
	TestTarget string `validate:"required"`
	// contains filtered or unexported fields
}

Worker represents a worker instance that manages proxy servers and request processing.

func (*Worker) Run

func (w *Worker) Run(targets []string, handler func([]byte))

Run initializes and starts the worker with the given targets and handler function. Parameters:

• targets: List of URLs to process
• handler: Callback function to process the response body