HttpClientPool

README

HttpClientPool

Easy orchestration of an HTTP client pool with built-in rate-limiting capabilities.

---

Disclaimer

This was made purely for personal use. While everything seems to work in my use cases and passes tests unexpected behavior may occur.

Overview

This package facilitates efficient concurrent HTTP requests by managing a pool of individual HTTP clients, each with its configuration and the ability to apply both per-client and global rate limits.

Usage

To use this package, create a ClientPool using the NewClientPool function, specifying the desired client delay, pool delay, optional proxies, and user-agent weights. For simple requests, use ClientPool.QuickRequest() with a RequestData bundle. This automatically marks it as active and deactivates it when the request is complete.

For greater flexibility, use ClientPool.GetClient() to get an available Client instance and use it as with a normal http.Client instance. Call Client.SetInactive() when done with the client to deactivate it.

Example

proxies := Utils.UrlsFromFile("proxies.txt")
clientDelay := time.Millisecond * 500//  Two requests per second
poolDelay := Utils.RpsToDuration(25)//25 requests per second
pool := HttpClientPool.NewClientPool(clientDelay, poolDelay, proxies, nil)
// Fetch first 1000 pages
for i:=0;i<1000;i++ {
	request := RequestData{
		Type: "GET",
		Url:  "http://api.com/getByIndex",
		Params: map[string][]string{
			"Index": {i},
		},
		Headers: map[string][]string{
			"Content-Type": {"application/json"},
		},
		Cookies: map[string]string{
			"Api-Key": "5318008",
		},
	}
	responseData, err := client.QuickRequest(request)
	if err != nil {
		t.Fatal(err, string(responseData.Body))
	}
    // Do something with responseData
}

Documentation

Overview

Package HttpClientPool allows easy orchestration of an HTTP client pool with built-in rate-limiting capabilities.

Overview:

This package enables efficient concurrent HTTP requests by managing a pool of individual HTTP clients,
each with its own configuration and the ability to apply both per-client and global rate limits.

Usage:

To use this package, create a ClientPool using the NewClientPool function, specifying the desired
client delay, pool delay, optional proxies, and user-agent weights.
To make simple requests call ClientPool.QuickRequest() with a RequestData bundle. This will
automatically mark it as active and deactivate it when the request is complete.

For greater flexibility call ClientPool.GetClient() to get an available Client instance and
and use it as with a normal http.Client instance. Call Client.SetInactive() when done with
the Client to deactivate it.

Example:

clientPool := HttpClientPool.NewClientPool(time.Millisecond*100, time.Second, nil, nil)

Features:

• Dynamic client pool creation with customizable delays.
• Rate-limiting for individual clients and the entire pool.
• Automatic proxy rotation by ratelimit.

GitHub repository: https://github.com/RootInit/HttpClientPool

Index

• type Client
  • func NewClient(proxy *url.URL, userAgent string, delay time.Duration) *Client
  • func (client *Client) GetDelay() time.Duration
  • func (client *Client) GetRequestTime() time.Time
  • func (client *Client) GetUserAgent() string
  • func (client *Client) IsAvailable() bool
  • func (client *Client) IsRunning() bool
  • func (client *Client) QuickRequest(reqData RequestData) (ResponseData, error)
  • func (client *Client) SetActive()
  • func (client *Client) SetDelay(delay time.Duration)
  • func (client *Client) SetInactive()
  • func (client *Client) SetUserAgent(userAgent string)
• type ClientPool
  • func NewClientPool(clientDelay, poolDelay time.Duration, proxies []*url.URL, ...) ClientPool
  • func (pool *ClientPool) AddClient(client *Client)
  • func (pool *ClientPool) Done()
  • func (pool *ClientPool) GetClient() *Client
  • func (pool *ClientPool) QuickRequest(reqData RequestData) (ResponseData, error)
  • func (pool *ClientPool) RemmoveClient(client *Client)
  • func (pool *ClientPool) SetClientDelay(clientDelay time.Duration)
  • func (pool *ClientPool) SetPoolDelay(poolDelay time.Duration)
• type RequestData
• type ResponseData

Types

type Client

type Client struct {
	// Client is the underlying HTTP client for making requests.
	*http.Client
	// contains filtered or unexported fields
}

Client represents an HTTP client with inbuilt ratelimiting.

This type can be used along with a ClientPool for orchestration of multiple clients.

func NewClient

func NewClient(proxy *url.URL, userAgent string, delay time.Duration) *Client

NewClient creates a new HTTP client with optional proxy, user agent, and request delay.

Parameters:

• proxy (*url.URL): The proxy URL to be used for the client. Use nil for no proxy.
• userAgent (string): The user agent string to be set in the client's requests.
• delay (time.Duration): The delay between requests made by the client. Use 0 for no delay.

Returns:

• *Client: A pointer to the initialized HTTP client.

func (*Client) GetDelay

func (client *Client) GetDelay() time.Duration

GetDelay returns the clients delay

Returns:

• time.Duration: The duration of the clients delay

func (*Client) GetRequestTime

func (client *Client) GetRequestTime() time.Time

GetRequestTime returns the last request time

Returns:

• time.Time: the client.lastReqTime value

func (*Client) GetUserAgent

func (client *Client) GetUserAgent() string

GetUserAgent returns the Clients user-agent

Returns:

• string: the client.userAgent value

func (*Client) IsAvailable

func (client *Client) IsAvailable() bool

IsAvailable returns true if the client is not currently running or rate-limited.

This method is used to check if the client is in an available state for new requests.

Returns:

• bool: True if the client is available; otherwise, false.

func (*Client) IsRunning

func (client *Client) IsRunning() bool

IsRunning returns true if the client is currently running.

This method is used to check if the client is actively processing requests.

Returns:

• bool: True if the client is running; otherwise, false.

func (*Client) QuickRequest

func (client *Client) QuickRequest(reqData RequestData) (ResponseData, error)

QuickRequest is a convenience wrapper arround http.Request allowing easy basic requests. Performs an HTTP request with various options and returns the response.

It allows making HTTP requests with different methods (GET, POST, etc.) and supports request options such as URL parameters, headers, user agent, cookies, and various payload types including JSON, form, and other data using the RawData field.

Parameters:

• reqData (RequestData): The RequestData struct containing HTTP request data.

Returns:

• ResponseData: A ResponseData struct containing HTTP response data.
• error: An error, if any, encountered during the HTTP request.

func (*Client) SetActive

func (client *Client) SetActive()

SetActive marks the HTTP client as active and updates the lastReqTime.

func (*Client) SetDelay

func (client *Client) SetDelay(delay time.Duration)

SetDelay ets the clients delay

Parameters:

• delay (time.Duration): The duration of the new delay

func (*Client) SetInactive

func (client *Client) SetInactive()

SetInactive marks the HTTP client as inactive.

func (*Client) SetUserAgent

func (client *Client) SetUserAgent(userAgent string)

GetUserAgent sets the Clients user-agent

Parameters:

• userAgent (string): the userAgent to set

type ClientPool

type ClientPool struct {
	// Clients is a slice containing pointers to the clients in the pool.
	Clients []*Client
	// contains filtered or unexported fields
}

ClientPool represents a pool of HTTP clients with easy per client and whole pool ratelimiting.

The pool is responsible for managing a collection of HTTP clients, each with its own configuration, and a shared delay applied between requests made by clients.

func NewClientPool

func NewClientPool(clientDelay, poolDelay time.Duration, proxies []*url.URL, userAgents map[string]float32) ClientPool

NewClientPool creates a pool of HTTP clients for concurrent requests.

Parameters:

• clientDelay (time.Duration): Time duration between client requests. Use 0 for no delay.
• poolDelay (time.Duration): Time duration between client pool requests. Use 0 for no delay.
• proxies ([]*url.URL): List of proxy URLs. Use nil for a single client with no proxy.
• userAgents (map[string]float32): Map of user agents with their respective weights.

Returns:

• ClientPool: The initialized client pool.

func (*ClientPool) AddClient

func (pool *ClientPool) AddClient(client *Client)

AddClient adds a new HTTP client to the client pool.

This function will accept duplicate clients and add them.

Parameters:

• client (*Client): The HTTP client to be added to the pool.

func (*ClientPool) Done

func (pool *ClientPool) Done()

Done blocks until all clients in the pool are inactive.

This method ensures that all active clients finish their ongoing requests before allowing the program to proceed.

func (*ClientPool) GetClient

func (pool *ClientPool) GetClient() *Client

GetClient returns an available HTTP client from the pool. The client is set as active and the lastReqTime is set to time.Now.

This method blocks until a client becomes available in the pool.

Returns:

• *Client: A pointer to the available HTTP client.

func (*ClientPool) QuickRequest

func (pool *ClientPool) QuickRequest(reqData RequestData) (ResponseData, error)

QuickRequest is a convenience function which fetches a Client with pool.GetClient and passes the RequestData to client.QuickRequest

Parameters:

• reqData (RequestData): The RequestData struct containing HTTP request data.

Returns:

• ResponseData: A ResponseData struct containing HTTP response data.
• error: An error, if any, encountered during the HTTP request.

func (*ClientPool) RemmoveClient

func (pool *ClientPool) RemmoveClient(client *Client)

RemoveClient removes a specific HTTP client from the client pool.

If the client is not in the pool the pool remains unchanged. If the client is duplicated in the pool only the first instance of the client will be removed.

Parameters:

• client (*Client): The HTTP client to be removed from the pool.

func (*ClientPool) SetClientDelay

func (pool *ClientPool) SetClientDelay(clientDelay time.Duration)

SetClientDelay sets the individual delay between requests for each client in the pool.

Parameters:

• clientDelay (time.Duration): The new shared delay. Use 0 for no delay.

func (*ClientPool) SetPoolDelay

func (pool *ClientPool) SetPoolDelay(poolDelay time.Duration)

SetPoolDelay sets the minimum delay between requests from all clients in the pool.

Parameters:

• poolDelay (time.Duration): The new shared delay. Use 0 for no delay.

type RequestData

type RequestData struct {
	// Type specifies the HTTP request method (e.g., GET, POST).
	Type string

	// Url is the URL of the HTTP request.
	Url string

	// Params contains the url parameters for the request. Key:Array of values
	Params map[string][]string

	// JsonData accepts any type with data to be sent in the request body as JSON.
	//
	// Cannot be used with FormData, FormFiles or RawData
	JsonData interface{}

	// FormData contains the data to be sent in the request body as form data.
	//
	// Cannot be used with JsonData or RawData
	FormData map[string]string
	// Files contains the files to be included as part of FormData.
	//
	// Cannot be used with JsonData or RawData
	FormFiles map[string]*os.File

	// RawData contains the raw request body as an io.Reader.
	//
	// Overrides both JsonData and FormData/FormFiles
	RawData *io.Reader

	// Headers contains the HTTP headers for the request. Key:Array of values
	Headers map[string][]string

	// Cookies contains the cookies to be included in the request.
	Cookies map[string]string
}

RequestData represents request data to be passed to QuickRequest

type ResponseData

type ResponseData struct {
	// Status is the human-readable status message of the HTTP response.
	Status string

	// StatusCode is the HTTP status code of the response.
	StatusCode int

	// Body contains the raw body of the HTTP response.
	Body []byte

	// Cookies contains the cookies received in the HTTP response.
	Cookies map[string]string
}

ResponseData represents data from an http.Response returned by QuickRequest