roundtripper

README

Proxy Roulette

This is a toy example of an implementation of the roulette-wheel selection via stochastic acceptance in Golang and http.RoundTripper for proxy selection. In other words it selects a random proxy from a set in O(1) time but also adds a rank function for increase/decrease probability depending on the proxy state.

Example

roulette := &roundtripper.ProxyRoulette{
	Proxies: proxies,
	Step:    15,
}

rt := &roundtripper.ProxyRoundTripper{
	ProxySelector: roulette,
	Tr: &http.Transport{
		DialContext: (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
		}).DialContext,
		TLSHandshakeTimeout: 10 * time.Second,
		DisableKeepAlives:   false,
		MaxIdleConnsPerHost: 1,
	},
}

var client = &http.Client{
	Transport: rt,
	Timeout:   30 * time.Second,
}

for i := 0; i < 50; i++ {
	req, _ := http.NewRequest("GET", "https://remote.site", nil)
	resp, err := client.Do(req)
	if err != nil {
		continue
	}

	if _, err := ioutil.ReadAll(resp.Body); err != nil {
		panic(err)
	}
}

Documentation

Index

• Variables
• type Proxy
• type ProxyRoulette
  • func (pr *ProxyRoulette) Select() (*Proxy, error)
  • func (pr *ProxyRoulette) WeightDown(proxy *Proxy)
  • func (pr *ProxyRoulette) WeightUp(proxy *Proxy)
• type ProxyRoundTripper
  • func (p *ProxyRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)
• type ProxySelector

Variables

var Result = make(map[int]int)

Types

type Proxy

type Proxy struct {
	Id       int
	Username string
	Password string
	Scheme   string
	Host     string
	Port     int
	Weight   float32
}

A proxy representation

type ProxyRoulette

type ProxyRoulette struct {
	Step    float32
	Proxies []*Proxy
	// contains filtered or unexported fields
}

func (*ProxyRoulette) Select

func (pr *ProxyRoulette) Select() (*Proxy, error)

Roulette-wheel selection via stochastic acceptance

func (*ProxyRoulette) WeightDown

func (pr *ProxyRoulette) WeightDown(proxy *Proxy)

floor clipping

func (*ProxyRoulette) WeightUp

func (pr *ProxyRoulette) WeightUp(proxy *Proxy)

ceil clipping

type ProxyRoundTripper

type ProxyRoundTripper struct {
	ProxySelector

	Tr *http.Transport
}

The proxy round tripper

func (*ProxyRoundTripper) RoundTrip

func (p *ProxyRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)

implements the http.RoundTripper interface

type ProxySelector

type ProxySelector interface {
	Select() (*Proxy, error)
	WeightDown(*Proxy)
	WeightUp(*Proxy)
}

ProxySelector is the interface that should be implemented by the algorithm behind the round tripper