rangetripper

README

rangetripper

import "github.com/cognusion/go-rangetripper"

• Overview
• Index

Overview

Package rangetripper provides a performant http.RoundTripper that handles byte-range downloads if the resulting HTTP server claims to support them in a HEAD request for the file. RangeTripper will download 1/Nth of the file asynchronously with each of the fileChunks specified in a New. N+1 actual downloaders are most likely as the +1 covers any gap from non-even division of content-length.

Index

• Constants
• type Client
• type RangeTripper
  • func New(fileChunks int, outputFilePath string) (*RangeTripper, error)
  • func NewWithLoggers(fileChunks int, outputFilePath string, timingLogger, debugLogger *log.Logger) (*RangeTripper, error)
  • func (rt *RangeTripper) Do(r *http.Request) (*http.Response, error)
  • func (rt *RangeTripper) RoundTrip(r *http.Request) (*http.Response, error)
  • func (rt *RangeTripper) SetChunkSize(chunkBytes int64)
  • func (rt *RangeTripper) SetClient(client Client)
  • func (rt *RangeTripper) SetMax(max int)
  • func (rt *RangeTripper) WithProgress() <-chan int64
• type RetryClient
  • func NewRetryClient(retries int, every, timeout time.Duration) *RetryClient
  • func NewRetryClientWithExponentialBackoff(retries int, initially, timeout time.Duration) *RetryClient
  • func (w *RetryClient) Do(req *http.Request) (*http.Response, error)

Package files

client.go retryclient.go rt.go

Constants

const (
    ContentLengthNumericError   = rtError("Content-Length value cannot be converted to a number")
    ContentLengthMismatchError  = rtError("downloaded file size does not match content-length")
    SingleRequestExhaustedError = rtError("one request has already been made with this RangeTripper")
)

Static errors to return

type Client

type Client interface {
    Do(*http.Request) (*http.Response, error)
}

Client is an interface that could refer to an http.Client or a rangetripper.RetryClient

var DefaultClient Client = NewRetryClient(10, 2*time.Second, 60*time.Second)

DefaultClient is what RangeTripper will use to actually make the individual GET requests. Change the values to change the outcome. Don't set the DefaultClient's Client.Transport to a RangeTripper, or :mindblown:. DefaultClient can be a lowly http.Client if you prefer

type RangeTripper

type RangeTripper struct {
    TimingsOut *log.Logger
    DebugOut   *log.Logger
    // contains filtered or unexported fields
}

RangeTripper is an http.RoundTripper to be used in an http.Client. This should not be used in its default state, instead by its New functions. A single RangeTripper must only be used for one request.

func New

func New(fileChunks int, outputFilePath string) (*RangeTripper, error)

New simply returns a RangeTripper or an error. Logged messages are discarded.

func NewWithLoggers

func NewWithLoggers(fileChunks int, outputFilePath string, timingLogger, debugLogger *log.Logger) (*RangeTripper, error)

NewWithLoggers returns a RangeTripper or an error. Logged messages are sent to the specified Logger, or discarded if nil.

func (*RangeTripper) Do

func (rt *RangeTripper) Do(r *http.Request) (*http.Response, error)

Do is a satisfier of the rangetripper.Client interface, and is identical to RoundTrip

func (*RangeTripper) RoundTrip

func (rt *RangeTripper) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip is called with a formed Request, writing the Body of the Response to to the specified output file. The Response should be ignored, but errors are important. Both the Request.Body and the RangeTripper.outFile will be closed when this function returns.

func (*RangeTripper) SetChunkSize

func (rt *RangeTripper) SetChunkSize(chunkBytes int64)

SetChunkSize overrides the fileChunks and instead will divide the resulting Content-Length by this to determine the appropriate chunk count dynamically. fileChunks will still be used to guide the maximum number of concurrent workers, unless SetMax() is used.

func (*RangeTripper) SetClient

func (rt *RangeTripper) SetClient(client Client)

SetClient allows for overriding the Client used to make the requests.

func (*RangeTripper) SetMax

func (rt *RangeTripper) SetMax(max int)

SetMax allows for setting the maximum number of concurrently-running workers

func (*RangeTripper) WithProgress

func (rt *RangeTripper) WithProgress() <-chan int64

WithProgress returns a read-only chan that will first provide the total length of the content (in bytes), followed by a stream of completed byte-lengths. CAUTION: It is a generally bad idea to call this and then ignore the resulting channel.

type RetryClient

type RetryClient struct {
    // contains filtered or unexported fields
}

RetryClient contains variables and methods to use when making smarter HTTP requests

func NewRetryClient

func NewRetryClient(retries int, every, timeout time.Duration) *RetryClient

NewRetryClient returns a RetryClient that will retry failed requests retries times, every every, and use timeout as a timeout

func NewRetryClientWithExponentialBackoff

func NewRetryClientWithExponentialBackoff(retries int, initially, timeout time.Duration) *RetryClient

NewRetryClientWithExponentialBackoff returns a RetryClient that will retry failed requests retries times, first after initially and exponentially longer each time, and use timeout as a timeout

func (*RetryClient) Do

func (w *RetryClient) Do(req *http.Request) (*http.Response, error)

Do takes a Request, and returns a Response or an error, following the rules of the RetryClient

---

Generated by godoc2md

Documentation

Overview

Package rangetripper provides a performant http.RoundTripper that handles byte-range downloads if the resulting HTTP server claims to support them in a HEAD request for the file. RangeTripper will download 1/Nth of the file asynchronously with each of the “fileChunks“ specified in a New. N+1 actual downloaders are most likely as the +1 covers any gap from non-even division of content-length.

Index

• Constants
• type Client
• type RangeTripper
  • func New(fileChunks int, outputFilePath string) (*RangeTripper, error)
  • func NewWithLoggers(fileChunks int, outputFilePath string, timingLogger, debugLogger *log.Logger) (*RangeTripper, error)
  • func (rt *RangeTripper) Do(r *http.Request) (*http.Response, error)
  • func (rt *RangeTripper) RoundTrip(r *http.Request) (*http.Response, error)
  • func (rt *RangeTripper) SetChunkSize(chunkBytes int64)
  • func (rt *RangeTripper) SetClient(client Client)
  • func (rt *RangeTripper) SetMax(max int)
  • func (rt *RangeTripper) WithProgress() <-chan int64
• type RetryClient
  • func NewRetryClient(retries int, every, timeout time.Duration) *RetryClient
  • func NewRetryClientWithExponentialBackoff(retries int, initially, timeout time.Duration) *RetryClient
  • func (w *RetryClient) Do(req *http.Request) (*http.Response, error)

Examples

• RangeTripper

Constants

const (
	ContentLengthNumericError   = rtError("Content-Length value cannot be converted to a number")
	ContentLengthMismatchError  = rtError("downloaded file size does not match content-length")
	SingleRequestExhaustedError = rtError("one request has already been made with this RangeTripper")
)

Static errors to return

Types

type Client

type Client interface {
	Do(*http.Request) (*http.Response, error)
}

Client is an interface that could refer to an http.Client or a rangetripper.RetryClient

var DefaultClient Client = NewRetryClient(10, 2*time.Second, 60*time.Second)

DefaultClient is what RangeTripper will use to actually make the individual GET requests. Change the values to change the outcome. Don't set the DefaultClient's Client.Transport to a RangeTripper, or :mindblown:. DefaultClient can be a lowly http.Client if you prefer

type RangeTripper

type RangeTripper struct {
	TimingsOut *log.Logger
	DebugOut   *log.Logger
	// contains filtered or unexported fields
}

RangeTripper is an http.RoundTripper to be used in an http.Client. This should not be used in its default state, instead by its New functions. A single RangeTripper *must* only be used for one request.

Example

// Set up a temporary file
tfile, err := os.CreateTemp("/tmp", "rt")
if err != nil {
	panic(err)
}
defer os.Remove(tfile.Name()) // clean up after ourselves

client := new(http.Client)     // make a new Client
rt, _ := New(10, tfile.Name()) // make a new RangeTripper (errors ignored for brevity. Don't be dumb)
client.Transport = rt          // Use the RangeTripper as the Transport

if _, err := client.Get("https://google.com/"); err != nil {
	panic(err)
}
// tfile is the google homepage

func New

func New(fileChunks int, outputFilePath string) (*RangeTripper, error)

New simply returns a RangeTripper or an error. Logged messages are discarded.

func NewWithLoggers

func NewWithLoggers(fileChunks int, outputFilePath string, timingLogger, debugLogger *log.Logger) (*RangeTripper, error)

NewWithLoggers returns a RangeTripper or an error. Logged messages are sent to the specified Logger, or discarded if nil.

func (*RangeTripper) Do

func (rt *RangeTripper) Do(r *http.Request) (*http.Response, error)

Do is a satisfier of the rangetripper.Client interface, and is identical to RoundTrip

func (*RangeTripper) RoundTrip

func (rt *RangeTripper) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip is called with a formed Request, writing the Body of the Response to to the specified output file. The Response should be ignored, but errors are important. Both the Request.Body and the RangeTripper.outFile will be closed when this function returns.

func (*RangeTripper) SetChunkSize

func (rt *RangeTripper) SetChunkSize(chunkBytes int64)

SetChunkSize overrides the “fileChunks“ and instead will divide the resulting Content-Length by this to determine the appropriate chunk count dynamically. “fileChunks“ will still be used to guide the maximum number of concurrent workers, unless “SetMax()“ is used.

func (*RangeTripper) SetClient

func (rt *RangeTripper) SetClient(client Client)

SetClient allows for overriding the Client used to make the requests.

func (*RangeTripper) SetMax

func (rt *RangeTripper) SetMax(max int)

SetMax allows for setting the maximum number of concurrently-running workers

func (*RangeTripper) WithProgress

func (rt *RangeTripper) WithProgress() <-chan int64

WithProgress returns a read-only chan that will first provide the total length of the content (in bytes), followed by a stream of completed byte-lengths. CAUTION: It is a generally bad idea to call this and then ignore the resulting channel.

type RetryClient

type RetryClient struct {
	// contains filtered or unexported fields
}

RetryClient contains variables and methods to use when making smarter HTTP requests

func NewRetryClient

func NewRetryClient(retries int, every, timeout time.Duration) *RetryClient

NewRetryClient returns a RetryClient that will retry failed requests “retries“ times, every “every“, and use “timeout“ as a timeout

func NewRetryClientWithExponentialBackoff

func NewRetryClientWithExponentialBackoff(retries int, initially, timeout time.Duration) *RetryClient

NewRetryClientWithExponentialBackoff returns a RetryClient that will retry failed requests “retries“ times, first after “initially“ and exponentially longer each time, and use “timeout“ as a timeout

func (*RetryClient) Do

func (w *RetryClient) Do(req *http.Request) (*http.Response, error)

Do takes a Request, and returns a Response or an error, following the rules of the RetryClient