gofetch

package module
Version: v0.0.0-...-42e5155 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 4, 2017 License: MPL-2.0 Imports: 17 Imported by: 0

README

Gofetch

Build Status GoDoc

Go library to download files from the internerds using Go 1.7 or greater.

Features

  • Resumes downloads if interrupted.
  • Allows parallel downloading of a single file by requesting multiple data chunks at once over HTTP.
  • Reports download progress through a Go channel if indicated to do so.
  • Supports file integrity verification if a checksum is provided.
  • Supports ETags, skipping downloading a file if it hasn't changed on the server.
  • Can be combined with https://github.com/cenkalti/backoff to support retrying with exponential back-off

Gotchas

When downloading file chunks concurrently, you may encounter some issues:

  • Servers may limit the number of concurrent connections you have open against them
  • Servers may accept the connections but will not send anything, in this case the default HTTP client will timeout for you. If you provide your own client, make sure it has proper timeouts or your connection will block for several seconds, depending on your operating system network timeouts.

Example

package main

import (
	"fmt"
	"io"
	"os"

	"github.com/c4milo/gofetch"
)

func main() {
	gf := gofetch.New(
		gofetch.WithDestDir(os.TempDir()),
		gofetch.WithConcurrency(10),
		gofetch.WithETag(),
	)

	progressCh := make(chan gofetch.ProgressReport)

	var myFile *os.File
	go func() {
		var err error
		myFile, err = gf.Fetch(
			"http://releases.ubuntu.com/15.10/ubuntu-15.10-server-amd64.iso",
			progressCh)
		if err != nil {
			panic(err)
		}
	}()

	// pogressCh is close by gofetch once a download finishes
	var totalWritten int64
	for p := range progressCh {
		// p.WrittenBytes does not accumulate, it represents the chunk size written
		// in the current operation.
		totalWritten += p.WrittenBytes
		fmt.Printf("%d of %d\n", totalWritten, p.Total)
	}

	destFile, err := os.Create("/tmp/ubuntu-15.10-server-amd64.iso")
	if err != nil {
		panic(err)
	}

	defer func() {
		if err := destFile.Close(); err != nil {
			panic(err)
		}
	}()

	if _, err := io.Copy(destFile, myFile); err != nil {
		panic(err)
	}
}

Documentation

Overview

Example
package main

import (
	"fmt"
	"os"

	"github.com/c4milo/gofetch"
)

func main() {
	gf := gofetch.New(
		gofetch.WithDestDir("/tmp"),
		gofetch.WithConcurrency(100),
		gofetch.WithETag(),
		gofetch.WithChecksum("sha256", "29a8b9009509b39d542ecb229787cdf48f05e739a932289de9e9858d7c487c80"),
	)

	progressCh := make(chan gofetch.ProgressReport)
	doneCh := make(chan struct{})

	var myFile *os.File
	go func() {
		defer close(doneCh)

		var err error
		myFile, err = gf.Fetch(
			"http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso",
			progressCh)
		if err != nil {
			panic(err)
		}
	}()

	// pogressCh is close by gofetch once a download finishes
	var totalWritten int64
	for p := range progressCh {
		// p.WrittenBytes does not accumulate, it represents the chunk size written
		// in the current operation.
		totalWritten += p.WrittenBytes
		fmt.Printf("\r%d of %d bytes", totalWritten, p.Total)
	}

	<-doneCh
	fmt.Printf("\nFile saved at %q\n", myFile.Name())
}
Output:

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher represents an instance of gofetch, holding global configuration options.

func New

func New(opts ...Option) *Fetcher

New creates a new instance of goFetch with the given options.

func (*Fetcher) Fetch

func (gf *Fetcher) Fetch(url string, progressCh chan<- ProgressReport) (*os.File, error)

Fetch downloads content from the provided URL. It supports resuming and parallelizing downloads while being very memory efficient.

type Option

type Option func(*Fetcher)

Option as explained in http://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html

func WithChecksum

func WithChecksum(alg, value string) Option

WithChecksum verifies the file once it is fully downloaded using the provided hash and expected value.

func WithConcurrency

func WithConcurrency(c int) Option

WithConcurrency allows you to set the number of goroutines used to download a specific file. By default it is set to 1.

func WithDestDir

func WithDestDir(dir string) Option

WithDestDir allows you to set the destination directory for the downloaded files. By default it is set to: ./

func WithETag

func WithETag() Option

WithETag enables ETag support, meaning that if an already downloaded file is currently on disk and matches the ETag value returned by the server, it will not be downloaded again. By default it is set to false. Be aware that different servers, serving the same file, are likely to return different ETag values, causing the file to be re-downloaded, even though it might already exist on disk.

func WithHTTPClient

func WithHTTPClient(c *http.Client) Option

WithHTTPClient allows to provide a custom HTTP client. By default a HTTP client with support for read/write timeouts is used.

type ProgressReport

type ProgressReport struct {
	// Total length in bytes of the file being downloaded
	Total int64
	// Written bytes to disk on a write by write basis. It does not accumulate.
	WrittenBytes int64
}

ProgressReport represents the current download progress of a given file

Source Files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL