splitter

README

Splitter

Simple Go package for file chunk async download that made just for fun. Choose a file, slice it up, download it. Please do not set 20 chunks for a 10kb file.

Installation

Fetch package

go get github.com/AlexyAV/splitter

import (
  splitter "github.com/AlexyAV/splitter"
)

Usage

func main() {
	// With absolute destination path
	pr := splitter.NewPathResolver(
		"https://via.placeholder.com/3000",
		"/tmp/",
		&http.Client{},
	)
	pi, err := pr.PathInfo()
	if err != nil {
		log.Fatal(err)
	}

	// Create Splitter instance with new PathInfo and 10 chunks
	s := splitter.NewSplitter(context.Background(), pi, 10, &http.Client{})

	// Start file download
	err = s.Download()
	if err != nil {
		log.Fatal(err)
	}
}

Documentation

Overview

Package splitter implements functionality to download files by chunks asynchronously.

The splitter package can handle only URL (RFC 3986) as source and save destination and file or directory. It won't create any new directory but file name only in case it was not provided.

The number of chunks into which the file will be split is determined when the splitter instance is initialized.

Example

package main

import (
	"context"
	"github.com/AlexyAV/splitter"
	"log"
	"net/http"
)

func main() {
	// With absolute destination path
	pr := splitter.NewPathResolver(
		"https://via.placeholder.com/3000",
		"/tmp/",
		&http.Client{},
	)
	pi, err := pr.PathInfo()
	if err != nil {
		log.Fatal(err)
	}

	// Create Splitter instance with new PathInfo and 10 chunks
	s := splitter.NewSplitter(context.Background(), pi, 10, &http.Client{})

	// Start file download
	err = s.Download()
	if err != nil {
		log.Fatal(err)
	}
}

Index

• Variables
• type DownloadRange
  • func (dr *DownloadRange) BuildRangeHeader() string
• type HTTPClient
• type PathInfo
• type PathResolver
  • func NewPathResolver(source string, dest string, client HTTPClient) *PathResolver
  • func (pr *PathResolver) PathInfo() (*PathInfo, error)
• type PathResolverError
  • func (pr *PathResolverError) Error() string
• type RangeBuilder
  • func NewRangeBuilder(length, chunkCount, offset int) *RangeBuilder
  • func (rb *RangeBuilder) NextRange() (DownloadRange, error)
• type Source
  • func NewSource(source *url.URL, client HTTPClient) (*Source, error)
• type SourceError
  • func (pr *SourceError) Error() string
• type Splitter
  • func NewSplitter(ctx context.Context, pi *PathInfo, chunkCnt int, c HTTPClient) *Splitter
  • func (s *Splitter) Download() error
  • func (s *Splitter) Resume() error

Examples

• Package
• NewPathResolver
• RangeBuilder.NextRange

Variables

var ErrOutOfRange = errors.New("ErrOutOfRange")

ErrOutOfRange is the error returned by NextRange when no more range is available.

Types

type DownloadRange

type DownloadRange struct {
	Start, End int
}

DownloadRange is a basic data structure for storing bytes range data. Min Start value is 0 and max End value is file size.

func (*DownloadRange) BuildRangeHeader

func (dr *DownloadRange) BuildRangeHeader() string

BuildRangeHeader builds bytes range for http Range header

type HTTPClient

type HTTPClient interface {
	Do(req *http.Request) (*http.Response, error)
	Get(url string) (resp *http.Response, err error)
}

HTTPClient is the interface that wraps the basic http methods from standard http.Client.

type PathInfo

type PathInfo struct {
	Source *Source
	Dest   *os.File
}

A PathInfo is simple storage for source and destination paths.

type PathResolver

type PathResolver struct {
	Source string
	Dest   string
	// contains filtered or unexported fields
}

A PathResolver allows to resolve source path and destination path. Provides single public method to resolve both values as *url.URL and *os.File respectively.

func NewPathResolver

func NewPathResolver(source string, dest string, client HTTPClient) *PathResolver

NewPathResolver creates new PathResolver instance.

Example

package main

import (
	"fmt"
	"github.com/AlexyAV/splitter"
	"log"
	"net/http"
)

func main() {
	// With absolute destination path
	pr := splitter.NewPathResolver(
		"https://via.placeholder.com/3000",
		"/tmp/",
		&http.Client{},
	)

	// Current directory will be used as destination
	// splitter.NewPathResolver("https://picsum.photos/200", ".")

	pi, err := pr.PathInfo()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%T\n", pi.Source)
	fmt.Printf("%T\n", pi.Dest)
}

Output:

*splitter.Source
*os.File

func (*PathResolver) PathInfo

func (pr *PathResolver) PathInfo() (*PathInfo, error)

PathInfo resolves provided source and dest path and creates PathInfo instance with resolved source as *url.URL and dest as *os.File.

type PathResolverError

type PathResolverError struct {
	// contains filtered or unexported fields
}

PathResolverError represent error message and context for path resolver.

func (*PathResolverError) Error

func (pr *PathResolverError) Error() string

type RangeBuilder

type RangeBuilder struct {
	// contains filtered or unexported fields
}

A RangeBuilder allows to iterate over convent length and split it on separate DownloadRange on each iteration.

func NewRangeBuilder

func NewRangeBuilder(length, chunkCount, offset int) *RangeBuilder

NewRangeBuilder creates an instance of RangeBuilder based on total length and chunks count into which total length will be split.

func (*RangeBuilder) NextRange

func (rb *RangeBuilder) NextRange() (DownloadRange, error)

NextRange iterates over content length and creates new DownloadRange instance on each iteration. If end of range was reached ErrOutOfRange error will be returned.

Example

package main

import (
	"fmt"
	"github.com/AlexyAV/splitter"
	"log"
)

func main() {
	contentLength := 55
	chunkCount := 6
	rb := splitter.NewRangeBuilder(contentLength, chunkCount, 0)

	for {
		r, err := rb.NextRange()

		if err == splitter.ErrOutOfRange {
			break
		}

		if err != nil {
			log.Fatal(err)
		}

		fmt.Printf(
			"Start - %d; End %d; Range header - %s\n",
			r.Start,
			r.End,
			r.BuildRangeHeader(),
		)
	}
}

Output:

Start - 0; End 10; Range header - bytes=0-9
Start - 10; End 19; Range header - bytes=10-18
Start - 19; End 28; Range header - bytes=19-27
Start - 28; End 37; Range header - bytes=28-36
Start - 37; End 46; Range header - bytes=37-45
Start - 46; End 55; Range header - bytes=46-54

type Source

type Source struct {
	Path *url.URL
	Size int
	Ext  string
	// contains filtered or unexported fields
}

A Source represents an attributes of target source. These attributes will be used to split request properly. Retrieving source attributes requires additional request.

func NewSource

func NewSource(source *url.URL, client HTTPClient) (*Source, error)

NewSource creates new Source instance.

type SourceError

type SourceError struct {
	// contains filtered or unexported fields
}

SourceError represent error message and context for target source.

func (*SourceError) Error

func (pr *SourceError) Error() string

type Splitter

type Splitter struct {
	Ctx      context.Context
	PI       *PathInfo
	ChunkCnt int
	// contains filtered or unexported fields
}

Splitter allows to download source file by chunks asynchronously.

func NewSplitter

func NewSplitter(ctx context.Context, pi *PathInfo, chunkCnt int, c HTTPClient) *Splitter

NewSplitter creates new Splitter instance.

func (*Splitter) Download

func (s *Splitter) Download() error

Download initialize download process. It checks for content length and creates DownloadRange iterator. Each file's chunk will be downloaded asynchronously.

func (*Splitter) Resume

func (s *Splitter) Resume() error

Resume resumes interrupted download process. It checks for the content length of a source file and destination file respectively. Base on the current destination file new DownloadRange iterator will be created. Each file's chunk will be downloaded asynchronously.

Unlike Download it will not override existing content. If you need a clean download use Download method.