downloader

package
Version: v0.0.0-...-760f784 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 9, 2018 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Index

Constants

View Source
const (
	REALTIME = iota + 1
	REDIS
)
View Source
const (
	URL_KEY_PREFIX      = "LAOSJ:URLS"
	WAITTING_KEY_PREFIX = URL_KEY_PREFIX + ":WAITTING"
)
View Source
const (
	URL_CACHE_KEY = URL_KEY_PREFIX + ":DOWNLOADED" // Key for downloaded url cache
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Downloader

type Downloader interface {
	Start()
	Stop()
	WaitCloser()
}

type RealtimeDownloader

type RealtimeDownloader struct {
	// exported
	ConcurrencyLimit int                      // max number of goroutines to download
	SourceQueue      string                   // url queue
	Store            rrstorage.StorageWrapper // for saving downloaded binary
	UrlChannelFactor int
	Urls             chan Url
	// contains filtered or unexported fields
}

func (*RealtimeDownloader) Start

func (s *RealtimeDownloader) Start()

func (*RealtimeDownloader) Stop

func (s *RealtimeDownloader) Stop()

Stop RealtimeDownloader

func (*RealtimeDownloader) WaitCloser

func (s *RealtimeDownloader) WaitCloser()

Wait all urls in redis queue be downloaded

type RedisDownloader

type RedisDownloader struct {
	// exported
	ConcurrencyLimit int                      // max number of goroutines to download
	RedisConnStr     string                   // redis connection string
	SourceQueue      string                   // url queue
	Store            rrstorage.StorageWrapper // for saving downloaded binary
	UrlChannelFactor int
	// contains filtered or unexported fields
}

RedisDownloader get urls from redis SourceQueue and download them concurrently then save downloaded binary to storage

func (*RedisDownloader) Start

func (s *RedisDownloader) Start()

Start RedisDownloader

func (*RedisDownloader) Stop

func (s *RedisDownloader) Stop()

Stop RedisDownloader

func (*RedisDownloader) WaitCloser

func (s *RedisDownloader) WaitCloser()

Wait all urls in redis queue be downloaded

type Url

type Url struct {
	V      string
	Header http.Header
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL