downloader

package
v0.0.0-...-1f8a15b Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 12, 2023 License: Apache-2.0 Imports: 47 Imported by: 0

Documentation

Overview

Package filepath implements utility routines for manipulating filename paths in a way compatible with the target operating system-defined file paths.

The filepath package uses either forward slashes or backslashes, depending on the operating system. To process paths such as URLs that always use forward slashes regardless of the operating system, see the path package.

Index

Constants

View Source
const (
	Separator     = os.PathSeparator
	ListSeparator = os.PathListSeparator
)

Variables

View Source
var SkipDir error = fs.SkipDir

SkipDir is used as a return value from WalkFuncs to indicate that the directory named in the call is to be skipped. It is not returned as an error by any function.

View Source
var Trackers = [][]string{
	udpOrHttpTrackers,
}

Trackers - break down by priority tier

Functions

func AllTorrentPaths

func AllTorrentPaths(dirs datadir.Dirs) ([]string, error)

func AllTorrentSpecs

func AllTorrentSpecs(dirs datadir.Dirs) (res []*torrent.TorrentSpec, err error)

func BuildTorrentFilesIfNeed

func BuildTorrentFilesIfNeed(ctx context.Context, dirs datadir.Dirs) error

BuildTorrentFilesIfNeed - create .torrent files from .seg files (big IO) - if .seg files were added manually

func BuildTorrentIfNeed

func BuildTorrentIfNeed(ctx context.Context, fName, root string) (torrentFilePath string, err error)

func Clean

func Clean(path string) string

Clean returns the shortest path name equivalent to path by purely lexical processing. It applies the following rules iteratively until no further processing can be done:

  1. Replace multiple Separator elements with a single one.
  2. Eliminate each . path name element (the current directory).
  3. Eliminate each inner .. path name element (the parent directory) along with the non-.. element that precedes it.
  4. Eliminate .. elements that begin a rooted path: that is, replace "/.." by "/" at the beginning of a path, assuming Separator is '/'.

The returned path ends in a slash only if it represents a root directory, such as "/" on Unix or `C:\` on Windows.

Finally, any occurrences of slash are replaced by Separator.

If the result of this process is an empty string, Clean returns the string ".".

See also Rob Pike, “Lexical File Names in Plan 9 or Getting Dot-Dot Right,” https://9p.io/sys/doc/lexnames.html

func CreateMetaInfo

func CreateMetaInfo(info *metainfo.Info, mi *metainfo.MetaInfo) (*metainfo.MetaInfo, error)

func CreateTorrentFileFromInfo

func CreateTorrentFileFromInfo(root string, info *metainfo.Info, mi *metainfo.MetaInfo) (err error)

func CreateTorrentFileIfNotExists

func CreateTorrentFileIfNotExists(root string, info *metainfo.Info, mi *metainfo.MetaInfo) error

func CreateTorrentFromMetaInfo

func CreateTorrentFromMetaInfo(root string, info *metainfo.Info, mi *metainfo.MetaInfo) error

func FromSlash

func FromSlash(path string) string

FromSlash returns the result of replacing each slash ('/') character in path with a separator character. Multiple slashes are replaced by multiple separators.

func IsAbs

func IsAbs(path string) bool

IsAbs reports whether the path is absolute.

func IsLocal deprecated

func IsLocal(path string) bool

Deprecated: use `filepath.IsLocal` after drop go1.19 support

func Join

func Join(elem ...string) string

Join joins any number of path elements into a single path, separating them with an OS specific Separator. Empty elements are ignored. The result is Cleaned. However, if the argument list is empty or all its elements are empty, Join returns an empty string. On Windows, the result will only be a UNC path if the first non-empty element is a UNC path.

func NewMdbxPieceCompletion

func NewMdbxPieceCompletion(db kv.RwDB) (ret storage.PieceCompletion, err error)

func Proto2InfoHash

func Proto2InfoHash(in *prototypes.H160) metainfo.Hash

func VerifyFile

func VerifyFile(ctx context.Context, t *torrent.Torrent, completePieces *atomic.Uint64) error

Types

type AggStats

type AggStats struct {
	MetadataReady, FilesTotal int32
	PeersUnique               int32
	ConnectionsTotal          uint64

	Completed bool
	Progress  float32

	BytesCompleted, BytesTotal     uint64
	DroppedCompleted, DroppedTotal uint64

	BytesDownload, BytesUpload uint64
	UploadRate, DownloadRate   uint64
}

type Downloader

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader - component which downloading historical files. Can use BitTorrent, or other protocols

func New

func New(ctx context.Context, cfg *downloadercfg.Cfg, dirs datadir.Dirs, logger log.Logger, verbosity log.Lvl, discover bool) (*Downloader, error)
func (d *Downloader) AddInfoHashAsMagnetLink(ctx context.Context, infoHash metainfo.Hash, name string) error

func (*Downloader) AddNewSeedableFile

func (d *Downloader) AddNewSeedableFile(ctx context.Context, name string) error

AddNewSeedableFile decides what we do depending on wether we have the .seg file or the .torrent file have .torrent no .seg => get .seg file from .torrent have .seg no .torrent => get .torrent from .seg

func (*Downloader) BuildTorrentFilesIfNeed

func (d *Downloader) BuildTorrentFilesIfNeed(ctx context.Context) error

func (*Downloader) Close

func (d *Downloader) Close()

func (*Downloader) MainLoopInBackground

func (d *Downloader) MainLoopInBackground(silent bool)

func (*Downloader) PeerID

func (d *Downloader) PeerID() []byte

func (*Downloader) ReCalcStats

func (d *Downloader) ReCalcStats(interval time.Duration)

func (*Downloader) SnapDir

func (d *Downloader) SnapDir() string

func (*Downloader) Stats

func (d *Downloader) Stats() AggStats

func (*Downloader) StopSeeding

func (d *Downloader) StopSeeding(hash metainfo.Hash) error

func (*Downloader) TorrentClient

func (d *Downloader) TorrentClient() *torrent.Client

func (*Downloader) VerifyData

func (d *Downloader) VerifyData(ctx context.Context, onlyFiles []string) error

type GrpcServer

type GrpcServer struct {
	proto_downloader.UnimplementedDownloaderServer
	// contains filtered or unexported fields
}

func NewGrpcServer

func NewGrpcServer(d *Downloader) (*GrpcServer, error)

func (*GrpcServer) Delete

Delete - stop seeding, remove file, remove .torrent

func (*GrpcServer) Download

Download - create new .torrent ONLY if initialSync, everything else Erigon can generate by itself

func (*GrpcServer) Stats

func (*GrpcServer) Verify

type WalkFunc

type WalkFunc func(path string, info fs.FileInfo, err error) error

WalkFunc is the type of the function called by Walk to visit each file or directory.

The path argument contains the argument to Walk as a prefix. That is, if Walk is called with root argument "dir" and finds a file named "a" in that directory, the walk function will be called with argument "dir/a".

The directory and file are joined with Join, which may clean the directory name: if Walk is called with the root argument "x/../dir" and finds a file named "a" in that directory, the walk function will be called with argument "dir/a", not "x/../dir/a".

The info argument is the fs.FileInfo for the named path.

The error result returned by the function controls how Walk continues. If the function returns the special value SkipDir, Walk skips the current directory (path if info.IsDir() is true, otherwise path's parent directory). If the function returns the special value SkipAll, Walk skips all remaining files and directories. Otherwise, if the function returns a non-nil error, Walk stops entirely and returns that error.

The err argument reports an error related to path, signaling that Walk will not walk into that directory. The function can decide how to handle that error; as described earlier, returning the error will cause Walk to stop walking the entire tree.

Walk calls the function with a non-nil err argument in two cases.

First, if an os.Lstat on the root directory or any directory or file in the tree fails, Walk calls the function with path set to that directory or file's path, info set to nil, and err set to the error from os.Lstat.

Second, if a directory's Readdirnames method fails, Walk calls the function with path set to the directory's path, info, set to an fs.FileInfo describing the directory, and err set to the error from Readdirnames.

type WebSeeds

type WebSeeds struct {
	// contains filtered or unexported fields
}

WebSeeds - allow use HTTP-based infrastrucutre to support Bittorrent network it allows download .torrent files and data files from trusted url's (for example: S3 signed url)

func (*WebSeeds) ByFileName

func (d *WebSeeds) ByFileName(name string) (metainfo.UrlList, bool)

func (*WebSeeds) Discover

func (d *WebSeeds) Discover(ctx context.Context, s3tokens []string, urls []*url.URL, files []string, rootDir string)

func (*WebSeeds) Len

func (d *WebSeeds) Len() int

func (*WebSeeds) TorrentUrls

func (d *WebSeeds) TorrentUrls() snaptype.TorrentUrls

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL