downloaders

package
v0.0.0-...-5224e43 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 12, 2026 License: GPL-3.0 Imports: 7 Imported by: 0

Documentation

Overview

Package downloaders provides a pluggable interface for downloading datasets from various scientific data repositories with comprehensive metadata tracking and provenance information.

Package downloaders provides registry functionality for managing different downloader implementations and mapping source types to their handlers.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidID         = &DownloaderError{Type: "invalid_id", Message: "invalid identifier format"}
	ErrNotFound          = &DownloaderError{Type: "not_found", Message: "dataset not found"}
	ErrAccessDenied      = &DownloaderError{Type: "access_denied", Message: "access denied"}
	ErrNetworkError      = &DownloaderError{Type: "network_error", Message: "network error"}
	ErrInsufficientSpace = &DownloaderError{Type: "insufficient_space", Message: "insufficient disk space"}
	ErrUnsupportedType   = &DownloaderError{Type: "unsupported_type", Message: "unsupported dataset type"}
)

Common error types.

View Source 
var DefaultRegistry = NewRegistry()

DefaultRegistry is the global registry instance.

Functions

func List 

func List() []string

List returns all source types from the default registry.

func Register 

func Register(downloader Downloader) error

Register adds a downloader to the default registry.

func RegisterAlias 

func RegisterAlias(alias, sourceType string) error

RegisterAlias creates an alias in the default registry.

func ShouldDownload 

func ShouldDownload(filename string, size int64, opts *DownloadOptions) bool

ShouldDownload reports whether a file should be downloaded given its name, size (bytes), and the active options. It evaluates filters in this order:

  1. IncludeRaw / ExcludeSupplementary coarse flags
  2. IncludeExts / ExcludeExts extension lists
  3. FilenameGlob pattern
  4. MaxFileSize limit
  5. Legacy CustomFilters map (kept for backward compatibility)

size may be -1 when the caller does not know the file size ahead of time; MaxFileSize and size-based CustomFilters are then skipped.

func Speed 

func Speed(bytes int64, d time.Duration) float64

Speed returns bytes/second, returning 0 instead of +Inf when duration is zero.

Types

type Action 

type Action int

Action represents user choices for conflict resolution. 

const (
	ActionProceed Action = iota
	ActionSkip
	ActionMerge
	ActionOverwrite
	ActionAbort
)

func (Action) String 

func (a Action) String() string

type Collection 

type Collection struct {
	Type          string   `json:"type"`
	ID            string   `json:"id"`
	Title         string   `json:"title"`
	Samples       []string `json:"samples,omitempty"`
	FileCount     int      `json:"file_count"`
	EstimatedSize int64    `json:"estimated_size"`
	UserConfirmed bool     `json:"user_confirmed"`
}

Collection represents a hierarchical dataset collection.

type DatasetRecord 

type DatasetRecord struct {
	DownloadTime time.Time        `json:"download_time"`
	Metadata     *Metadata        `json:"metadata"`
	Options      *DownloadOptions `json:"options,omitempty"`
	Source       string           `json:"source"`
	OriginalID   string           `json:"original_id"`
	ResolvedURL  string           `json:"resolved_url,omitempty"`
}

DatasetRecord captures provenance for a single dataset/accession within a shared output directory. It is appended to WitnessFile.Datasets whenever a second (or later) distinct accession is downloaded into the same folder.

type DirectoryStatus 

type DirectoryStatus struct {
	TargetPath string   `json:"target_path"`
	Conflicts  []string `json:"conflicts,omitempty"`
	FreeSpace  int64    `json:"free_space,omitempty"`
	Exists     bool     `json:"exists"`
	HasWitness bool     `json:"has_witness"`
}

DirectoryStatus represents the state of the target download directory.

type DownloadOptions 

type DownloadOptions struct {
	// File-level filters (Phase 1)
	IncludeExts  []string `json:"include_exts,omitempty"`  // only download files with these extensions (e.g. ".h5ad", ".csv.gz")
	ExcludeExts  []string `json:"exclude_exts,omitempty"`  // skip files with these extensions
	FilenameGlob string   `json:"filename_glob,omitempty"` // only download filenames matching this glob
	MaxFileSize  int64    `json:"max_file_size,omitempty"` // skip files larger than this (bytes, 0 = no limit)

	// Source-specific filters (Phase 2)
	Subset   []string `json:"subset,omitempty"`   // only download these sub-items (e.g. specific GSMs within a GSE)
	Organism string   `json:"organism,omitempty"` // skip datasets whose organism doesn't match (case-insensitive partial)
	DryRun   bool     `json:"dry_run,omitempty"`  // enumerate files without downloading

	// Testing / throttling
	LimitFiles int  `json:"limit_files,omitempty"` // stop after downloading this many files (0 = no limit)
	IncludeSRA bool `json:"include_sra,omitempty"` // also download raw FASTQ files via SRA/ENA

	CustomFilters        map[string]string `json:"custom_filters,omitempty"`
	MaxConcurrent        int               `json:"max_concurrent"`
	IncludeRaw           bool              `json:"include_raw"`
	ExcludeSupplementary bool              `json:"exclude_supplementary"`
	Resume               bool              `json:"resume"`
	SkipExisting         bool              `json:"skip_existing"`
	NonInteractive       bool              `json:"non_interactive"`
	Force                bool              `json:"force"` // overwrite existing files without prompting
}

DownloadOptions provides configuration for download behavior.

type DownloadRequest 

type DownloadRequest struct {
	Options   *DownloadOptions `json:"options,omitempty"`
	Metadata  *Metadata        `json:"metadata,omitempty"`
	ID        string           `json:"id"`
	OutputDir string           `json:"output_dir"`
}

DownloadRequest encapsulates all parameters for a download operation.

type DownloadResult 

type DownloadResult struct {
	Metadata        *Metadata     `json:"metadata"`
	Checksum        string        `json:"checksum,omitempty"`
	ChecksumType    string        `json:"checksum_type,omitempty"`
	WitnessFile     string        `json:"witness_file"`
	Files           []FileInfo    `json:"files"`
	Collections     []Collection  `json:"collections,omitempty"`
	Errors          []string      `json:"errors,omitempty"`
	Warnings        []string      `json:"warnings,omitempty"`
	Duration        time.Duration `json:"duration"`
	BytesTotal      int64         `json:"bytes_total"`
	BytesDownloaded int64         `json:"bytes_downloaded"`
	Success         bool          `json:"success"`
}

DownloadResult contains the outcome and metadata of a download operation.

func Download 

func Download(ctx context.Context, sourceType string, req *DownloadRequest) (*DownloadResult, error)

Download performs a download using the default registry.

type DownloadStats 

type DownloadStats struct {
	Duration        time.Duration `json:"duration"`
	BytesTotal      int64         `json:"bytes_total"`
	BytesDownloaded int64         `json:"bytes_downloaded"`
	FilesTotal      int           `json:"files_total"`
	FilesDownloaded int           `json:"files_downloaded"`
	FilesSkipped    int           `json:"files_skipped"`
	FilesFailed     int           `json:"files_failed"`
	AverageSpeed    float64       `json:"average_speed_bps"` // Bytes per second
	MaxConcurrent   int           `json:"max_concurrent"`
	ResumedDownload bool          `json:"resumed_download"`
}

DownloadStats contains performance and operational statistics.

type Downloader 

type Downloader interface {
	// Validate checks if the ID is valid for this source type
	Validate(ctx context.Context, id string) (*ValidationResult, error)

	// GetMetadata retrieves dataset information without downloading
	GetMetadata(ctx context.Context, id string) (*Metadata, error)

	// Download performs the actual download with progress tracking
	Download(ctx context.Context, req *DownloadRequest) (*DownloadResult, error)

	// GetSourceType returns the source type identifier (e.g., "geo", "figshare")
	GetSourceType() string
}

Downloader defines the interface for downloading datasets from specific sources.

func Get 

func Get(sourceType string) (Downloader, error)

Get retrieves a downloader from the default registry.

type DownloaderError 

type DownloaderError struct {
	Type    string `json:"type"`
	Message string `json:"message"`
	Source  string `json:"source,omitempty"`
	ID      string `json:"id,omitempty"`
}

Error types for specific error conditions.

func (*DownloaderError) Error 

func (e *DownloaderError) Error() string

type FileInfo 

type FileInfo struct {
	DownloadTime time.Time `json:"download_time"`
	Path         string    `json:"path"`
	OriginalName string    `json:"original_name"`
	Checksum     string    `json:"checksum,omitempty"`
	ChecksumType string    `json:"checksum_type,omitempty"`
	SourceURL    string    `json:"source_url"`
	ContentType  string    `json:"content_type,omitempty"`
	Size         int64     `json:"size"`
	CacheHit     bool      `json:"cache_hit,omitempty"`
}

FileInfo contains metadata about an individual downloaded file.

type FileWitness 

type FileWitness struct {
	DownloadTime time.Time `json:"download_time"`
	Path         string    `json:"path"`
	OriginalName string    `json:"original_name"`
	Checksum     string    `json:"checksum,omitempty"`
	ChecksumType string    `json:"checksum_type,omitempty"`
	SourceURL    string    `json:"source_url"`
	ContentType  string    `json:"content_type,omitempty"`
	Size         int64     `json:"size"`
	CacheHit     bool      `json:"cache_hit,omitempty"`
}

FileWitness contains detailed provenance information for each file.

type HierarchyTree 

type HierarchyTree struct {
	Name     string           `json:"name"`
	Type     string           `json:"type"`
	Children []*HierarchyTree `json:"children,omitempty"`
	Size     int64            `json:"size,omitempty"`
}

HierarchyTree represents the structure of a hierarchical dataset.

type Metadata 

type Metadata struct {
	LastModified time.Time      `json:"last_modified"`
	Created      time.Time      `json:"created,omitempty"`
	Custom       map[string]any `json:"custom,omitempty"`
	DOI          string         `json:"doi,omitempty"`
	ID           string         `json:"id"`
	Title        string         `json:"title"`
	Description  string         `json:"description"`
	Source       string         `json:"source"`
	Version      string         `json:"version,omitempty"`
	License      string         `json:"license,omitempty"`
	Authors      []string       `json:"authors,omitempty"`
	Tags         []string       `json:"tags,omitempty"`
	Keywords     []string       `json:"keywords,omitempty"`
	Collections  []Collection   `json:"collections,omitempty"`
	TotalSize    int64          `json:"total_size"`
	FileCount    int            `json:"file_count"`
}

Metadata contains comprehensive information about a dataset.

func GetMetadata 

func GetMetadata(ctx context.Context, sourceType, id string) (*Metadata, error)

GetMetadata retrieves metadata using the default registry.

type ProgressCallback 

type ProgressCallback func(bytesDownloaded, bytesTotal int64, filename string)

ProgressCallback is called during download operations to report progress.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry manages the collection of available downloaders.

func NewRegistry 

func NewRegistry() *Registry

NewRegistry creates a new downloader registry.

func (*Registry) AutoDetect 

func (r *Registry) AutoDetect(ctx context.Context, id string) (string, *ValidationResult, error)

AutoDetect attempts to determine the source type from an ID.

func (*Registry) Download 

func (r *Registry) Download(ctx context.Context, sourceType string, req *DownloadRequest) (*DownloadResult, error)

Download performs a download using the appropriate downloader.

func (*Registry) Get 

func (r *Registry) Get(sourceType string) (Downloader, error)

Get retrieves a downloader by source type or alias.

func (*Registry) GetMetadata 

func (r *Registry) GetMetadata(ctx context.Context, sourceType, id string) (*Metadata, error)

GetMetadata retrieves metadata for a dataset using the appropriate downloader.

func (*Registry) List 

func (r *Registry) List() []string

List returns all registered source types.

func (*Registry) ListWithAliases 

func (r *Registry) ListWithAliases() map[string][]string

ListWithAliases returns all registered source types and their aliases.

func (*Registry) Register 

func (r *Registry) Register(downloader Downloader) error

Register adds a downloader to the registry.

func (*Registry) RegisterAlias 

func (r *Registry) RegisterAlias(alias, sourceType string) error

RegisterAlias creates an alias for an existing downloader.

func (*Registry) Validate 

func (r *Registry) Validate(ctx context.Context, sourceType, id string) (*ValidationResult, error)

Validate checks if an ID is valid for any registered downloader.

type SearchOptions 

type SearchOptions struct {
	Organism  string // filter by organism (added as a query field operator)
	EntryType string // filter by entry type (e.g. "GSE", "GSM")
	Limit     int    // maximum number of results (0 → source default)
}

SearchOptions configures a dataset search operation.

type SearchResult 

type SearchResult struct {
	Accession   string `json:"accession"`
	Title       string `json:"title"`
	Organism    string `json:"organism,omitempty"`
	EntryType   string `json:"entry_type,omitempty"`
	DatasetType string `json:"dataset_type,omitempty"`
	Date        string `json:"date,omitempty"`
	SampleCount int    `json:"sample_count,omitempty"`
	FileSize    int64  `json:"file_size,omitempty"` // total bytes of all files in the dataset
}

SearchResult represents a single search hit from a data repository.

type Searcher 

type Searcher interface {
	Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)
}

Searcher is an optional capability a Downloader can implement to support free-text dataset discovery.

type ValidationResult 

type ValidationResult struct {
	ID         string   `json:"id"`
	SourceType string   `json:"source_type"`
	Errors     []string `json:"errors,omitempty"`
	Warnings   []string `json:"warnings,omitempty"`
	Valid      bool     `json:"valid"`
}

ValidationResult contains the outcome of ID validation.

func AutoDetect 

func AutoDetect(ctx context.Context, id string) (string, *ValidationResult, error)

AutoDetect attempts to auto-detect source type using the default registry.

func Validate 

func Validate(ctx context.Context, sourceType, id string) (*ValidationResult, error)

Validate validates an ID using the default registry.

type Verification 

type Verification struct {
	VerifyTime time.Time `json:"verify_time"`
	Method     string    `json:"method"`
	Expected   string    `json:"expected,omitempty"`
	Actual     string    `json:"actual,omitempty"`
	Errors     []string  `json:"errors,omitempty"`
	Verified   bool      `json:"verified"`
}

Verification contains integrity verification information.

type WitnessFile 

type WitnessFile struct {
	DownloadTime  time.Time        `json:"download_time"`
	Metadata      *Metadata        `json:"metadata"`
	DownloadStats *DownloadStats   `json:"download_stats"`
	Verification  *Verification    `json:"verification,omitempty"`
	Options       *DownloadOptions `json:"options,omitempty"`
	HapiqVersion  string           `json:"hapiq_version"`
	Source        string           `json:"source"`
	OriginalID    string           `json:"original_id"`
	ResolvedURL   string           `json:"resolved_url,omitempty"`
	Files         []FileWitness    `json:"files"`
	Collections   []Collection     `json:"collections,omitempty"`
	// Datasets accumulates per-accession provenance when multiple distinct
	// datasets are downloaded into the same output directory.
	Datasets []DatasetRecord `json:"datasets,omitempty"`
}

WitnessFile represents the hapiq.json metadata file for provenance tracking.

Source Files

View all Source files

Directories

Path Synopsis
Package biostudies downloads datasets from EBI BioStudies (https://www.ebi.ac.uk/biostudies).
 Package biostudies downloads datasets from EBI BioStudies (https://www.ebi.ac.uk/biostudies).
Package common provides shared utilities for downloader implementations including filesystem operations, progress tracking, and user interaction.
 Package common provides shared utilities for downloader implementations including filesystem operations, progress tracking, and user interaction.
Package ensembl provides download functionality for Ensembl Genomes datasets.
 Package ensembl provides download functionality for Ensembl Genomes datasets.
Package experimenthub implements a hapiq downloader for Bioconductor's ExperimentHub.
 Package experimenthub implements a hapiq downloader for Bioconductor's ExperimentHub.
Package figshare provides download functionality for different Figshare dataset types including articles, collections, and projects with comprehensive file handling.
 Package figshare provides download functionality for different Figshare dataset types including articles, collections, and projects with comprehensive file handling.
Package geo provides download functionality for different GEO dataset types using NCBI E-utilities for metadata discovery and FTP for file downloads.
 Package geo provides download functionality for different GEO dataset types using NCBI E-utilities for metadata discovery and FTP for file downloads.
Package hca downloads count matrices and processed files from the Human Cell Atlas Data Portal via the Azul service API.
 Package hca downloads count matrices and processed files from the Human Cell Atlas Data Portal via the Azul service API.
Package scanpy implements a hapiq downloader for the curated datasets shipped with scanpy.datasets (the network-fetched ones; package-bundled datasets are out of scope).
 Package scanpy implements a hapiq downloader for the curated datasets shipped with scanpy.datasets (the network-fetched ones; package-bundled datasets are out of scope).
Package scperturb downloads datasets from the scPerturb collection (Peidli et al., Nature Methods 2024), a standardised compendium of single-cell perturbation studies.
 Package scperturb downloads datasets from the scPerturb collection (Peidli et al., Nature Methods 2024), a standardised compendium of single-cell perturbation studies.
Package sharepoint resolves anonymous SharePoint / OneDrive-for-Business "share links" into direct, byte-serving download URLs that hapiq's generic url downloader can stream.
 Package sharepoint resolves anonymous SharePoint / OneDrive-for-Business "share links" into direct, byte-serving download URLs that hapiq's generic url downloader can stream.
Package sra downloads raw sequencing reads (FASTQ) from the ENA/SRA via EBI's public HTTPS mirror.
 Package sra downloads raw sequencing reads (FASTQ) from the ENA/SRA via EBI's public HTTPS mirror.
Package url implements a hapiq downloader that fetches directly from an arbitrary HTTP(S) URL.
 Package url implements a hapiq downloader that fetches directly from an arbitrary HTTP(S) URL.
Package czi downloads datasets from the CZI Virtual Cell Platform (VCP).
 Package czi downloads datasets from the CZI Virtual Cell Platform (VCP).
Package zenodo provides download functionality for Zenodo datasets with progress tracking, file management, and error handling.
 Package zenodo provides download functionality for Zenodo datasets with progress tracking, file management, and error handling.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.