agc

Documentation

Overview

Package agc reads AGC (Assembled Genomes Compressor) archives by shelling out to the upstream `agc` binary. atb fetches and caches that binary next to itself; agc is never linked into atb, so atb stays pure Go and statically linked. The package is read-only: it lists samples/contigs and extracts FASTA, but never creates or modifies archives.

Index

• Constants
• func ArchiveDir(dataDir, override string) string
• func FindBinary() (string, error)
• func GetCollection(archive string, w io.Writer, o Options) error
• func GetContigs(archive string, queries []string, w io.Writer, o Options) error
• func GetSamples(archive string, samples []string, w io.Writer, o Options) error
• func Info(archive string) (string, error)
• func InstallBinary(progress func(string)) error
• func ListContigs(archive, sample string) ([]string, error)
• func ListSamples(archive string) ([]string, error)
• func ReferenceSample(archive string) (string, error)
• func RefsForGroups(groups map[string][]string, byName map[string]ArchiveRef) (refs map[string]ArchiveRef, missing []string)
• func RefsFromIndex(idx *osf.Index) map[string]ArchiveRef
• func WholeArchiveGroups(refs []ArchiveRef) (groups map[string][]string, byName map[string]ArchiveRef)
• type ArchiveMap
  • func FetchMap(cacheDir, mapURL string, force bool) (ArchiveMap, error)
  • func ParseMap(r io.Reader) (ArchiveMap, error)
• type ArchiveRef
  • func SelectBySpecies(idx *osf.Index, species string) []ArchiveRef
• type FetchError
• type FetchResult
  • func FetchGenomes(groups map[string][]string, spec FetchSpec) (FetchResult, error)
• type FetchSpec
• type LocateResult
  • func Locate(accessions []string, m ArchiveMap, byName map[string]ArchiveRef) []LocateResult
• type LocateStatus
• type Options
• type Unresolved
  • func ResolveArchives(dataDir, mapURL string, accessions []string, refresh bool) (groups map[string][]string, unresolved []Unresolved, err error)

Constants

const CacheMaxAge = 7 * 24 * time.Hour

CacheMaxAge is how long a cached archive map is considered fresh.

Functions

func ArchiveDir

func ArchiveDir(dataDir, override string) string

ArchiveDir returns the directory where .agc archives are cached: override when set, otherwise <dataDir>/agc. Mirrors the sketch database layout.

func FindBinary

func FindBinary() (string, error)

FindBinary locates the agc binary. Search order:

1. The directory containing the running atb binary.
2. The system PATH.

If neither has it, the returned error points at `atb agc install`.

func GetCollection

func GetCollection(archive string, w io.Writer, o Options) error

GetCollection extracts the entire archive as FASTA into w (agc getcol). getcol does not support streaming mode, so Options.Streaming is ignored.

func GetContigs

func GetContigs(archive string, queries []string, w io.Writer, o Options) error

GetContigs extracts the given contig queries as FASTA into w (agc getctg). Each query is contig[@sample][:from-to].

func GetSamples

func GetSamples(archive string, samples []string, w io.Writer, o Options) error

GetSamples extracts whole samples as FASTA into w (agc getset).

func Info

func Info(archive string) (string, error)

Info returns agc's archive metadata report verbatim (agc info).

func InstallBinary

func InstallBinary(progress func(string)) error

InstallBinary downloads the pinned agc release asset for the host platform, extracts the binary, and installs it (chmod 0755) next to the atb binary.

func ListContigs

func ListContigs(archive, sample string) ([]string, error)

ListContigs returns the contig names within sample (agc listctg).

func ListSamples

func ListSamples(archive string) ([]string, error)

ListSamples returns the sample names in the archive (agc listset).

func ReferenceSample

func ReferenceSample(archive string) (string, error)

ReferenceSample returns the archive's reference sample name (agc listref).

func RefsForGroups

func RefsForGroups(groups map[string][]string, byName map[string]ArchiveRef) (refs map[string]ArchiveRef, missing []string)

RefsForGroups splits a ResolveArchives result against the batch index. For every archive in groups it looks up the index refs (byName, from RefsFromIndex): archives found are returned in refs (which feeds FetchSpec.Refs exactly as species-mode does); archives absent from the index - a batch named by the map but not yet crawlable, e.g. a collection node still uploading - are returned, sorted, in missing, so the caller can report them as "not yet available" distinctly from an unknown accession. refs is always non-nil.

func RefsFromIndex

func RefsFromIndex(idx *osf.Index) map[string]ArchiveRef

RefsFromIndex keys every batch in the index by its archive stem, the same key space the fetch engine uses for its work groups. This is the R2 resolution table: archive name -> {URL, md5}.

func WholeArchiveGroups

func WholeArchiveGroups(refs []ArchiveRef) (groups map[string][]string, byName map[string]ArchiveRef)

WholeArchiveGroups turns a by-species selection into the two maps the fetch engine needs for Mode A. groups keys every batch name to an empty accession slice — the whole-archive sentinel that routes FetchGenomes through `agc getcol` (extract the entire batch) instead of per-accession `getset`. byName is the R2 table (batch name -> {URL, md5}) handed to FetchSpec.Refs so each archive downloads from its opaque OSF GUID with a free md5 check. Both maps are always non-nil; keying by Name collapses any accidental duplicate refs.

Types

type ArchiveMap

type ArchiveMap map[string]string

ArchiveMap maps an ATB sample accession to the archive name (no ".agc") that contains it.

func FetchMap

func FetchMap(cacheDir, mapURL string, force bool) (ArchiveMap, error)

FetchMap returns the parsed archive map, using a cached copy in cacheDir when it is younger than CacheMaxAge. force=true always re-downloads. An empty mapURL falls back to sources.AGCArchiveMapURL. The body is streamed to a temp file and renamed into place so a partial download never poisons the cache; the file is large. The cached artifact is stored exactly as downloaded (a gzip) and openMap decompresses it on read.

func ParseMap

func ParseMap(r io.Reader) (ArchiveMap, error)

ParseMap reads the whitespace-delimited sample->archive map: column 1 is the accession, column 2 is the archive name. Extra columns and blank or single-column lines are ignored. Lookup is by exact accession (not the prototype's substring grep).

type ArchiveRef

type ArchiveRef struct {
	Name    string
	URL     string
	MD5     string
	SizeMB  float64
	Node    string // OSF node id (osf.Entry.ProjectID); "" for non-collection sources
	Species string // index species (osf.Entry.Project); "" when unknown
}

ArchiveRef is a single AGC batch resolved from the OSF index: its archive name (without ".agc"), the opaque OSF download URL, the md5 for integrity, and the size in MB. Carrying the URL in the data is the key adaptation for OSF — archive download links are per-file GUIDs that cannot be built from the name, so the engine must be handed a real URL rather than constructing one.

func SelectBySpecies

func SelectBySpecies(idx *osf.Index, species string) []ArchiveRef

SelectBySpecies returns every batch whose species (the index Project column) matches the query exactly after normalisation. Matching is case-insensitive and treats spaces and underscores as equivalent. GTDB letter-suffixed species ("Streptococcus_suis_AA") and the "subthreshold_remainder" batch require their full name; a bare prefix does not match. The result may be empty, which the caller reports as "no batches for that species".

type FetchError

type FetchError struct {
	Accession string
	Error     string
}

FetchError records a per-accession failure.

type FetchResult

type FetchResult struct {
	Completed int
	Failed    int
	Errors    []FetchError
}

FetchResult summarises a FetchGenomes run.

func FetchGenomes

func FetchGenomes(groups map[string][]string, spec FetchSpec) (FetchResult, error)

FetchGenomes downloads each group's archive cache-first, then extracts the requested samples with `agc getset`. With Combine, all samples stream to spec.Combined; otherwise each sample is written to its own file in spec.OutputDir. Failures are collected per accession (continue-on-error); the run still returns a FetchResult so the caller can report and set the exit code. A download failure fails every accession in that archive's group.

type FetchSpec

type FetchSpec struct {
	Combine    bool                  // one combined stream instead of per-sample files
	OutputDir  string                // per-sample output directory (ignored when Combine)
	Combined   io.Writer             // combined output target (used when Combine)
	ArchiveDir string                // where .agc archives are cached
	BaseURL    string                // archive base URL for batches without an OSF ref; "" requires refs (fail-closed)
	Refs       map[string]ArchiveRef // archive name -> OSF {URL, md5}; overrides BaseURL per archive
	Parallel   int                   // parallel archive downloads
	Force      bool                  // re-download archives even if cached
	Options    Options               // agc getset/getcol flags
}

FetchSpec configures a FetchGenomes run.

type LocateResult

type LocateResult struct {
	Accession string
	Batch     string // archive stem; "" when Status is LocateUnresolved
	Species   string // index species (Project); "" unless Status is LocateFound
	Node      string // OSF node id (ProjectID); "" unless Status is LocateFound
	URL       string // OSF download URL; "" unless Status is LocateFound
	Status    LocateStatus
}

LocateResult is one accession's resolution for `atb agc locate`.

func Locate

func Locate(accessions []string, m ArchiveMap, byName map[string]ArchiveRef) []LocateResult

Locate resolves each accession against the accession->batch map m and the batch index refs (byName, from RefsFromIndex), preserving input order. It performs no I/O: callers fetch m and byName first.

type LocateStatus

type LocateStatus int

LocateStatus classifies how an accession resolved against the map and index.

const (
	// LocateFound: in the map and its batch is in the index.
	LocateFound LocateStatus = iota
	// LocateUnresolved: not in the accession->batch map.
	LocateUnresolved
	// LocateNotYetAvailable: maps to a batch that is not (yet) in the index -
	// the collection node holding it is still being published.
	LocateNotYetAvailable
)

type Options

type Options struct {
	Threads    int  // -t
	LineLength int  // -l
	GzipLevel  int  // -g
	Streaming  bool // -s
}

Options maps to agc's common get-flags. Zero values mean "use agc's default": LineLength 0 keeps agc's 80, GzipLevel 0 leaves output uncompressed, Threads 0 uses NumCPU-1. Streaming applies to getctg/getset only (getcol ignores it).

type Unresolved

type Unresolved struct {
	Accession string
	Reason    string
}

Unresolved records an accession that could not be mapped to an archive.

func ResolveArchives

func ResolveArchives(dataDir, mapURL string, accessions []string, refresh bool) (groups map[string][]string, unresolved []Unresolved, err error)

ResolveArchives maps each accession to its archive using the companion map (cached under <dataDir>/agc), grouping accessions by archive and preserving input order within each group. Accessions absent from the map are returned in unresolved rather than failing the whole call. refresh=true re-downloads the map. The SQLite index fast-path is intentionally deferred; this signature is forward-compatible with adding it later.