Help us shape the future of Go! Take the 2026 Go Developer Survey

bakpack

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 9, 2026 License: MIT Imports: 25 Imported by: 0

README

bakpack

bakpack is a Go command line tool and library for compressing Bakta JSON annotation files while keeping them reconstructable and checkable against the original JSON content.

This repository was developed with substantial coding assistance from OpenAI Codex and Claude Code, which helped with implementation, refactoring, tests, documentation, and benchmarking under human direction and review.

Documentation: bakpack.readthedocs.io

Install

The simplest way to install bakpack is to download the latest prebuilt binary from the GitHub releases page:

Choose the archive or binary matching your OS and CPU architecture. Release pages also include a SHA-256 checksum file named like bakpack-v0.1.0-checksums.txt. Use it to verify downloaded archives before installing.

After installing, check the version with:

bakpack --version

If you want to build locally instead:

git clone https://github.com/martinghunt/bakpack.git
cd bakpack
./build.sh

The binary is written to:

build/bakpack

With Go directly:

go install github.com/martinghunt/bakpack/cmd/bakpack@latest

Archive creation uses the command line xz program:

xz -9e -T1 -c

Put xz in PATH. AGC genome input requires agc in PATH.

Quick Start

Make one reduced JSON file:

bakpack reduce sample.bakta.json sample.fa -o sample.reduced.bakta.json

Reconstruct original JSON content:

bakpack restore sample.reduced.bakta.json sample.fa -o sample.bakta.json

Render a Bakta-style GFF3 file from original or reduced JSON:

bakpack gff3 sample.bakta.json sample.fa -o sample.gff3

Omit the terminal FASTA section when only annotation rows are needed:

bakpack gff3 sample.bakta.json sample.fa -o sample.gff3 --annotation-only

Build a compressed archive:

bakpack build \
  --annotations annotations.tar.xz \
  --genomes genomes.tar.xz \
  --output annotations.bakpack

Build from a combined manifest:

bakpack build \
  --manifest samples.tsv \
  --output annotations.bakpack

Extract one reconstructed annotation and its genome FASTA:

bakpack extract annotations.bakpack SAMN1 \
  --genomes genomes.tar.xz \
  --original \
  --gff3-annotation-only \
  --genome \
  --output-dir out

The archive path can also be an HTTP(S) URL when the server supports byte-range requests:

bakpack extract https://example.org/annotations.bakpack SAMN1 \
  --genomes genomes.tar.xz \
  --original \
  --output-dir out

Extract several reduced annotations efficiently:

bakpack extract annotations.bakpack SAMN1 SAMN2 SAMN3 \
  --reduced \
  --output-dir out

Go library

Import the root package from another Go module:

import "github.com/martinghunt/bakpack"

Use Archive.Extract for extracted bytes, or Archive.ExtractFiles / ExtractArchive to write the same files as the CLI from library code:

archive, err := bakpack.OpenArchive(ctx, "annotations.bakpack")
if err != nil {
	panic(err)
}
defer archive.Close()

err = archive.ExtractFiles(ctx, bakpack.ExtractRequest{
	Genomes:  genomes,
	Samples:  []string{"SAMN1"},
	Original: true,
	Genome:   true,
}, "out")

See Library usage for the full API examples.

Documentation

Overview

Package bakpack provides tools for reducing, packing, reading, and extracting Bakta annotation JSON archives.

Use OpenArchive with Archive.Extract when callers want extracted bytes in memory, Archive.ExtractFiles when callers want standard bakpack output files in a directory, and ExtractArchive for one-shot file extraction from an archive path or HTTP(S) URL.

Index

Constants

View Source 
const (
	ArchiveMagic   = "BAKPACK1"
	ArchiveVersion = 1
)

Variables

View Source 
var AnnotationJSONSuffixes = []string{".bakta.json", ".json"}

AnnotationJSONSuffixes lists the recognized Bakta annotation JSON filename suffixes, in the order checked. It's exported so cmd/bakpack's sampleIDFromPath (used by reduce/restore/gff3, which take an explicit JSON path rather than scanning a directory) can derive a sample ID from the same suffix list instead of maintaining its own separate copy.

Functions

func BaktaGFF3 added in v0.2.0 

func BaktaGFF3(annotation []byte, genome Genome) ([]byte, error)

BaktaGFF3 renders a Bakta JSON annotation as GFF3 using the observed Bakta JSON/GFF3 file formats. The input may be either original JSON or bakpack reduced JSON; reduced input is restored before rendering.

func BaktaGFF3WithOptions added in v0.2.0 

func BaktaGFF3WithOptions(annotation []byte, genome Genome, opts BaktaGFF3Options) ([]byte, error)

BaktaGFF3WithOptions renders a Bakta JSON annotation as GFF3 with explicit output options.

func BuildArchive 

func BuildArchive(ctx context.Context, opts BuildOptions) error

func CanonicalJSON 

func CanonicalJSON(value any) ([]byte, error)

func CanonicalJSONSHA256 

func CanonicalJSONSHA256(value any) (string, error)

func DecodeJSON 

func DecodeJSON(data []byte) (any, error)

func ExtractArchive 

func ExtractArchive(ctx context.Context, opts ExtractOptions) error

ExtractArchive opens opts.ArchivePath and writes the selected sample outputs to opts.OutputDir using the same filenames as the CLI. It is a convenience wrapper around OpenArchive and Archive.ExtractFiles for callers that do not need to keep the archive open across requests.

func FeatureNT 

func FeatureNT(feature map[string]any, genome Genome) (string, bool)

func JSONBytesCanonicalSHA256 

func JSONBytesCanonicalSHA256(data []byte) (string, error)

func OpenManifestSources 

func OpenManifestSources(path string) (FileSource, FileSource, error)

func PrettyJSON 

func PrettyJSON(value any) ([]byte, error)

func SHA256Hex 

func SHA256Hex(data []byte) string

func SupportedBaktaFeatureTypes added in v0.2.0 

func SupportedBaktaFeatureTypes() []string

SupportedBaktaFeatureTypes returns the Bakta feature types currently handled by reduction, archive compression/extraction, and GFF3 rendering.

Types

type AGCGenomeSource 

type AGCGenomeSource struct {
	Path    string
	Command string
	Threads int
}

func (AGCGenomeSource) Get 

func (s AGCGenomeSource) Get(ctx context.Context, sample string) (FileRecord, error)

func (AGCGenomeSource) Order 

func (s AGCGenomeSource) Order(ctx context.Context) ([]string, error)

func (AGCGenomeSource) Records 

func (s AGCGenomeSource) Records(ctx context.Context) ([]FileRecord, error)

type Archive 

type Archive struct {
	// contains filtered or unexported fields
}

Archive is an opened bakpack archive. It keeps the archive index in memory and can extract one or more samples without reopening the archive.

func OpenArchive 

func OpenArchive(ctx context.Context, path string, opts ...OpenArchiveOptions) (*Archive, error)

OpenArchive opens a local .bakpack path or HTTP(S) URL. HTTP(S) archives are read with byte-range requests.

func (*Archive) Close 

func (a *Archive) Close() error

Close closes the underlying archive reader.

func (*Archive) Extract 

func (a *Archive) Extract(ctx context.Context, req ExtractRequest) ([]ExtractedSample, error)

Extract extracts one or more samples from the archive. Without OnSample, results are returned in the same order as req.Samples. With OnSample, results are delivered to the callback and the returned slice is nil.

func (*Archive) ExtractFiles added in v0.2.0 

func (a *Archive) ExtractFiles(ctx context.Context, req ExtractRequest, outputDir string) error

ExtractFiles extracts one or more samples and writes the selected outputs to outputDir using the same filenames as the CLI: SAMPLE.reduced.bakta.json, SAMPLE.bakta.json, SAMPLE.fa, and SAMPLE.gff3. If no output mode is selected, reduced JSON files are written. If outputDir is empty, "." is used.

ExtractFiles streams samples through OnSample and does not accumulate output bytes in memory. If req.OnSample is set, it is called after files for that sample have been written.

func (*Archive) Index 

func (a *Archive) Index() ArchiveIndex

Index returns a copy of the archive index.

func (*Archive) SampleIDs 

func (a *Archive) SampleIDs() []string

SampleIDs returns sample IDs in archive order.

type ArchiveIndex 

type ArchiveIndex struct {
	Format        string        `json:"format"`
	Version       int           `json:"version"`
	PayloadFormat string        `json:"payload_format"`
	ChunkSize     int           `json:"chunk_size"`
	Chunks        []ChunkIndex  `json:"chunks"`
	Samples       []SampleIndex `json:"samples"`
}

func ReadArchiveIndex 

func ReadArchiveIndex(path string) (ArchiveIndex, error)

ReadArchiveIndex reads the front index from a local or HTTP(S) archive using a background context.

func ReadArchiveIndexContext 

func ReadArchiveIndexContext(ctx context.Context, path string, opts ...OpenArchiveOptions) (ArchiveIndex, error)

ReadArchiveIndexContext reads the front index from a local or HTTP(S) archive. HTTP(S) archives are accessed with byte-range requests.

type BaktaGFF3Options added in v0.2.0 

type BaktaGFF3Options struct {
	// AnnotationOnly omits the terminal ##FASTA section containing contig
	// sequences.
	AnnotationOnly bool
	// BakpackVersion overrides the version written in the generated-by header.
	// When empty, the build-time bakpack version is used.
	BakpackVersion string
}

BaktaGFF3Options configures Bakta JSON to GFF3 rendering.

type BuildOptions 

type BuildOptions struct {
	Annotations FileSource
	Genomes     FileSource
	Order       []string
	ChunkSize   int
	OutputPath  string
	XZThreads   int

	// AnnotationSpoolCompression controls temporary files made when building
	// from annotation tar.xz sources whose order differs from genome order.
	// Supported values are "", "gzip", "none", and "raw"; "" defaults to gzip.
	AnnotationSpoolCompression string
}

type ChunkIndex 

type ChunkIndex struct {
	ID               int                `json:"id"`
	Offset           int64              `json:"offset"`
	CompressedSize   int64              `json:"compressed_size"`
	UncompressedSize int64              `json:"uncompressed_size"`
	TopKeys          []string           `json:"top_keys,omitempty"`
	ValueSchemas     []SchemaIndexEntry `json:"value_schemas,omitempty"`
	FeatureSchemas   []SchemaIndexEntry `json:"feature_schemas,omitempty"`
	FeatureFields    []string           `json:"feature_fields,omitempty"`
	FieldCodecs      []FieldCodec       `json:"field_codecs,omitempty"`
}

type DirSource 

type DirSource struct {
	Dir  string
	Role string
}

func (DirSource) Get 

func (s DirSource) Get(ctx context.Context, sample string) (FileRecord, error)

func (DirSource) Order 

func (s DirSource) Order(ctx context.Context) ([]string, error)

func (DirSource) Records 

func (s DirSource) Records(ctx context.Context) ([]FileRecord, error)

type ExtractOptions 

type ExtractOptions struct {
	// ArchivePath is a local .bakpack path or HTTP(S) URL.
	ArchivePath string
	// OpenOptions configures archive reads, including custom HTTP clients for
	// HTTP(S) archives.
	OpenOptions OpenArchiveOptions
	// Genomes is required when Original, Genome, or GFF3 output is requested.
	Genomes FileSource
	// Samples are the sample IDs to extract.
	Samples []string
	// OutputDir receives extracted files. If empty, "." is used.
	OutputDir string
	// Reduced writes SAMPLE.reduced.bakta.json.
	Reduced bool
	// Original writes SAMPLE.bakta.json.
	Original bool
	// Genome writes SAMPLE.fa.
	Genome bool
	// GFF3 writes SAMPLE.gff3.
	GFF3 bool
	// GFF3AnnotationOnly writes SAMPLE.gff3 without the terminal ##FASTA
	// section. It implies GFF3.
	GFF3AnnotationOnly bool
}

ExtractOptions configures one-shot extraction from an archive path or HTTP(S) URL to files on disk.

type ExtractRequest 

type ExtractRequest struct {
	// Genomes is required when Original, Genome, or GFF3 output is requested.
	Genomes FileSource
	// Samples are the sample IDs to extract.
	Samples []string
	// Reduced returns reduced Bakta JSON.
	Reduced bool
	// Original reconstructs original Bakta JSON and verifies its canonical hash.
	Original bool
	// Genome returns matching genome FASTA.
	Genome bool
	// GFF3 renders a Bakta-style GFF3 annotation.
	GFF3 bool
	// GFF3AnnotationOnly returns GFF3 output without the terminal ##FASTA section.
	// It implies GFF3.
	GFF3AnnotationOnly bool
	// OnSample is called for each extracted sample. When nil, Extract returns
	// accumulated results in the same order as Samples.
	OnSample func(ExtractedSample) error
}

ExtractRequest configures extraction from an opened archive.

type ExtractedSample 

type ExtractedSample struct {
	SampleID                    string
	AnnotationName              string
	GenomeName                  string
	ReducedJSON                 []byte
	OriginalJSON                []byte
	GenomeFASTA                 []byte
	GFF3                        []byte
	OriginalJSONCanonicalSHA256 string
	ReducedJSONCanonicalSHA256  string
}

ExtractedSample contains the bytes extracted or reconstructed for one sample.

type FieldCodec 

type FieldCodec struct {
	Field  string   `json:"field"`
	Kind   string   `json:"kind"`
	Value  any      `json:"value,omitempty"`
	Values []string `json:"values,omitempty"`
}

type FileRecord 

type FileRecord struct {
	SampleID string
	Name     string
	Bytes    []byte
}

type FileSource 

type FileSource interface {
	Records(context.Context) ([]FileRecord, error)
	Get(context.Context, string) (FileRecord, error)
	Order(context.Context) ([]string, error)
}

func OpenSource 

func OpenSource(path, kind, role string) (FileSource, error)

type Genome 

type Genome struct {
	SampleID string
	Filename string
	Contigs  []seqio.SeqRecord
	// contains filtered or unexported fields
}

func ReadGenome 

func ReadGenome(sampleID, filename string, data []byte) (Genome, error)

func (*Genome) Contig 

func (g *Genome) Contig(name string) ([]byte, bool)

Contig has a pointer receiver so the lazily-built byName cache actually persists on the caller's Genome instead of being discarded with a copy; ReadGenome always populates byName up front, but a Genome built directly (e.g. Genome{Contigs: ...}) relies on this cache being retained across calls.

func (Genome) FASTABytes 

func (g Genome) FASTABytes(wrap int) ([]byte, error)

type JSONChecksums 

type JSONChecksums struct {
	CanonicalSHA256 string `json:"canonical_sha256"`
}

type ListSource 

type ListSource struct {
	Path string
	Role string
}

func (ListSource) Get 

func (s ListSource) Get(ctx context.Context, sample string) (FileRecord, error)

func (ListSource) Order 

func (s ListSource) Order(ctx context.Context) ([]string, error)

func (ListSource) Records 

func (s ListSource) Records(ctx context.Context) ([]FileRecord, error)

type Manifest 

type Manifest struct {
	Path    string
	Entries []ManifestEntry
}

func ReadManifest 

func ReadManifest(path string) (Manifest, error)

type ManifestEntry 

type ManifestEntry struct {
	SampleID       string
	AnnotationPath string
	GenomePath     string
}

type ManifestSource 

type ManifestSource struct {
	Manifest Manifest
	Role     string
}

func (ManifestSource) Get 

func (s ManifestSource) Get(ctx context.Context, sample string) (FileRecord, error)

func (ManifestSource) Order 

func (s ManifestSource) Order(ctx context.Context) ([]string, error)

func (ManifestSource) Records 

func (s ManifestSource) Records(ctx context.Context) ([]FileRecord, error)

type OpenArchiveOptions 

type OpenArchiveOptions struct {
	// HTTPClient is used for byte-range requests when the archive path is an
	// HTTP(S) URL. A nil client uses http.DefaultClient.
	HTTPClient *http.Client
}

OpenArchiveOptions configures archive reads.

type ReduceResult 

type ReduceResult struct {
	ReducedJSON []byte
	Original    JSONChecksums
	Reduced     JSONChecksums
	// contains filtered or unexported fields
}

func ReduceBaktaJSON 

func ReduceBaktaJSON(original []byte, genome Genome) (ReduceResult, error)

type RestoreResult 

type RestoreResult struct {
	OriginalJSON []byte
	Original     JSONChecksums
}

func RestoreBaktaJSON 

func RestoreBaktaJSON(reduced []byte, genome Genome) (RestoreResult, error)

type SampleIndex 

type SampleIndex struct {
	SampleID                    string `json:"sample_id"`
	AnnotationName              string `json:"annotation_name"`
	GenomeName                  string `json:"genome_name"`
	ChunkID                     int    `json:"chunk_id"`
	OriginalJSONCanonicalSHA256 string `json:"original_json_canonical_sha256"`
	ReducedJSONCanonicalSHA256  string `json:"reduced_json_canonical_sha256"`
}

type SchemaIndexEntry 

type SchemaIndexEntry struct {
	SchemaID int      `json:"schema_id"`
	Keys     []string `json:"keys"`
}

type SingleFileSource added in v0.3.0 

type SingleFileSource struct {
	File sourceFile
}

SingleFileSource exposes one genome file as a source. Its sample ID is inferred from the file name, including recognised compression suffixes.

func (SingleFileSource) Get added in v0.3.0 

func (s SingleFileSource) Get(ctx context.Context, sample string) (FileRecord, error)

func (SingleFileSource) Order added in v0.3.0 

func (s SingleFileSource) Order(context.Context) ([]string, error)

func (SingleFileSource) Records added in v0.3.0 

func (s SingleFileSource) Records(ctx context.Context) ([]FileRecord, error)

func (SingleFileSource) WithSampleID added in v0.3.0 

func (s SingleFileSource) WithSampleID(sampleID string) SingleFileSource

WithSampleID returns a copy that exposes the genome as sampleID. It is used by the CLI when a single genome file accompanies an explicit sample argument.

type TarXZSource 

type TarXZSource struct {
	Path string
	Role string
}

func (TarXZSource) Get 

func (s TarXZSource) Get(ctx context.Context, sample string) (FileRecord, error)

func (TarXZSource) Order 

func (s TarXZSource) Order(ctx context.Context) ([]string, error)

func (TarXZSource) Records 

func (s TarXZSource) Records(ctx context.Context) ([]FileRecord, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
bakpack command
internal
buildinfo

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.