carchivum

README

carchivum

About

Carchivum is a package for working with compressed archives. Archivum is latin for archives. Carchivum is not an application, just a package that other applications can use to add compressed archive support.

These compressed archives can be in either tar or zip format.

Supported archival formats

Default: tar

Tar, tape archive, is the default archive format that carchivum uses. Carchivum does not support all of the compression formats that tar does. It may, at some point, support compression formats that tar does not. Tar archives generated by Carchivum may not be extractable by tar; compatibility depends on the algorithm used for compression. If compatibility with tar is desired, make sure the compression format being used is one that tar supports.

Carchivum's default compression format for tarballs is gzip.

In the future, the carchivum archives may be more than a tar, which will make .car files incompatible with tar. This will probably be implemented in a manner that continues to support the tar format, but no gurantees. If those does occur, a flag will be added for tar compatibility. This flag will not guarantee that tar will be able to extract a .car file as this will also depend on the compression algorithm used. It will guarantee that the archive is created as a tar.

zip

The zip format includes compression and its standard extension is .zip. No other compression schemes are supported.

Supported Compression Algorithms

Carchivum supports a number of compression algorithms. More may be implemented in the future. Carchivum does not support all of the compression algorithms that tar does. Carchivum does support some compression algorithms that tar does not. If compatibility with tar is important to you, make sure that the compression algorithm used is supported by tar. By default, Carchivum uses gzip for compression; this is compatible with tar.

• bzip2
• gzip
• lzw
• lz4

Currently there is no support for specifying the compression level, the defaults compression levels are used.

Options

format       string          The archive format to use. (DEFAULT=tar)
type         string          The compression type to use. This is only used
                             when the archive format is tar. (DEFAULT=gzip)
usefullpath  bool            If files should be archives using their fullpath
                             or relative paths. (DEFAULT=false)

Adding carchivum to your application

import github.com/mohae/carchivum

For an example implementation, please see car, my cross-platform CLI tool for creating car files.

Enable logging

This package uses the standard log package and logs all errors as log.Print() or log.Printf() prior to returning.

It may be that this package stops logging directly and only returns the error, relying on the caller to do any logging it deems appropriate. This is a design decision I have punted, for now.

Functionality wishlist

• Add support for exclude filters.
• Add support for include filters.
• Add support for archiving since a date.
• Add support for archiving using relative datetime.

Documentation

Overview

Package carchivum works with compressed archives.

Carchivum supports zip and tar. For tar, archiver also supports the following compression:

gzip
bzip2
lz4

When creating a tar, compression is not optional. Carchivum does not support everything tar does. If a compression algorithm is used that tar does not support, tar will not be able to decompress it; otherwise it should be compatible with tar, for now.

Index

• Variables
• func Extract(dst, src string) error
• func IsSupported(format magicnum.Format) bool
• func ZipBytes(b []byte, name string) (n int, zipped []byte, err error)
• type Car
  • func (c *Car) AddFile(root, p string, fi os.FileInfo, err error) error
  • func (c *Car) Delta() float64
  • func (c *Car) Message() string
• type Tar
  • func NewTar(n string) *Tar
  • func (t *Tar) Create(src ...string) (cnt int, err error)
  • func (t *Tar) CreateGZip(w io.Writer) (err error)
  • func (t *Tar) CreateLZ4(w io.Writer) (err error)
  • func (t *Tar) Delete() error
  • func (t *Tar) Extract() error
  • func (t *Tar) ExtractArchive(src io.Reader) error
  • func (t *Tar) ExtractGzip(src io.Reader) (err error)
  • func (t *Tar) ExtractLZ4(src io.Reader) error
  • func (t *Tar) ExtractTar(src io.Reader) (err error)
  • func (t *Tar) ExtractTbz(src io.Reader) error
  • func (t *Tar) ExtractTgz(src io.Reader) error
  • func (t *Tar) Write() (*sync.WaitGroup, error)
• type Zip
  • func NewZip(n string) *Zip
  • func (z *Zip) Create(src ...string) (cnt int, err error)
  • func (z *Zip) Extract() error

Variables

var CPUMultiplier = 4

CPUMultiplier set the multiplier to some default value.

var MaxRand = 10000

MaxRand default max random number for pseudo-random number generation.

Functions

func Extract

func Extract(dst, src string) error

Extract can handle the processing and extraction of a source file. Dst is the destination directory of the output, if a location other than the CWD is desired. The source file can be a zip, tar, or compressed tar.

func IsSupported

func IsSupported(format magicnum.Format) bool

IsSupported returns whether or not a specific format is supported.

func ZipBytes

func ZipBytes(b []byte, name string) (n int, zipped []byte, err error)

ZipBytes takes a string and bytes and returns a zip archive of the bytes using the name.

Types

type Car

type Car struct {

	// Name of the archive, this includes path information, if any.
	Name       string
	UseLongExt bool
	OutDir     string
	// Create operation modifiers
	Owner int
	Group int
	os.FileMode
	// Extract operation modifiers
	UseFullpath bool

	DeleteArchived bool
	// Exclude file processing
	Exclude         string
	ExcludeExt      []string
	ExcludeExtCount int
	ExcludeAnchored string
	// Include file processing
	Include         string
	IncludeExt      []string
	IncludeExtCount int
	IncludeAnchored string
	// File time format handling
	Newer      string
	NewerMTime time.Time
	NewerFile  string

	// Processing queue
	FileCh chan *os.File

	𝛥t float64
	// contains filtered or unexported fields
}

Car is a Compressed Archive. The struct holds information about Cars and their processing.

func (*Car) AddFile

func (c *Car) AddFile(root, p string, fi os.FileInfo, err error) error

AddFile reads a file and pipes it to the zipper goroutine.

func (*Car) Delta

func (c *Car) Delta() float64

Delta returns the time delta between operation start and end.

func (*Car) Message

func (c *Car) Message() string

Message provides summary information about the operation performed.

type Tar

type Tar struct {
	Car
	*tar.Writer
	magicnum.Format
	// contains filtered or unexported fields
}

Tar is a struct for a tar, tape archive.

func NewTar

func NewTar(n string) *Tar

NewTar returns an initialized Tar struct ready for use.

func (*Tar) Create

func (t *Tar) Create(src ...string) (cnt int, err error)

Create creates a tarfile from the passed src('s) and saves it to the dst.

func (*Tar) CreateGZip

func (t *Tar) CreateGZip(w io.Writer) (err error)

CreateGZip creates a GZip using the passed writer.

func (*Tar) CreateLZ4

func (t *Tar) CreateLZ4(w io.Writer) (err error)

CreateLZ4 creates a LZ4 compressed tarball using the passed writer.

func (*Tar) Delete

func (t *Tar) Delete() error

Delete is not implemented

func (*Tar) Extract

func (t *Tar) Extract() error

Extract extracts the files from the src and writes them to the dst. The src is either a tar or a compressed tar.

func (*Tar) ExtractArchive

func (t *Tar) ExtractArchive(src io.Reader) error

ExtractArchive takes a compressed tar archive, as an io.Reader. If the compression format used is supported, it will decompress and extract the contents of the tar; otherwise it will return an error.

func (*Tar) ExtractGzip

func (t *Tar) ExtractGzip(src io.Reader) (err error)

ExtractGzip reads a GZip using the passed reader.

func (*Tar) ExtractLZ4

func (t *Tar) ExtractLZ4(src io.Reader) error

ExtractLZ4 extracts LZ4 compressed tarballs.

func (*Tar) ExtractTar

func (t *Tar) ExtractTar(src io.Reader) (err error)

ExtractTar extracts a tar file using the passed reader

func (*Tar) ExtractTbz

func (t *Tar) ExtractTbz(src io.Reader) error

ExtractTbz extracts Bzip2 compressed tarballs.

func (*Tar) ExtractTgz

func (t *Tar) ExtractTgz(src io.Reader) error

ExtractTgz extracts GZip'd tarballs.

func (*Tar) Write

func (t *Tar) Write() (*sync.WaitGroup, error)

Write adds the files received on the channel to the tarball.

type Zip

type Zip struct {
	Car
	*zip.Writer
	*os.File
}

Zip handles .zip archives.

func NewZip

func NewZip(n string) *Zip

NewZip returns an initialized Zip struct ready for use.

func (*Zip) Create

func (z *Zip) Create(src ...string) (cnt int, err error)

Create creates a zip file from src in the dst

func (*Zip) Extract

func (z *Zip) Extract() error

Extract the content of src, a zip archive. The destination is CWD, unless OutputDir is specified; then it will be a child of the output dir.