compress

Documentation

Overview

Package compress implements decoders for the FITS tile-compression algorithms defined by the "Compressed Images" FITS convention (Pence et al. 2010). It is a pure bytes-in / bytes-out library that knows nothing about FITS headers or binary tables — the FITS-side wrapping lives in the root fits package as the CompressedImageHDU type.

Supported algorithms

RICE_1      — Rice coding with per-block k parameter (Pence 1998,
              White & Greenfield 1998). The workhorse for JWST and
              most HST CCD data.
GZIP_1      — Raw pixel bytes through zlib DEFLATE.
GZIP_2      — Byte-shuffled GZIP: bytes are grouped by significance
              before gzip. Improves ratio on smooth imagery.
NOCOMPRESS  — Identity pass-through (used by the tile-level
              fallback chain when compression would inflate a tile).
PLIO_1      — IRAF Planio run-length encoding for integer masks.
HCOMPRESS_1 — H-transform with quadtree coding (White 1991).

Coverage status

All six algorithms listed above are recognized by Select. RICE_1, GZIP_1, GZIP_2, NOCOMPRESS, and PLIO_1 have full decoders. HCOMPRESS_1 currently returns ErrUnsupportedCompression and will ship in a follow-up — it is less common in modern data products than the others.

Index

• Constants
• Variables
• func Dequantize(input []int32, out []float64, scale, zero float64, method DitherMethod, ...)
• func DequantizeFloat32(input []int32, out []float32, scale, zero float64, method DitherMethod, ...)
• func FitsRandoms() []float32
• func ReplaceSentinelWithNaNBits(buf []float32, sentinel float32) int
• func SubstituteNaN32(data []float32, sentinel float32) int
• type Algorithm
  • func ParseAlgorithm(s string) Algorithm
  • func (a Algorithm) String() string
• type Decoder
  • func Select(algo Algorithm, params Params) (Decoder, error)
• type DitherMethod
  • func ParseDitherMethod(s string) DitherMethod
• type Encoder
  • func SelectEncoder(algo Algorithm, params Params) (Encoder, error)
• type ImageStatsF32
  • func ImgStatsFloat32(array []float32, nx, ny int64, nullcheck bool, nullvalue float32) ImageStatsF32
• type Params
  • func (p Params) Get(name string, def int64) int64
• type QuantizeResult
  • func QuantizeFloat32(row int64, fdata []float32, nxpix, nypix int64, nullcheck bool, ...) (QuantizeResult, bool)

Constants

const NRandom = 10000

NRandom is the size of the PRNG lookup table. cfitsio's fitsio2.h defines this as 10000.

const NullValueInt32 int32 = -2147483647

NullValueInt32 is the sentinel stored in the quantized integer stream in place of a null input pixel. cfitsio's quantize.c defines this as -2147483647.

const ZeroValue int32 = -2147483646

ZeroValue is the sentinel integer stored in place of a true 0.0 float when SUBTRACTIVE_DITHER_2 is in effect. The dequantizer maps it back to exact 0.0.

Variables

var (
	// ErrUnsupportedCompression is returned when the algorithm is
	// recognized but not implemented. Kept as a forward-compatibility
	// sentinel — every algorithm in the Pence et al. 2010 convention
	// (RICE_1, GZIP_1, GZIP_2, HCOMPRESS_1, PLIO_1, NOCOMPRESS) is now
	// fully supported for both read and write.
	ErrUnsupportedCompression = errors.New("fits/compress: unsupported compression algorithm")

	// ErrUnknownCompression is returned when the algorithm string does
	// not match any standard FITS compression type.
	ErrUnknownCompression = errors.New("fits/compress: unknown compression algorithm")

	// ErrCorrupt is returned when a compressed stream is malformed or
	// truncated.
	ErrCorrupt = errors.New("fits/compress: corrupt compressed stream")
)

Sentinel errors.

var ErrBufferTooSmall = fmt.Errorf("fits/compress: output buffer too small")

ErrBufferTooSmall is returned by Encoder.Encode when the caller's destination buffer cannot hold the compressed output. Callers that hit this should retry with a larger buffer or use the NOCOMPRESS fallback for this specific tile.

Functions

func Dequantize

func Dequantize(input []int32, out []float64, scale, zero float64, method DitherMethod, tileRow, zdither0 int)

Dequantize applies the per-pixel dither + scale + zero reconstruction to a slice of quantized integers, writing float64 values to out.

tileRow is the 1-based tile number in its binary table (nrow), NOT yet adjusted for ZDITHER0 — this function does the adjustment internally to match cfitsio's convention.

For NoDither this degenerates to the plain linear reconstruction input*scale + zero.

For SubtractiveDither2, integer values equal to ZeroValue map to exactly 0.0 on output, preserving the encoder's zero-exact guarantee.

func DequantizeFloat32

func DequantizeFloat32(input []int32, out []float32, scale, zero float64, method DitherMethod, tileRow, zdither0 int)

DequantizeFloat32 is the float32 variant, writing directly to a float32 slice without an intermediate float64 copy. Used by the FITS-side tile pipeline where the final pixel type is float32.

func FitsRandoms

func FitsRandoms() []float32

FitsRandoms returns the Park-Miller random lookup table used by the dither algorithm. On first call it generates the sequence and verifies the cfitsio invariant that the internal seed equals 1043618065 after exactly NRandom iterations. Thread-safe via sync.Once.

func ReplaceSentinelWithNaNBits

func ReplaceSentinelWithNaNBits(buf []float32, sentinel float32) int

ReplaceSentinelWithNaNBits walks a float32 byte buffer and, wherever the 4-byte value equals the sentinel, overwrites those bytes with the NaN bit pattern. For the lossless NO_QUANTIZE write path where the float tile is gzipped as raw bytes. Ports imcomp_float2nan from reference/cfitsio/imcompress.c:7944.

buf is interpreted as a host-order float32 array (same layout cfitsio uses — it compresses in host byte order and lets the reader byte-swap on decode).

func SubstituteNaN32

func SubstituteNaN32(data []float32, sentinel float32) int

SubstituteNaN32 replaces NaN pixels in place with the given sentinel value, returning the number of substitutions performed. Callers should invoke this before QuantizeFloat32 on any tile that might contain NaN, because the quantizer detects nulls by float equality and NaN != NaN in IEEE 754 — a raw NaN would slip through nullcheck and produce undefined int32 output.

The two-phase "NaN -> sentinel -> NullValueInt32" approach mirrors cfitsio's writer (reference/cfitsio/imcompress.c:2729 imcomp_convert_tile_tfloat, which feeds FLOATNULLVALUE into fits_quantize_float).

Types

type Algorithm

type Algorithm int

Algorithm identifies a tile-compression algorithm.

const (
	// Unknown is the zero value; returned by ParseAlgorithm for unrecognized
	// strings.
	Unknown Algorithm = iota
	// RICE1 is the RICE_1 algorithm (Rice coding with per-block k).
	RICE1
	// GZIP1 is plain zlib DEFLATE over the raw pixel bytes.
	GZIP1
	// GZIP2 is zlib DEFLATE over byte-shuffled pixel data.
	GZIP2
	// HCOMPRESS1 is H-transform + quadtree coding (White 1991).
	HCOMPRESS1
	// PLIO1 is IRAF Planio run-length encoding for integer masks.
	PLIO1
	// NoCompress is an identity pass-through. Used by cfitsio when a tile
	// would inflate under the primary compressor.
	NoCompress
)

func ParseAlgorithm

func ParseAlgorithm(s string) Algorithm

ParseAlgorithm maps a ZCMPTYPE value to an Algorithm. Leading and trailing whitespace is trimmed; the comparison is case-insensitive. Returns Unknown for values not in the standard set.

func (Algorithm) String

func (a Algorithm) String() string

String returns the canonical FITS ZCMPTYPE string for this algorithm.

type Decoder

type Decoder interface {
	Decode(src, dst []byte, nelem, elemSize int) error
}

Decoder decompresses the bytes of a single tile. The same Decoder instance may be reused across multiple tiles in a FITS image — it is stateless with respect to the FITS layer.

src is the compressed bytes for one tile (the contents of one row of the COMPRESSED_DATA variable-length column). dst is a pre-allocated buffer that must be large enough for the uncompressed output: for an integer tile, len(dst) == nelem * elemSize bytes.

On success, dst is filled with big-endian uncompressed pixel bytes in the same byte order FITS uses on disk. nelem is the number of pixels in the tile; elemSize is the on-disk byte size of each pixel (1, 2, 4, or 8) as given by ZBITPIX / BYTEPIX.

func Select

func Select(algo Algorithm, params Params) (Decoder, error)

Select returns the Decoder for the given algorithm, configured from params. Returns ErrUnknownCompression for unknown algorithm strings, or ErrUnsupportedCompression if a future FITS standard revision adds an algorithm this package does not support.

type DitherMethod

type DitherMethod int

DitherMethod identifies the quantization / dither convention used when a float image was compressed to integer coefficients.

const (
	// NoDither — plain quantize, no random offset applied.
	NoDither DitherMethod = 0
	// SubtractiveDither1 — subtract a random [0,1) offset on encode;
	// add it back on decode. Default for RICE_1 float tiles.
	SubtractiveDither1 DitherMethod = 1
	// SubtractiveDither2 — same as SubtractiveDither1 but preserves
	// exact-zero input pixels (encoded as ZeroValue, decoded as 0.0).
	SubtractiveDither2 DitherMethod = 2
)

func ParseDitherMethod

func ParseDitherMethod(s string) DitherMethod

ParseDitherMethod maps the ZQUANTIZ keyword string to a DitherMethod. Unknown or empty values default to NoDither.

type Encoder

type Encoder interface {
	Encode(src, dst []byte, nelem, elemSize int) (int, error)
}

Encoder is the compression counterpart of Decoder. Given a tile's uncompressed bytes in FITS big-endian order, it produces the compressed bytes that should be stored in the COMPRESSED_DATA VLA column for that row.

The caller allocates dst large enough to hold the worst case; the encoder returns the actual number of bytes written. If dst is too small, it returns ErrBufferTooSmall so the caller can retry with a larger buffer or fall back to NOCOMPRESS for that tile.

func SelectEncoder

func SelectEncoder(algo Algorithm, params Params) (Encoder, error)

SelectEncoder returns the Encoder for the given algorithm, configured from params. Mirrors Select() for decoders.

type ImageStatsF32

type ImageStatsF32 struct {
	NGood  int64   // number of non-null pixels
	Min    float32 // minimum non-null value
	Max    float32 // maximum non-null value
	Mean   float64 // mean of non-null pixels
	Sigma  float64 // RMS sigma of non-null pixels
	Noise1 float64 // 1st-order differences, sigma-clipped per row, median of rows
	Noise2 float64 // 2nd-order MAD of non-null pixels
	Noise3 float64 // 3rd-order MAD (primary noise estimator for quantization)
	Noise5 float64 // 5th-order MAD
}

ImageStatsF32 carries the per-tile statistics that fits_img_stats_float fills in. Any field may be zero if the corresponding path in the C function wasn't taken (e.g. sigma-clipping iterations bailing out).

func ImgStatsFloat32

func ImgStatsFloat32(array []float32, nx, ny int64, nullcheck bool, nullvalue float32) ImageStatsF32

ImgStatsFloat32 is the Go counterpart of cfitsio's fits_img_stats_float. array is a 2D image of shape (nx, ny) laid out in row-major order, first axis varying fastest. When nullcheck is true, pixels equal to nullvalue are treated as null and excluded from every statistic.

This function matches cfitsio's numerical output bit-for-bit on the fixtures in testdata/cref/golden/.

type Params

type Params map[string]int64

Params carries the per-algorithm compression parameters (ZNAMEi=ZVALi pairs from the header). Each algorithm extracts what it needs:

RICE_1:      BLOCKSIZE (default 32), BYTEPIX (1/2/4/8)
GZIP_1:      none (byte stream is self-describing)
GZIP_2:      BYTEPIX (derived from ZBITPIX if absent)
HCOMPRESS_1: SCALE, SMOOTH
PLIO_1:      none

Values are int64 to match the FITS integer-card representation.

func (Params) Get

func (p Params) Get(name string, def int64) int64

Get returns the parameter value or def if absent.

type QuantizeResult

type QuantizeResult struct {
	BScale  float64 // FITS BSCALE: one unit in idata == BScale in fdata
	BZero   float64 // FITS BZERO: integer 0 in idata == BZero in fdata
	IMinVal int32   // minimum value actually written to idata (excluding sentinels)
	IMaxVal int32   // maximum value actually written to idata (excluding sentinels)
}

QuantizeResult holds the output of a QuantizeFloat32 call when the quantizer succeeded.

func QuantizeFloat32

func QuantizeFloat32(row int64, fdata []float32, nxpix, nypix int64, nullcheck bool,
	inNullValue float32, qlevel float32, ditherMethod DitherMethod, idata []int32) (QuantizeResult, bool)

QuantizeFloat32 quantizes a float32 tile to int32, producing a per-tile bscale/bzero. The function returns true if the tile was quantized and idata is populated; false if quantization was skipped (constant tile, delta overflow, or too few pixels).

Parameters mirror fits_quantize_float exactly:

row           if > 0 and ditherMethod > 0, used to seed the
              Park-Miller PRNG offset for dither. Matches
              `(row - 1) % NRandom` in cfitsio.
fdata         input pixels, row-major with first axis varying fastest
nxpix, nypix  image shape
nullcheck     if true, fdata[i] == inNullValue is treated as null
inNullValue   null sentinel (cfitsio's approach: caller pre-substitutes
              NaN with a chosen sentinel before calling)
qlevel        positive: delta = stdev / qlevel  (qlevel==0 => stdev/4)
              negative: delta = -qlevel         (absolute step)
ditherMethod  NoDither / SubtractiveDither1 / SubtractiveDither2
idata         caller-allocated output buffer, len(idata) >= nxpix*nypix

The output bscale/bzero are written into the returned QuantizeResult.

This function is intended to be bit-exact against cfitsio. See testdata/cref for the validation harness.