packfile

package
v6.0.0-...-675edb8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 20, 2025 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package packfile implements encoding and decoding of packfile format. 

== pack-*.pack files have the following format:

  - A header appears at the beginning and consists of the following:

    4-byte signature:
        The signature is: {'P', 'A', 'C', 'K'}

    4-byte version number (network byte order):
        GIT currently accepts version number 2 or 3 but
        generates version 2 only.

    4-byte number of objects contained in the pack (network byte order)

    Observation: we cannot have more than 4G versions ;-) and
    more than 4G objects in a pack.

  - The header is followed by number of object entries, each of
    which looks like this:

    (undeltified representation)
    n-byte type and length (3-bit type, (n-1)*7+4-bit length)
    compressed data

    (deltified representation)
    n-byte type and length (3-bit type, (n-1)*7+4-bit length)
    20-byte base object name
    compressed delta data

    Observation: length of each object is encoded in a variable
    length format and is not constrained to 32-bit or anything.

 - The trailer records 20-byte SHA1 checksum of all of the above.

Source: https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-protocol.txt

Index

Constants

View Source 
const (
	// VersionSupported is the packfile version supported by this package
	VersionSupported uint32 = 2
)

Variables

View Source 
var (
	// ErrInvalidObject is returned by Decode when an invalid object is
	// found in the packfile.
	ErrInvalidObject = NewError("invalid git object")
	// ErrZLib is returned by Decode when there was an error unzipping
	// the packfile contents.
	ErrZLib = NewError("zlib reading error")
)
View Source 
var (
	// ErrReferenceDeltaNotFound is returned when the reference delta is not
	// found.
	ErrReferenceDeltaNotFound = errors.New("reference delta not found")

	// ErrNotSeekableSource is returned when the source for the parser is not
	// seekable and a storage was not provided, so it can't be parsed.
	ErrNotSeekableSource = errors.New("parser source is not seekable and storage was not provided")

	// ErrDeltaNotCached is returned when the delta could not be found in cache.
	ErrDeltaNotCached = errors.New("delta could not be found in cache")
)
View Source 
var (
	ErrInvalidDelta = errors.New("invalid delta")
	ErrDeltaCmd     = errors.New("wrong delta command")
)
View Source 
var (
	// ErrEmptyPackfile is returned by ReadHeader when no data is found in the packfile.
	ErrEmptyPackfile = NewError("empty packfile")
	// ErrBadSignature is returned by ReadHeader when the signature in the packfile is incorrect.
	ErrBadSignature = NewError("malformed pack file signature")
	// ErrMalformedPackfile is returned when the packfile format is incorrect.
	ErrMalformedPackfile = NewError("malformed pack file")
	// ErrUnsupportedVersion is returned by ReadHeader when the packfile version is
	// different than VersionSupported.
	ErrUnsupportedVersion = NewError("unsupported packfile version")
	// ErrSeekNotSupported returned if seek is not support.
	ErrSeekNotSupported = NewError("not seek support")
)
View Source 
var T = []uint32{}/* 256 elements not displayed */

Functions

func ApplyDelta 

func ApplyDelta(target, base plumbing.EncodedObject, delta []byte) (err error)

ApplyDelta writes to target the result of applying the modification deltas in delta to base.

func DiffDelta 

func DiffDelta(src, tgt []byte) []byte

DiffDelta returns the delta that transforms src into tgt.

func GetDelta 

func GetDelta(base, target plumbing.EncodedObject) (plumbing.EncodedObject, error)

GetDelta returns an EncodedObject of type OFSDeltaObject. Base and Target object, will be loaded into memory to be able to create the delta object. To generate target again, you will need the obtained object and "base" one. Error will be returned if base or target object cannot be read.

func PatchDelta 

func PatchDelta(src, delta []byte) ([]byte, error)

PatchDelta returns the result of applying the modification deltas in delta to src. An error will be returned if delta is corrupted (ErrInvalidDelta) or an action command is not copy from source or copy from delta (ErrDeltaCmd).

func ReaderFromDelta 

func ReaderFromDelta(base plumbing.EncodedObject, deltaRC io.Reader) (io.ReadCloser, error)

func UpdateObjectStorage 

func UpdateObjectStorage(s storer.Storer, packfile io.Reader) error

UpdateObjectStorage updates the storer with the objects in the given packfile.

func WritePackfileToObjectStorage 

func WritePackfileToObjectStorage(
	sw storer.PackfileWriter,
	packfile io.Reader,
) (err error)

WritePackfileToObjectStorage writes all the packfile objects into the given object storage.

Types

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder gets the data from the storage and write it into the writer in PACK format

func NewEncoder 

func NewEncoder(w io.Writer, s storer.EncodedObjectStorer, useRefDeltas bool) *Encoder

NewEncoder creates a new packfile encoder using a specific Writer and EncodedObjectStorer. By default deltas used to generate the packfile will be OFSDeltaObject. To use Reference deltas, set useRefDeltas to true.

func (*Encoder) Encode 

func (e *Encoder) Encode(
	hashes []plumbing.Hash,
	packWindow uint,
) (plumbing.Hash, error)

Encode creates a packfile containing all the objects referenced in hashes and writes it to the writer in the Encoder. `packWindow` specifies the size of the sliding window used to compare objects for delta compression; 0 turns off delta compression entirely.

type Error 

type Error struct {
	// contains filtered or unexported fields
}

Error specifies errors returned during packfile parsing.

func NewError 

func NewError(reason string) *Error

NewError returns a new error.

func (*Error) AddDetails 

func (e *Error) AddDetails(format string, args ...interface{}) *Error

AddDetails adds details to an error, with additional text.

func (*Error) Error 

func (e *Error) Error() string

Error returns a text representation of the error.

type FSObject 

type FSObject struct {
	// contains filtered or unexported fields
}

FSObject is an object from the packfile on the filesystem.

func NewFSObject 

func NewFSObject(
	hash plumbing.Hash,
	finalType plumbing.ObjectType,
	offset int64,
	contentSize int64,
	index idxfile.Index,
	fs billy.Filesystem,
	pack billy.File,
	packPath string,
	cache cache.Object,
) *FSObject

NewFSObject creates a new filesystem object.

func (*FSObject) Hash 

func (o *FSObject) Hash() plumbing.Hash

Hash implements the plumbing.EncodedObject interface.

func (*FSObject) Reader 

func (o *FSObject) Reader() (io.ReadCloser, error)

Reader implements the plumbing.EncodedObject interface.

func (*FSObject) SetSize 

func (o *FSObject) SetSize(int64)

SetSize implements the plumbing.EncodedObject interface. This method is a noop.

func (*FSObject) SetType 

func (o *FSObject) SetType(plumbing.ObjectType)

SetType implements the plumbing.EncodedObject interface. This method is a noop.

func (*FSObject) Size 

func (o *FSObject) Size() int64

Size implements the plumbing.EncodedObject interface.

func (*FSObject) Type 

func (o *FSObject) Type() plumbing.ObjectType

Type implements the plumbing.EncodedObject interface.

func (*FSObject) Writer 

func (o *FSObject) Writer() (io.WriteCloser, error)

Writer implements the plumbing.EncodedObject interface. This method always returns a nil writer. 

type Header struct {
	Version    Version
	ObjectsQty uint32
}

type ObjectHeader 

type ObjectHeader struct {
	Type            plumbing.ObjectType
	Offset          int64
	ContentOffset   int64
	Size            int64
	Reference       plumbing.Hash
	OffsetReference int64
	Crc32           uint32
	Hash            plumbing.Hash
	Hash256         *plumbing.Hash256
	// contains filtered or unexported fields
}

ObjectHeader contains the information related to the object, this information is collected from the previous bytes to the content of the object.

type ObjectToPack 

type ObjectToPack struct {
	// The main object to pack, it could be any object, including deltas.
	Object plumbing.EncodedObject
	// Base is the object that a delta is based on, which could also be another delta.
	// Nil when the main object is not a delta.
	Base *ObjectToPack
	// Original is the object that we can generate applying the delta to
	// Base, or the same object as Object in the case of a non-delta
	// object.
	Original plumbing.EncodedObject
	// Depth is the amount of deltas needed to resolve to obtain Original
	// (delta based on delta based on ...)
	Depth int

	// offset in pack when object has been already written, or 0 if it
	// has not been written yet
	Offset int64
	// contains filtered or unexported fields
}

ObjectToPack is a representation of an object that is going to be into a pack file.

func (*ObjectToPack) BackToOriginal 

func (o *ObjectToPack) BackToOriginal()

BackToOriginal converts that ObjectToPack to a non-deltified object if it was one

func (*ObjectToPack) CleanOriginal 

func (o *ObjectToPack) CleanOriginal()

CleanOriginal sets Original to nil

func (*ObjectToPack) Hash 

func (o *ObjectToPack) Hash() plumbing.Hash

func (*ObjectToPack) IsDelta 

func (o *ObjectToPack) IsDelta() bool

func (*ObjectToPack) IsWritten 

func (o *ObjectToPack) IsWritten() bool

IsWritten returns if that ObjectToPack was already written into the packfile or not

func (*ObjectToPack) MarkWantWrite 

func (o *ObjectToPack) MarkWantWrite()

MarkWantWrite marks this ObjectToPack as WantWrite to avoid delta chain loops

func (*ObjectToPack) SaveOriginalMetadata 

func (o *ObjectToPack) SaveOriginalMetadata()

SaveOriginalMetadata saves size, type and hash of Original object

func (*ObjectToPack) SetDelta 

func (o *ObjectToPack) SetDelta(base *ObjectToPack, delta plumbing.EncodedObject)

func (*ObjectToPack) SetOriginal 

func (o *ObjectToPack) SetOriginal(obj plumbing.EncodedObject)

SetOriginal sets both Original and saves size, type and hash. If object is nil Original is set but previous resolved values are kept

func (*ObjectToPack) Size 

func (o *ObjectToPack) Size() int64

func (*ObjectToPack) Type 

func (o *ObjectToPack) Type() plumbing.ObjectType

func (*ObjectToPack) WantWrite 

func (o *ObjectToPack) WantWrite() bool

WantWrite checks if this ObjectToPack was marked as WantWrite before

type Observer 

type Observer interface {
	// OnHeader is called when a new packfile is opened.
	OnHeader(count uint32) error
	// OnInflatedObjectHeader is called for each object header read.
	OnInflatedObjectHeader(t plumbing.ObjectType, objSize int64, pos int64) error
	// OnInflatedObjectContent is called for each decoded object.
	OnInflatedObjectContent(h plumbing.Hash, pos int64, crc uint32, content []byte) error
	// OnFooter is called when decoding is done.
	OnFooter(h plumbing.Hash) error
}

Observer interface is implemented by index encoders.

type PackData 

type PackData struct {
	Section SectionType
	// contains filtered or unexported fields
}

func (PackData) Value 

func (p PackData) Value() interface{}

type Packfile 

type Packfile struct {
	idxfile.Index
	// contains filtered or unexported fields
}

Packfile allows retrieving information from inside a packfile.

func NewPackfile 

func NewPackfile(
	file billy.File,
	opts ...PackfileOption,
) *Packfile

NewPackfile returns a packfile representation for the given packfile file and packfile idx. If the filesystem is provided, the packfile will return FSObjects, otherwise it will return MemoryObjects.

func (*Packfile) Close 

func (p *Packfile) Close() error

Close the packfile and its resources.

func (*Packfile) Get 

func (p *Packfile) Get(h plumbing.Hash) (plumbing.EncodedObject, error)

Get retrieves the encoded object in the packfile with the given hash.

func (*Packfile) GetAll 

func (p *Packfile) GetAll() (storer.EncodedObjectIter, error)

GetAll returns an iterator with all encoded objects in the packfile. The iterator returned is not thread-safe, it should be used in the same thread as the Packfile instance.

func (*Packfile) GetByOffset 

func (p *Packfile) GetByOffset(offset int64) (plumbing.EncodedObject, error)

GetByOffset retrieves the encoded object from the packfile at the given offset.

func (*Packfile) GetByType 

func (p *Packfile) GetByType(typ plumbing.ObjectType) (storer.EncodedObjectIter, error)

GetByType returns all the objects of the given type.

func (*Packfile) GetSizeByOffset 

func (p *Packfile) GetSizeByOffset(offset int64) (size int64, err error)

GetSizeByOffset retrieves the size of the encoded object from the packfile with the given offset.

func (*Packfile) ID 

func (p *Packfile) ID() (plumbing.Hash, error)

ID returns the ID of the packfile, which is the checksum at the end of it.

func (*Packfile) Scanner deprecated 

func (p *Packfile) Scanner() (*Scanner, error)

Returns the Packfile's inner scanner.

Deprecated: this will be removed in future versions of the packfile package to avoid exposing the package internals and to improve its thread-safety.

type PackfileOption 

type PackfileOption func(*Packfile)

func WithCache 

func WithCache(cache cache.Object) PackfileOption

WithCache sets the cache to be used throughout Packfile operations. Use this to share existing caches with the Packfile. If not used, a new cache instance will be created.

func WithFs 

func WithFs(fs billy.Filesystem) PackfileOption

WithFs sets the filesystem to be used.

func WithIdx 

func WithIdx(idx idxfile.Index) PackfileOption

WithIdx sets the idxfile for the packfile.

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

Parser decodes a packfile and calls any observer associated to it. Is used to generate indexes.

func NewParser 

func NewParser(data io.Reader, opts ...ParserOption) *Parser

NewParser creates a new Parser. When a storage is set, the objects are written to storage as they are parsed.

func (*Parser) Parse 

func (p *Parser) Parse() (plumbing.Hash, error)

Parse start decoding phase of the packfile.

type ParserOption 

type ParserOption func(*Parser)

func WithScannerObservers 

func WithScannerObservers(ob ...Observer) ParserOption

WithScannerObservers sets the observers to be notified during the scanning or parsing of a pack file. The scanner is responsible for notifying observers around general pack file information, such as header and footer. The scanner also notifies object headers for non-delta objects.

Delta objects are notified as part of the parser logic.

func WithStorage 

func WithStorage(storage storer.EncodedObjectStorer) ParserOption

WithStorage sets the storage to be used while parsing a pack file.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner provides sequential access to the data stored in a Git packfile.

A Git packfile is a compressed binary format that stores multiple Git objects, such as commits, trees, delta objects and blobs. These packfiles are used to reduce the size of data when transferring or storing Git repositories.

A Git packfile is structured as follows: 

+----------------------------------------------------+
|                 PACK File Header                   |
+----------------------------------------------------+
| "PACK"  | Version Number | Number of Objects       |
| (4 bytes)  |  (4 bytes)   |    (4 bytes)           |
+----------------------------------------------------+
|                  Object Entry #1                   |
+----------------------------------------------------+
|  Object Header  |  Compressed Object Data / Delta  |
| (type + size)   |  (var-length, zlib compressed)   |
+----------------------------------------------------+
|                         ...                        |
+----------------------------------------------------+
|                  PACK File Footer                  |
+----------------------------------------------------+
|                SHA-1 Checksum (20 bytes)           |
+----------------------------------------------------+

For upstream docs, refer to https://git-scm.com/docs/gitformat-pack.

func NewScanner 

func NewScanner(rs io.Reader, opts ...ScannerOption) *Scanner

NewScanner creates a new instance of Scanner.

func (*Scanner) Data 

func (r *Scanner) Data() PackData

Data returns the pack data based on the last call to Scan().

func (*Scanner) Error 

func (r *Scanner) Error() error

Data returns the first error that occurred on the last call to Scan(). Once an error occurs, calls to Scan() becomes a no-op.

func (Scanner) Flush 

func (r Scanner) Flush() error

func (Scanner) Read 

func (r Scanner) Read(p []byte) (n int, err error)

func (Scanner) ReadByte 

func (r Scanner) ReadByte() (b byte, err error)

func (*Scanner) Reset 

func (r *Scanner) Reset()

Reset resets the current scanner, enabling it to be used to scan the same Packfile again.

func (*Scanner) Scan 

func (r *Scanner) Scan() bool

Scan scans a Packfile sequently. Each call will navigate from a section to the next, until the entire file is read.

The section data can be accessed via calls to Data(). Example: 

for scanner.Scan() {
    v := scanner.Data().Value()

	switch scanner.Data().Section {
	case HeaderSection:
		header := v.(Header)
		fmt.Println("[Header] Objects Qty:", header.ObjectsQty)
	case ObjectSection:
		oh := v.(ObjectHeader)
		fmt.Println("[Object] Object Type:", oh.Type)
	case FooterSection:
		checksum := v.(plumbing.Hash)
		fmt.Println("[Footer] Checksum:", checksum)
	}
}

func (Scanner) Seek 

func (r Scanner) Seek(offset int64, whence int) (int64, error)

Seek seeks to a location. If the underlying reader is not an io.ReadSeeker, then only whence=io.SeekCurrent is supported, any other operation fails.

func (*Scanner) SeekFromStart 

func (r *Scanner) SeekFromStart(offset int64) error

func (*Scanner) WriteObject 

func (s *Scanner) WriteObject(oh *ObjectHeader, writer io.Writer) error

type ScannerOption 

type ScannerOption func(*Scanner)

func WithSHA256 

func WithSHA256() ScannerOption

WithSHA256 enables the SHA256 hashing while scanning a pack file.

type SectionType 

type SectionType int
const (
	HeaderSection SectionType = iota
	ObjectSection
	FooterSection
)

type Version 

type Version uint32
const (
	V2 Version = 2
)

func (Version) Supported 

func (v Version) Supported() bool

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.