Documentation

Overview

    Package idxfile implements encoding and decoding of packfile idx files.

    == Original (version 1) pack-*.idx files have the following format:
    
      - The header consists of 256 4-byte network byte order
        integers.  N-th entry of this table records the number of
        objects in the corresponding pack, the first byte of whose
        object name is less than or equal to N.  This is called the
        'first-level fan-out' table.
    
      - The header is followed by sorted 24-byte entries, one entry
        per object in the pack.  Each entry is:
    
       4-byte network byte order integer, recording where the
       object is stored in the packfile as the offset from the
       beginning.
    
       20-byte object name.
    
     - The file is concluded with a trailer:
    
       A copy of the 20-byte SHA1 checksum at the end of
       corresponding packfile.
    
       20-byte SHA1-checksum of all of the above.
    
     Pack Idx file:
    
          --  +--------------------------------+
     fanout   | fanout[0] = 2 (for example)    |-.
     table    +--------------------------------+ |
              | fanout[1]                      | |
              +--------------------------------+ |
              | fanout[2]                      | |
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
              | fanout[255] = total objects    |---.
          --  +--------------------------------+ | |
     main     | offset                         | | |
     index    | object name 00XXXXXXXXXXXXXXXX | | |
     tab      +--------------------------------+ | |
              | offset                         | | |
              | object name 00XXXXXXXXXXXXXXXX | | |
              +--------------------------------+<+ |
            .-| offset                         |   |
            | | object name 01XXXXXXXXXXXXXXXX |   |
            | +--------------------------------+   |
            | | offset                         |   |
            | | object name 01XXXXXXXXXXXXXXXX |   |
            | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |
            | | offset                         |   |
            | | object name FFXXXXXXXXXXXXXXXX |   |
          --| +--------------------------------+<--+
    trailer | | packfile checksum              |
            | +--------------------------------+
            | | idxfile checksum               |
            | +--------------------------------+
            .---------.
                      |
    Pack file entry: <+
    
       packed object header:
       1-byte size extension bit (MSB)
             type (next 3 bit)
             size0 (lower 4-bit)
           n-byte sizeN (as long as MSB is set, each 7-bit)
           size0..sizeN form 4+7+7+..+7 bit integer, size0
           is the least significant part, and sizeN is the
           most significant part.
       packed object data:
           If it is not DELTA, then deflated bytes (the size above
           is the size before compression).
       If it is REF_DELTA, then
         20-byte base object name SHA1 (the size above is the
           size of the delta data that follows).
             delta data, deflated.
       If it is OFS_DELTA, then
         n-byte offset (see below) interpreted as a negative
           offset from the type-byte of the header of the
           ofs-delta entry (the size above is the size of
           the delta data that follows).
         delta data, deflated.
    
       offset encoding:
         n bytes with MSB set in all but the last one.
         The offset is then the number constructed by
         concatenating the lower 7 bit of each byte, and
         for n >= 2 adding 2^7 + 2^14 + ... + 2^(7*(n-1))
         to the result.
    
     == Version 2 pack-*.idx files support packs larger than 4 GiB, and
        have some other reorganizations.  They have the format:
    
       - A 4-byte magic number '\377tOc' which is an unreasonable
         fanout[0] value.
    
       - A 4-byte version number (= 2)
    
       - A 256-entry fan-out table just like v1.
    
       - A table of sorted 20-byte SHA1 object names.  These are
         packed together without offset values to reduce the cache
         footprint of the binary search for a specific object name.
    
       - A table of 4-byte CRC32 values of the packed object data.
         This is new in v2 so compressed data can be copied directly
         from pack to pack during repacking without undetected
         data corruption.
    
       - A table of 4-byte offset values (in network byte order).
         These are usually 31-bit pack file offsets, but large
         offsets are encoded as an index into the next table with
         the msbit set.
    
       - A table of 8-byte offset entries (empty for pack files less
         than 2 GiB).  Pack files are organized with heavily used
         objects toward the front, so most object references should
         not need to refer to this table.
    
       - The same trailer as a v1 pack file:
    
         A copy of the 20-byte SHA1 checksum at the end of
         corresponding packfile.
    
         20-byte SHA1-checksum of all of the above.
    

    Source: https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-format.txt

    Index

    Constants

    View Source
    const (
    	// VersionSupported is the only idx version supported.
    	VersionSupported = 2
    )

    Variables

    View Source
    var (
    	// ErrUnsupportedVersion is returned by Decode when the idx file version
    	// is not supported.
    	ErrUnsupportedVersion = errors.New("Unsuported version")
    	// ErrMalformedIdxFile is returned by Decode when the idx file is corrupted.
    	ErrMalformedIdxFile = errors.New("Malformed IDX file")
    )

    Functions

    This section is empty.

    Types

    type Decoder

    type Decoder struct {
    	*bufio.Reader
    }

      Decoder reads and decodes idx files from an input stream.

      func NewDecoder

      func NewDecoder(r io.Reader) *Decoder

        NewDecoder builds a new idx stream decoder, that reads from r.

        func (*Decoder) Decode

        func (d *Decoder) Decode(idx *MemoryIndex) error

          Decode reads from the stream and decode the content into the MemoryIndex struct.

          type Encoder

          type Encoder struct {
          	io.Writer
          	// contains filtered or unexported fields
          }

            Encoder writes MemoryIndex structs to an output stream.

            func NewEncoder

            func NewEncoder(w io.Writer) *Encoder

              NewEncoder returns a new stream encoder that writes to w.

              func (*Encoder) Encode

              func (e *Encoder) Encode(idx *MemoryIndex) (int, error)

                Encode encodes an MemoryIndex to the encoder writer.

                type Entry

                type Entry struct {
                	Hash   plumbing.Hash
                	CRC32  uint32
                	Offset uint64
                }

                  Entry is the in memory representation of an object entry in the idx file.

                  type EntryIter

                  type EntryIter interface {
                  	// Next returns the next entry in the packfile index.
                  	Next() (*Entry, error)
                  	// Close closes the iterator.
                  	Close() error
                  }

                    EntryIter is an iterator that will return the entries in a packfile index.

                    type Index

                    type Index interface {
                    	// Contains checks whether the given hash is in the index.
                    	Contains(h plumbing.Hash) (bool, error)
                    	// FindOffset finds the offset in the packfile for the object with
                    	// the given hash.
                    	FindOffset(h plumbing.Hash) (int64, error)
                    	// FindCRC32 finds the CRC32 of the object with the given hash.
                    	FindCRC32(h plumbing.Hash) (uint32, error)
                    	// FindHash finds the hash for the object with the given offset.
                    	FindHash(o int64) (plumbing.Hash, error)
                    	// Count returns the number of entries in the index.
                    	Count() (int64, error)
                    	// Entries returns an iterator to retrieve all index entries.
                    	Entries() (EntryIter, error)
                    	// EntriesByOffset returns an iterator to retrieve all index entries ordered
                    	// by offset.
                    	EntriesByOffset() (EntryIter, error)
                    }

                      Index represents an index of a packfile.

                      type MemoryIndex

                      type MemoryIndex struct {
                      	Version uint32
                      	Fanout  [256]uint32
                      	// FanoutMapping maps the position in the fanout table to the position
                      	// in the Names, Offset32 and CRC32 slices. This improves the memory
                      	// usage by not needing an array with unnecessary empty slots.
                      	FanoutMapping    [256]int
                      	Names            [][]byte
                      	Offset32         [][]byte
                      	CRC32            [][]byte
                      	Offset64         []byte
                      	PackfileChecksum [20]byte
                      	IdxChecksum      [20]byte
                      	// contains filtered or unexported fields
                      }

                        MemoryIndex is the in memory representation of an idx file.

                        func NewMemoryIndex

                        func NewMemoryIndex() *MemoryIndex

                          NewMemoryIndex returns an instance of a new MemoryIndex.

                          func (*MemoryIndex) Contains

                          func (idx *MemoryIndex) Contains(h plumbing.Hash) (bool, error)

                            Contains implements the Index interface.

                            func (*MemoryIndex) Count

                            func (idx *MemoryIndex) Count() (int64, error)

                              Count implements the Index interface.

                              func (*MemoryIndex) Entries

                              func (idx *MemoryIndex) Entries() (EntryIter, error)

                                Entries implements the Index interface.

                                func (*MemoryIndex) EntriesByOffset

                                func (idx *MemoryIndex) EntriesByOffset() (EntryIter, error)

                                  EntriesByOffset implements the Index interface.

                                  func (*MemoryIndex) FindCRC32

                                  func (idx *MemoryIndex) FindCRC32(h plumbing.Hash) (uint32, error)

                                    FindCRC32 implements the Index interface.

                                    func (*MemoryIndex) FindHash

                                    func (idx *MemoryIndex) FindHash(o int64) (plumbing.Hash, error)

                                      FindHash implements the Index interface.

                                      func (*MemoryIndex) FindOffset

                                      func (idx *MemoryIndex) FindOffset(h plumbing.Hash) (int64, error)

                                        FindOffset implements the Index interface.

                                        type Writer

                                        type Writer struct {
                                        	// contains filtered or unexported fields
                                        }

                                          Writer implements a packfile Observer interface and is used to generate indexes.

                                          func (*Writer) Add

                                          func (w *Writer) Add(h plumbing.Hash, pos uint64, crc uint32)

                                            Add appends new object data.

                                            func (*Writer) Finished

                                            func (w *Writer) Finished() bool

                                            func (*Writer) Index

                                            func (w *Writer) Index() (*MemoryIndex, error)

                                              Index returns a previously created MemoryIndex or creates a new one if needed.

                                              func (*Writer) OnFooter

                                              func (w *Writer) OnFooter(h plumbing.Hash) error

                                                OnFooter implements packfile.Observer interface.

                                                func (*Writer) OnHeader

                                                func (w *Writer) OnHeader(count uint32) error

                                                  OnHeader implements packfile.Observer interface.

                                                  func (*Writer) OnInflatedObjectContent

                                                  func (w *Writer) OnInflatedObjectContent(h plumbing.Hash, pos int64, crc uint32, _ []byte) error

                                                    OnInflatedObjectContent implements packfile.Observer interface.

                                                    func (*Writer) OnInflatedObjectHeader

                                                    func (w *Writer) OnInflatedObjectHeader(t plumbing.ObjectType, objSize int64, pos int64) error

                                                      OnInflatedObjectHeader implements packfile.Observer interface.