README
¶
go-fil-commp-hashhash
A hash.Hash implementation of fil-commitment-unsealed
Package commp allows calculating a Filecoin Unsealed Commitment (commP/commD) given a bytestream. It is implemented as a standard hash.Hash() interface, with the entire padding and treebuilding algorithm written in golang.
The returned digest is a 32-byte raw commitment payload. Use something like DataCommitmentV1ToCID in order to convert it to a proper cid.Cid.
The output of this library is 100% identical to ffi.GeneratePieceCIDFromFile()
Snapshot Layer Capture
NewCalcWithSnapshot(layerIdx) creates a calculator that captures an intermediate Merkle tree layer during streaming CommP computation.
This is useful for PDP proof caching: the captured layer allows proofs to be generated from small data sections rather than rebuilding the full tree.
cp := commp.NewCalcWithSnapshot(commp.SnapshotLayerIndex(4 << 20)) // ~4 MiB sections
defer cp.Reset()
io.Copy(cp, reader)
commP, paddedSize, snapshot, err := cp.DigestWithSnapshot()
// snapshot.Nodes contains the captured layer hashes
// snapshot is nil if the data was too small for the requested layer
See the GoDoc for details.
Lead Maintainer
License
Documentation
¶
Overview ¶
Package commp allows calculating a Filecoin Unsealed Commitment (commP/commD) given a bytestream. It is implemented as a standard hash.Hash() interface, with the entire padding and treebuilding algorithm written in golang.
The returned digest is a 32-byte raw commitment payload. Use something like https://pkg.go.dev/github.com/filecoin-project/go-fil-commcid#DataCommitmentV1ToCID in order to convert it to a proper cid.Cid.
The output of this library is 100% identical to https://github.com/filecoin-project/filecoin-ffi/blob/d82899449741ce19/proofs.go#L177-L196
Index ¶
- Constants
- func PadCommP(sourceCommP []byte, sourcePaddedSize, targetPaddedSize uint64) ([]byte, error)
- func SnapshotLayerIndex(targetPaddedBytesPerNode uint64) int
- type Calc
- func (cp *Calc) BlockSize() int
- func (cp *Calc) Digest() (commP []byte, paddedPieceSize uint64, err error)
- func (cp *Calc) DigestWithSnapshot() (commP []byte, paddedPieceSize uint64, snapshot *SnapshotLayer, err error)
- func (cp *Calc) Reset()
- func (cp *Calc) Size() int
- func (cp *Calc) Sum(buf []byte) []byte
- func (cp *Calc) Write(input []byte) (int, error)
- type SnapshotLayer
Constants ¶
const MaxLayers = uint(50)
MaxLayers is the maximum height of the merkle tree. This supports pieces up to ~32 PiB padded (2^55 bytes). The cost is a fixed stackedNulPadding array of 50 * 32 = 1600 bytes computed once at init.
const MaxPiecePayload = MaxPieceSize / 128 * 127
MaxPiecePayload is the maximum amount of data that one can Write() to the Calc object, before needing to derive a Digest(). Constrained by the value of MaxLayers.
const MaxPieceSize = uint64(1 << (MaxLayers + 5))
MaxPieceSize is the maximum padded piece size supported by this library.
const MinPiecePayload = uint64(65)
MinPiecePayload is the smallest amount of data for which FR32 padding has a defined result. It is not possible to derive a Digest() before Write()ing at least this amount of bytes.
Variables ¶
This section is empty.
Functions ¶
func SnapshotLayerIndex ¶ added in v0.4.0
SnapshotLayerIndex computes the tree layer index for a target padded section size. Each node at the returned layer covers approximately targetPaddedBytesPerNode bytes of FR32-padded data. The minimum return value is 1 (pairs of leaves, 64 padded bytes per node).
Types ¶
type Calc ¶
type Calc struct {
// contains filtered or unexported fields
}
Calc is an implementation of a commP "hash" calculator, implementing the familiar hash.Hash interface. The zero-value of this object is ready to accept Write()s without further initialization.
Write() starts background goroutines that must be cleaned up. Always defer Reset() to ensure cleanup, especially if Digest() may return an error:
cp := &commp.Calc{}
defer cp.Reset()
if _, err := io.Copy(cp, reader); err != nil {
return err
}
commP, paddedSize, err := cp.Digest()
if err != nil {
return err
}
func NewCalcWithSnapshot ¶ added in v0.4.0
NewCalcWithSnapshot creates a CommP calculator that captures the merkle tree layer at the given index during streaming computation. The minimum layer index is 1 (pairs of leaves); higher indices capture coarser nodes. Values below 1 are clamped.
Each node at layer L covers 2^L leaves, and each leaf is 32 padded bytes, so a node's padded coverage is 32 << L bytes. Use SnapshotLayerIndex to compute the layer for a target section size. For example, layer 17 gives ~4 MiB sections, layer 6 gives ~2 KiB sections.
The layer index is validated against the final tree height during DigestWithSnapshot. If the requested layer exceeds the tree height for the data actually written, DigestWithSnapshot returns a nil snapshot.
func (*Calc) BlockSize ¶
BlockSize is the amount of bytes consumed by the commP algorithm in one go. Write()ing data in multiples of BlockSize would obviate the need to maintain an internal carry buffer. The BlockSize of this module is 127 bytes.
func (*Calc) Digest ¶
Digest collapses the internal hash state and returns the resulting raw 32 bytes of commP and the padded piece size, or alternatively an error in case of insufficient accumulated state. On success, the internal state is reset and all goroutines are terminated. On error, callers must call Reset() to clean up background goroutines before abandoning the Calc object.
func (*Calc) DigestWithSnapshot ¶ added in v0.4.0
func (cp *Calc) DigestWithSnapshot() (commP []byte, paddedPieceSize uint64, snapshot *SnapshotLayer, err error)
DigestWithSnapshot returns the CommP digest and the captured snapshot layer. The Calc must have been created with NewCalcWithSnapshot; returns an error otherwise.
If the requested layer index exceeds the tree height for the data written, the CommP and padded size are still returned but snapshot will be nil (err == nil). This allows callers to always obtain a valid CommP regardless of whether the data was large enough for the requested snapshot granularity.
Calling plain Digest() on a snapshot-enabled Calc discards collected snapshot nodes and returns only the CommP.
On success, the internal state is reset. On error, callers must call Reset() before abandoning the Calc.
func (*Calc) Reset ¶
func (cp *Calc) Reset()
Reset re-initializes the accumulator object, clearing its state and terminating all background goroutines. It is safe to Reset() an accumulator in any state.
func (*Calc) Size ¶
Size is the amount of bytes returned on Sum()/Digest(), which is 32 bytes for this module.
func (*Calc) Sum ¶
Sum is a thin wrapper around Digest() and is provided solely to satisfy the hash.Hash interface. It panics on errors returned from Digest(). Note that unlike classic (hash.Hash).Sum(), calling this method is destructive: the internal state is reset and all goroutines kicked off by Write() are terminated.
func (*Calc) Write ¶
Write adds bytes to the accumulator, for a subsequent Digest(). Upon the first call of this method a few goroutines are started in the background to service each layer of the digest tower. If you wrote some data and then decide to abandon the object without invoking Digest(), you need to call Reset() to terminate all remaining background workers. Unlike a typical (hash.Hash).Write, calling this method can return an error when the total amount of bytes is about to go over the maximum currently supported by Filecoin.
type SnapshotLayer ¶ added in v0.4.0
type SnapshotLayer struct {
// LayerIndex is the tree layer that was captured (0 = leaf level).
LayerIndex int
// Nodes contains the 32-byte hashes in sequential order, zero-padded to
// the expected count for pieces that don't fill their padded size.
Nodes [][32]byte
}
SnapshotLayer holds a captured intermediate merkle tree layer from a streaming CommP computation. Each node is the root hash of a subtree covering a contiguous section of the piece data.
Source Files
Directories