provider

README

Index Provider 📢

A golang implementation of index provider

This repo provides a reference index provider implementation that can be used to advertise content to indexer nodes and serve retrieval requests over graphsync both as a standalone service or embedded into an existing Golang application via a reusable library.

A list of features include:

• provider CLI that can:
  • Run as a standalone provider daemon instance.
  • Generate and publish indexing advertisements directly from CAR files.
  • Serve retrieval requests for the advertised content over GraphSync.
  • list advertisements published by a provider instance
  • verify ingestion of multihashes by an indexer node from CAR files, detached CARv2 indices or from an index provider's advertisement chain.
• A Golang SDK to embed indexing integration into existing applications, which includes:
  • Programmatic advertisement for content via index provider Engine with built-in chunking functionality
  • Announcement of changes to the advertised content over GossipSub using go-legs
  • MultihashLister integration point for fully customizable look up of advertised multihashes.
  • Utilities to advertise multihashes directly from CAR files or detached CARv2 index files.
  • Index advertisement metadata schema for retrieval over graphsync and bitswap

Current status 🚧

This implementation is under active development.

Background

The protocol implemented by this repository is the index provider portion of a larger indexing protocol documented here . The indexer node implementation can be found at storetheindex repository.

For more details on the ingestion protocol itself see Providing data to a network indexer .

Install

Prerequisite:

• Go 1.16+

To use the provider as a Go library, execute:

go get github.com/filecoin-project/index-provider

To install the latest provider CLI, run:

go install github.com/filecoin-project/index-provider/cmd/provider@latest

Alternatively, download the executables directly from the releases.

Usage

Running an standalone provider daemon

To run a provider service first initialize it by executing:

provider init

Initialization generates a default configuration for the provider instance along with a randomly generated identity keypair. The configuration is stored at user home under .index-provider/config in JSON format. The root configuration path can be overridden by setting the PROVIDER_PATH environment variable

Once initialized, start the service daemon by executing:

provider daemon

The running daemon allows advertisement for new content to the indexer nodes and retrieval of content over GraphSync. Additionally, it starts an admin HTTP server that enables administrative operations using the provider CLI tool. By default, the admin server is bound to http://localhost:3102.

You can then advertise content by importing/removing CAR files via the provider CLI, for example:

provider import car -l http://localhost:3102 -i <path-to-car-file>

Both CARv1 and CARv2 formats are supported. Index is regenerated on the fly if one is not present.

Exposing reframe server from provider (experimental)

Provider can export a reframe server. Reframe is a protocol that allows IPFS nodes to advertise their contents to indexers alongside DHT. Reframe server is off by default. To enable it, add the following configuration block to the provider config file.

{
  ...
  Reframe {
    ListenMultiaddr: "/ip4/0.0.0.0/tcp/50617 (example)"
  }
  ...
}

Embedding index provider integration

The root go module offers a set of reusable libraries that can be used to embed index provider support into existing application. The core provider.Interface is implemented by engine.Engine.

The provider Engine exposes a set of APIs that allows a user to programmatically announce the availability or removal of content to the indexer nodes referred to as “advertisement”. Advertisements are represented as an IPLD DAG, chained together via a link to the previous advertisement. An advertisement effectively captures the "diff" of the content that is either added or is no longer provided.

Each advertisement contains:

• Provider ID: the libp2p peer ID of the content provider.
• Addresses: a list of addresses from which the content can be retrieved.
• Metadata: a blob of bytes capturing how to retrieve the data.
• Entries: a link pointing to a list of chunked multihashes.
• Context ID: a key for the content being advertised.

The Entries link points to the IPLD node that contains a list of mulitihashes being advertised. The list is represented as a chain of "Entry Chunk"s where each chunk contains a list of multihashes and a link to the next chunk. This is to accommodate pagination for a large number of multihashes.

The engine can be configured to dynamically look up the list of multihashes that correspond to the context ID of an advertisement. To do this, the engine requires a MultihashLister to be registered. The MultihashLister is then used to look up the list of multihashes associated to a content advertisement. For an example on how to start up a provider engine, register a lister and advertise content, see:

• engine/example_test.go

Publishing ads with extended providers

Extended providers field allows for specification of provider families, in cases where a provider operates multiple PeerIDs, perhaps with different transport protocols between them, but over the same database of content.

Such ads can be composed manually or using a convenience builder ExtendedProvidersAdBuilder.

  adv, err := ep.NewExtendedProviderAdBuilder(providerID, priv, addrs). // the main ad's providerID, private key and addresses
    WithContextID(contextID). // optional context id
    WithMetadata(metadata). // optional metadata
    WithOverride(override). // override flag, false by default
    WithExtendedProviders(extendedProviders). // one or more extended providers to be included in the ad, represented by ExtendedProviderInfo struct
    WithLastAdID(lastAdId). // cid of the last published ad, which is false by default
    BuildAndSign()

  if err != nil {
    //...
  }

  engine.Publish(ctx, *adv)
)

Identity of the main provider will be added to the extended providers list automatically and should not be passed in explicitly

provider CLI

The provider CLI can be used to interact with a running daemon via the admin server to perform a range of administrative operations. For example, the provider CLI can be used to import a CAR file and advertise its content to the indexer nodes by executing:

provider import car -l http://localhost:3102 -i <path-to-car-file>

For full usage, execute provider. Usage:

NAME:
   provider - Indexer Reference Provider Implementation

USAGE:
   provider [global options] command [command options] [arguments...]

VERSION:
   v0.2.7

COMMANDS:
   daemon             Starts a reference provider
   find               Query an indexer for indexed content
   index              Push a single content index into an indexer
   init               Initialize reference provider config file and identity
   connect            Connects to an indexer through its multiaddr
   import, i          Imports sources of multihashes to the index provider.
   register           Register provider information with an indexer that trusts the provider
   remove, rm         Removes previously advertised multihashes by the provider.
   verify-ingest, vi  Verifies ingestion of multihashes to an indexer node from a CAR file or a CARv2 Index
   list               Lists advertisements
   help, h            Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help (default: false)
   --version, -v  print the version (default: false)

Storage Consumption

The index provider engine uses a given datastore to persist two general category of data:

1. Internal advertisement mappings, and
2. Chunked entries chain cache

If the datastore passed to the engine is reused, it is recommended to wrap it in a namespace prior to instantiating the engine.

Internal advertisement mappings

The internal advertisement mappings are purely used by the engine to efficiently handle publication requests. It generally includes:

• mapping to the latest advertisement
• mappings between advertisement CIDs, their context ID and their corresponding metadata.

The storage consumed by such mappings is negligible and grows linearly as a factor of the number of advertisements published.

Chunked entries chain cache

This category stores chunked entries generated by publishing an advertisement with a never seen before context ID. The chunks are stored in an LRU cache, the maximum size of which is configured by the following configuration parameters in Ingest config:

• LinkChunkSize - The maximum number of multihashes in a chunk (defaults to 16,384)
• LinkCacheSize - The maximum number of entries links to chace (defaults to 1024)

The exact storage usage depends on the size of multihashes. For example, using the default config to advertise 128-bit long multihashes will result in chunk sizes of 0.25MiB with maximum cache growth of 256 MiB.

To delete the cache set PurgeLinkCache to true and restart the engine.

Note that the LRU cache may grow beyond its max size if the generated chain of chunks is longer than the configured LinkChunkSize. This is to avoid partial caching of chunks within a single advertisement. The cache expansion is logged in INFO level at provider/engine logging subsystem.

Related Resources

• Indexer Ingestion Interface
• Indexer Ingestion IPLD Schema
• Indexer Node Design
• Providing data to a network indexer
• storetheindex: indexer node implementation
• storetheindex documentation
• go-indexer-core: Core index key-value store

License

SPDX-License-Identifier: Apache-2.0 OR MIT

Documentation

Overview

Package provider represents a reference implementation of an index provider. It integrates with the indexer node protocol, "storetheinex" in order to advertise the availability of a list of multihashes as an IPLD DAG. For the complete advertisement IPLD schema, see:

• https://github.com/filecoin-project/storetheindex/blob/main/api/v0/ingest/schema/schema.ipldsch

A reference implementation of provider.Interface can be found in engine.Engine.

Index

• Variables
• type Interface
• type MultihashIterator
  • func CarMultihashIterator(idx carindex.IterableIndex) (MultihashIterator, error)
  • func EntryChunkMultihashIterator(l ipld.Link, ls ipld.LinkSystem) (MultihashIterator, error)
  • func HamtMultihashIterator(root *hamt.HashMapRoot, ls ipld.LinkSystem) MultihashIterator
  • func SliceMultihashIterator(mhs []multihash.Multihash) MultihashIterator
• type MultihashLister

Variables

var (
	// ErrNoMultihashLister signals that no provider.MultihashLister is registered for lookup.
	ErrNoMultihashLister = errors.New("no multihash lister is registered")

	// ErrContextIDNotFound signals that no item is associated to the given context ID.
	ErrContextIDNotFound = errors.New("context ID not found")

	// ErrAlreadyAdvertised signals that an advertisement for identical content was already
	// published.
	ErrAlreadyAdvertised = errors.New("advertisement already published")
)

Types

type Interface

type Interface interface {
	// PublishLocal appends adv to the locally stored advertisement chain and
	// returns the corresponding CID to it.  This function does not publish the
	// changes to the advertisement chain onto gossip pubsub channel.  Use
	// Publish instead if indexer nodes must be made aware of the appended
	// advertisement.
	//
	// See: Publish.
	PublishLocal(context.Context, schema.Advertisement) (cid.Cid, error)

	// Publish appends adv to the locally stored advertisement chain, and
	// publishes the new advertisement onto gossip pubsub.  The CID returned
	// represents the ID of the advertisement appended to the chain.
	Publish(context.Context, schema.Advertisement) (cid.Cid, error)

	// RegisterMultihashLister registers the hook that is used by the provider to look up
	// a list of multihashes by context ID. Only a single registration is
	// supported; repeated calls to this function will replace the previous
	// registration.
	RegisterMultihashLister(MultihashLister)

	// NotifyPut signals the provider that the list of multihashes looked up by
	// the given provider and contextIDs is available. The given
	// provider and contextIDs are then used to look up the list of multihashes via MultihashLister.
	// An advertisement is then generated, appended to the chain of advertisements and published onto
	// the gossip pubsub channel. Advertisements for different provider IDs are placed onto the same chain.
	// Use an empty provider string for the default configured provider.
	//
	// A MultihashLister must be registered prior to using this function.
	// ErrNoMultihashLister is returned if no such lister is registered.
	//
	// The metadata is data that provides hints about how to retrieve data and
	// is protocol dependant. The metadata must at least specify a protocol
	// ID, but its data is optional.
	//
	// If provider, contextID and metadata are the same as a previous call to
	// NotifyPut, then ErrAlreadyAdvertised is returned.
	//
	// If provider is nil then the default configured provider will be assumed.
	//
	// This function returns the ID of the advertisement published.
	NotifyPut(ctx context.Context, provider *peer.AddrInfo, contextID []byte, md metadata.Metadata) (cid.Cid, error)

	// NotifyRemove signals to the provider that the multihashes that
	// corresponded to the given provider and contextID are no longer available.  An advertisement
	// is then generated, appended to the chain of advertisements and published
	// onto the gossip pubsub channel.
	// The given provider and contextID tuple must have previously been put via NotifyPut.
	// If not found ErrContextIDNotFound is returned.
	//
	// If providerID is empty then the default configured provider will be assumed.
	//
	// This function returns the ID of the advertisement published.
	NotifyRemove(ctx context.Context, providerID peer.ID, contextID []byte) (cid.Cid, error)

	// GetAdv gets the advertisement that corresponds to the given cid.
	GetAdv(context.Context, cid.Cid) (*schema.Advertisement, error)

	// GetLatestAdv gets the latest advertisement on this provider's
	// advertisement chain and the CID to which it corresponds.
	GetLatestAdv(context.Context) (cid.Cid, *schema.Advertisement, error)

	// Shutdown shuts down this provider, and blocks until all resources
	// occupied by it are discarded.  After calling this function the provider
	// is no longer available for use.
	Shutdown() error
}

Interface represents an index provider that manages the advertisement for multihashes to indexer nodes.

type MultihashIterator

type MultihashIterator interface {
	// Next returns the next multihash in the list of mulitihashes.  The
	// iterator fails fast: errors that occur during iteration are returned
	// immediately.  This function returns a zero multihash and io.EOF when
	// there are no more elements to return.
	Next() (multihash.Multihash, error)
}

MultihashIterator iterates over a list of multihashes.

See: CarMultihashIterator.

func CarMultihashIterator

func CarMultihashIterator(idx carindex.IterableIndex) (MultihashIterator, error)

CarMultihashIterator constructs a new MultihashIterator from a CAR index.

This iterator supplies multihashes in deterministic order of their corresponding CAR offset. The order is maintained consistently regardless of the underlying IterableIndex implementation. Returns error if duplicate offsets detected.

func EntryChunkMultihashIterator

func EntryChunkMultihashIterator(l ipld.Link, ls ipld.LinkSystem) (MultihashIterator, error)

EntryChunkMultihashIterator constructs a MultihashIterator that iterates over the global list of chained multihashes starting from the given link. It dynamically loads the next EntryChunk from the given ipld.LinkSystem as needed.

func HamtMultihashIterator

func HamtMultihashIterator(root *hamt.HashMapRoot, ls ipld.LinkSystem) MultihashIterator

HamtMultihashIterator constructs a MultihashIterator backed by the given root HAMT. The links from root are dynamically loaded as needed using the given link system.

func SliceMultihashIterator

func SliceMultihashIterator(mhs []multihash.Multihash) MultihashIterator

SliceMultihashIterator constructs a new MultihashIterator from a slice of multihashes.

type MultihashLister

type MultihashLister func(ctx context.Context, provider peer.ID, contextID []byte) (MultihashIterator, error)

MultihashLister lists the multihashes that correspond to a given provider and contextID. The lister must be deterministic: it must produce the same list of multihashes in the same order for the same (provider, contextID) tuple.

empty provider means falling back to the default. See: Interface.NotifyPut, Interface.NotifyRemove, MultihashIterator.