README
¶
StoreTheIndex 🗂️
Find a CID stored by storage providers
This repo provides an indexer implementation that can be used to index data stored by a range of participating storage providers.
Design
Current Status
Released for production: The current production release is running at https://cid.contact
This project and is currently under active development 🚧.
Install
This assumes go is already installed.
Install storetheindex:
go install github.com/ipni/storetheindex@latest
Initialize the storetheindex repository and configuration:
storetheindex init
Optionally, edit the configuration
edit ~/.storetheindex/config
Running the Indexer Service
To run storetheindex as a service, run the daemon command. The service watches for providers to index, and exposes a query / content routing client interface.
The daemon is configured by the config file in the storetheindex repository. The config file and repo are created when storetheindex is initialized, using the init command. This repo is located in the local file system. By default, the repo is located at ~/.storetheindex. To change the repo location, set the $STORETHEINDEX_PATH environmental variable.
Provider Removal Policy
After a configured amount of time without any updates from a provider (PollInterval), the indexer will poll the provider at its last know publisher address, for any index updates. If there is no response from the provider after at least one attempt to poll, then the provider is considered inactive and is not returned in any find results. The indexer will continue polling on an interval (PollRetryAfter) until a time limit (PollStopAfter) is reached. If there is still no response to the poll attempts after this time limit is reached, then the provider is removed from the indexer and its records are garbage-collected and will need to be refetched.
The configuration that controls this is documented here. A custom polling configuration may be applied for specific providers using the PollOverrides configuration value to specify per-provider Polling configuration.
Indexer Administration CLI Commands
There are a number of administrative commands supported by storetheindex. These commands allow you to perform operations on a running indexer daemon. For a list of admin commands, see:
./storetheindex admin -help
Help
To see a list of available commands, see storetheindex --help. For help with command usage, see storetheindex <command> --help.
Configuration
The storetheindex config file documentation
Assigner Service
The assigner service is responsible for assigning content advertisement publishers to indexers, and forwarding advertisement announcement messages to indexers. This allows the ingestion load to be divided among a pool of indexers, with a configurable replication factor.
Indexers that are under control of an assigner must be configured to use an assigner. If an assigner-controlled indexer runs out of storage space and enters frozen mode, then indexing is handed off to the next available indexer in the pool.
An assigner may also be used to forward announcement messages to indexers that are not in the assigner's indexer pool or to other assigners. For example, in the case of multiple redundant indexers it is not necessary to manage index publisher assignments, and forwarding announcements is all that is needed. This forwarding can be configured by specifying each recipient's ingest API base URL in the assigner's ForwardAnnounceURLs configuration.
Assigner Install
Install storetheindex, which contains the assigner service, as described above.
Initialize the assigner service:
storetheindex assigner init
Edit the assigner. Usually this includes configuring each Indexer in the assigner's IndexerPool.
edit ~/.assigner/config
To configure an indexer to use the assigner, set UseAssigner to true in the indexer's Discovery configuration
Assigner Configuration
The assigner service config file documentation
License
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
admin
|
|
|
The assigner service is responsible for assigning content advertisement publishers to indexers, and forwarding advertisement announcement messages to indexers.
|
The assigner service is responsible for assigning content advertisement publishers to indexers, and forwarding advertisement announcement messages to indexers. |
|
scripts
command
|
|
|
Package filestore stores files in various types of storage systems.
|
Package filestore stores files in various types of storage systems. |
|
gc
|
|
|
internal
|
|
|
httpserver
Package httpserver provides common HTTP server functionality.
|
Package httpserver provides common HTTP server functionality. |
|
metrics/pprof
Package pprof is separated out from metrics to isolate the 'init' functionality of pprof, so thatit is included when used by binaries, but not if other packages get used or integrated into clients that don't expect the pprof side effect to have taken effect.
|
Package pprof is separated out from metrics to isolate the 'init' functionality of pprof, so thatit is included when used by binaries, but not if other packages get used or integrated into clients that don't expect the pprof side effect to have taken effect. |
|
revision
Package revision provides the rcs revision, embedded by the compiler, as a global variable.
|
Package revision provides the rcs revision, embedded by the compiler, as a global variable. |
|
Package peerutil provides utilities around peer ID values.
|
Package peerutil provides utilities around peer ID values. |
|
scripts
|
|
|
compare_providers
command
|
|
|
cross_announce
command
|
|
|
gen_identity
command
|
|
|
peer_id_from_priv_key
command
|
|
|
server
|
|
|
test
|
|