ipfs-search

command module
Version: v0.0.0-...-90d34b5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 26, 2022 License: AGPL-3.0 Imports: 9 Imported by: 0

README

ipfs-search.com

pipeline status Maintainability Test Coverage Documentation Status Go Reference Backers on Open Collective Sponsors on Open Collective

Search engine for the Interplanetary Filesystem. Sniffs the DHT gossip and indexes file and directory hashes.

Metadata and contents are extracted using ipfs-tika, searching is done using OpenSearch 7, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.

The ipfs-search command consists of two components: the crawler and the sniffer. The sniffer extracts hashes from the gossip between nodes. The crawler extracts data from the hashes and indexes them.

Docs

Documentation is hosted on Read the Docs, based on files contained in the docs folder. In addition, there's extensive Go docs for the internal API as well as SwaggerHub OpenAPI documentation for the REST API.

Contact

Please find us on our Freenode/Riot/Matrix channel #ipfs-search:matrix.org.

Snapshots

ipfs-search provides the daily snapshot for all of the indexed data using snapshots. To learn more about downloading and restoring snapshots please refer to the relevant section in our documentation.

Contributors wanted

Building a search engine like this takes a considerable amount of resources (money and TLC). If you are able to help out with either of them, do reach out (see the contact section in this file).

Please read the Contributing.md file before contributing.

Roadmap

For discussing and suggesting features, look at the issues.

External dependencies

  • Go 1.19
  • OpenSearch 2.3.x
  • RabbitMQ / AMQP server
  • NodeJS 9.x
  • IPFS 0.7
  • Redis

Internal dependencies

Building

$ go get ./...
$ make

Running

Docker

The most convenient way to run the crawler is through Docker. Simply run:

docker-compose up

This will start the crawler, the sniffer and all its dependencies. Hashes can also be queued for crawling manually by running ipfs-search a <hash> from within the running container. For example:

docker-compose exec ipfs-crawler ipfs-search add QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
Ansible deployment

Automated deployment can be done on any (virtual) Ubuntu 16.04 machine. The full production stack is automated and can be found in it's own repository.

Contributors

This project exists thanks to all the people who contribute.

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors


ipfs-search is supported by NLNet through the EU's Next Generation Internet (NGI0) programme.


RedPencil is supporting the hosting of ipfs-search.com.

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

Documentation

Overview

Search engine for IPFS using OpenSearch, RabbitMQ and Tika.

Source Files

Directories

Path Synopsis
Package commands contains commands as they would be called by the end user.
Package commands contains commands as they would be called by the end user.
Package components groups together the main operational components of the crawler.
Package components groups together the main operational components of the crawler.
crawler
Package crawler is grouped around the Crawler component, crawling and indexing content from an AnnotatedResource.
Package crawler is grouped around the Crawler component, crawling and indexing content from an AnnotatedResource.
extractor
Package extractor is grouped around the Extractor component, extracting metadata from an AnnotatedResource.
Package extractor is grouped around the Extractor component, extracting metadata from an AnnotatedResource.
index
Package index is grouped around the Index component, representing an index which stores and retrieves document properties.
Package index is grouped around the Index component, representing an index which stores and retrieves document properties.
protocol
Package protocol is grouped around the Protocol component, the interface with one or multiple protocols.
Package protocol is grouped around the Protocol component, the interface with one or multiple protocols.
queue
Package queue provides queueing semantics.
Package queue provides queueing semantics.
sniffer
Package sniffer contains sniffer components which can be wired into a libp2p dht node by proxying the datastore.
Package sniffer contains sniffer components which can be wired into a libp2p dht node by proxying the datastore.
Package config provides central and canonical representation, reading, parsing and validation of configuration for components.
Package config provides central and canonical representation, reading, parsing and validation of configuration for components.
Package instr contains common instrumentation tooling.
Package instr contains common instrumentation tooling.
Package types consists of common datatypes used by other components for internal representation, not for exporting or indexing.
Package types consists of common datatypes used by other components for internal representation, not for exporting or indexing.
Package utils contains common utility functions used in other components.
Package utils contains common utility functions used in other components.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL