graphsync

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2021 License: Apache-2.0, MIT Imports: 7 Imported by: 2

README

go-graphsync

Matrix IRC Discord Coverage Status Build Status

An implementation of the graphsync protocol in go!

Table of Contents

Background

GraphSync is a protocol for synchronizing IPLD graphs among peers. It allows a host to make a single request to a remote peer for all of the results of traversing an IPLD selector on the remote peer's local IPLD graph.

go-graphsync provides an implementation of the Graphsync protocol in go.

Go-IPLD-Prime

go-graphsync relies on go-ipld-prime to traverse IPLD Selectors in an IPLD graph. go-ipld-prime implements the IPLD specification in go and is an alternative to older implementations such as go-ipld-format and go-ipld-cbor. In order to use go-graphsync, some understanding and use of go-ipld-prime concepts is necessary.

If your existing library (i.e. go-ipfs or go-filecoin) uses these other older libraries, you can largely use go-graphsync without switching to go-ipld-prime across your codebase, but it will require some translations

Install

go-graphsync requires Go >= 1.11 and can be installed using Go modules

Usage

Initializing a GraphSync Exchange
import (
  graphsync "github.com/ipfs/go-graphsync/impl"
  gsnet "github.com/ipfs/go-graphsync/network"
  ipld "github.com/ipld/go-ipld-prime"
)

var ctx context.Context
var host libp2p.Host
var loader ipld.Loader
var storer ipld.Storer

network := gsnet.NewFromLibp2pHost(host)
exchange := graphsync.New(ctx, network, loader, storer)

Parameter Notes:

  1. context is just the parent context for all of GraphSync
  2. network is a network abstraction provided to Graphsync on top of libp2p. This allows graphsync to be tested without the actual network
  3. loader is used to load blocks from content ids from the local block store. It's used when RESPONDING to requests from other clients. It should conform to the IPLD loader interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go
  4. storer is used to store incoming blocks to the local block store. It's used when REQUESTING a graphsync query, to store blocks locally once they are validated as part of the correct response. It should conform to the IPLD storer interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go
Using GraphSync With An IPFS BlockStore

GraphSync provides two convenience functions in the storeutil package for integrating with BlockStore's from IPFS.

import (
  graphsync "github.com/ipfs/go-graphsync/impl"
  gsnet "github.com/ipfs/go-graphsync/network"
  storeutil "github.com/ipfs/go-graphsync/storeutil"
  ipld "github.com/ipld/go-ipld-prime"
  blockstore "github.com/ipfs/go-ipfs-blockstore"
)

var ctx context.Context
var host libp2p.Host
var bs blockstore.Blockstore

network := gsnet.NewFromLibp2pHost(host)
loader := storeutil.LoaderForBlockstore(bs)
storer := storeutil.StorerForBlockstore(bs)

exchange := graphsync.New(ctx, network, loader, storer)
Write A Loader An IPFS BlockStore

If you are using a traditional go-ipfs-blockstore, your link loading function looks like this:

type BlockStore interface {
	Get(lnk cid.Cid) (blocks.Block, error)
}

or, more generally:

type Cid2BlockFn func (lnk cid.Cid) (blocks.Block, error)

in go-ipld-prime, the signature for a link loader is as follows:

type Loader func(lnk Link, lnkCtx LinkContext) (io.Reader, error)

go-ipld-prime intentionally keeps its interfaces as abstract as possible to limit dependencies on other ipfs/filecoin specific packages. An IPLD Link is an abstraction for a CID, and IPLD expects io.Reader's rather than an actual block. IPLD provides a cidLink package for working with Links that use CIDs as the underlying data, and it's safe to assume that's the type in use if your code deals only with CIDs. A conversion would look something like this:

import (
   ipld "github.com/ipld/go-ipld-prime"
   cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func LoaderFromCid2BlockFn(cid2BlockFn Cid2BlockFn) ipld.Loader {
	return func(lnk ipld.Link, lnkCtx ipld.LinkContext) (io.Reader, error) {
		asCidLink, ok := lnk.(cidlink.Link)
		if !ok {
			return nil, fmt.Errorf("Unsupported Link Type")
		}
		block, err := cid2BlockFn(asCidLink.Cid)
		if err != nil {
			return nil, err
		}
		return bytes.NewReader(block.RawData()), nil
	}
}
Write A Storer From An IPFS BlockStore

If you are using a traditional go-ipfs-blockstore, your storage function looks like this:

type BlockStore interface {
	Put(blocks.Block) error
}

or, more generally:

type BlockStoreFn func (blocks.Block) (error)

in go-ipld-prime, the signature for a link storer is a bit different:

type StoreCommitter func(Link) error
type Storer func(lnkCtx LinkContext) (io.Writer, StoreCommitter, error)

go-ipld-prime stores in two parts to support streaming -- the storer is called and returns an IO.Writer and a function to commit changes when finished. Here's how you can write a storer from a traditional block storing signature.

import (
	blocks "github.com/ipfs/go-block-format"
  ipld "github.com/ipld/go-ipld-prime"
  cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func StorerFromBlockStoreFn(blockStoreFn BlockStoreFn) ipld.Storer {
	return func(lnkCtx ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) {
		var buffer bytes.Buffer
		committer := func(lnk ipld.Link) error {
			asCidLink, ok := lnk.(cidlink.Link)
			if !ok {
				return fmt.Errorf("Unsupported Link Type")
			}
			block := blocks.NewBlockWithCid(buffer.Bytes(), asCidLink.Cid)
			return blockStoreFn(block)
		}
		return &buffer, committer, nil
	}
}
Calling Graphsync
var exchange graphsync.GraphSync
var ctx context.Context
var p peer.ID
var selector ipld.Node
var rootLink ipld.Link

var responseProgress <-chan graphsync.ResponseProgress
var errors <-chan error

responseProgress, errors = exchange.Request(ctx context.Context, p peer.ID, root ipld.Link, selector ipld.Node)

Paramater Notes:

  1. ctx is the context for this request. To cancel an in progress request, cancel the context.
  2. p is the peer you will send this request to
  3. link is an IPLD Link, i.e. a CID (cidLink.Link{Cid})
  4. selector is an IPLD selector node. Recommend using selector builders from go-ipld-prime to construct these
Response Type

type ResponseProgress struct {
  Node      ipld.Node // a node which matched the graphsync query
  Path      ipld.Path // the path of that node relative to the traversal start
	LastBlock struct {  // LastBlock stores the Path and Link of the last block edge we had to load.
		ipld.Path
		ipld.Link
	}
}

The above provides both immediate and relevant metadata for matching nodes in a traversal, and is very similar to the information provided by a local IPLD selector traversal in go-ipld-prime

Contribute

PRs are welcome!

Before doing anything heavy, checkout the Graphsync Architecture

See our Contributing Guidelines for more info.

License

This library is dual-licensed under Apache 2.0 and MIT terms.

Copyright 2019. Protocol Labs, Inc.

Documentation

Index

Constants

View Source 
const (

	// ExtensionMetadata provides response metadata for a Graphsync request and is
	// documented at
	// https://gitlab.dms3.io/ld/specs/blob/master/block-layer/graphsync/known_extensions.md
	ExtensionMetadata = ExtensionName("graphsync/response-metadata")

	// ExtensionDoNotSendCIDs tells the responding peer not to send certain blocks if they
	// are encountered in a traversal and is documented at
	// https://gitlab.dms3.io/ld/specs/blob/master/block-layer/graphsync/known_extensions.md
	ExtensionDoNotSendCIDs = ExtensionName("graphsync/do-not-send-cids")

	// ExtensionDeDupByKey tells the responding peer to only deduplicate block sending
	// for requests that have the same key. The data for the extension is a string key
	ExtensionDeDupByKey = ExtensionName("graphsync/dedup-by-key")
)
View Source 
const (

	// RequestAcknowledged means the request was received and is being worked on.
	RequestAcknowledged = ResponseStatusCode(10)
	// AdditionalPeers means additional peers were found that may be able
	// to satisfy the request and contained in the extra block of the response.
	AdditionalPeers = ResponseStatusCode(11)
	// NotEnoughGas means fulfilling this request requires payment.
	NotEnoughGas = ResponseStatusCode(12)
	// OtherProtocol means a different type of response than GraphSync is
	// contained in extra.
	OtherProtocol = ResponseStatusCode(13)
	// PartialResponse may include blocks and metadata about the in progress response
	// in extra.
	PartialResponse = ResponseStatusCode(14)
	// RequestPaused indicates a request is paused and will not send any more data
	// until unpaused
	RequestPaused = ResponseStatusCode(15)

	// RequestCompletedFull means the entire fulfillment of the GraphSync request
	// was sent back.
	RequestCompletedFull = ResponseStatusCode(20)
	// RequestCompletedPartial means the response is completed, and part of the
	// GraphSync request was sent back, but not the complete request.
	RequestCompletedPartial = ResponseStatusCode(21)

	// RequestRejected means the node did not accept the incoming request.
	RequestRejected = ResponseStatusCode(30)
	// RequestFailedBusy means the node is too busy, try again later. Backoff may
	// be contained in extra.
	RequestFailedBusy = ResponseStatusCode(31)
	// RequestFailedUnknown means the request failed for an unspecified reason. May
	// contain data about why in extra.
	RequestFailedUnknown = ResponseStatusCode(32)
	// RequestFailedLegal means the request failed for legal reasons.
	RequestFailedLegal = ResponseStatusCode(33)
	// RequestFailedContentNotFound means the respondent does not have the content.
	RequestFailedContentNotFound = ResponseStatusCode(34)
	// RequestCancelled means the responder was processing the request but decided to top, for whatever reason
	RequestCancelled = ResponseStatusCode(35)
)

GraphSync Response Status Codes

Variables

View Source 
var (
	// ErrExtensionAlreadyRegistered means a user extension can be registered only once
	ErrExtensionAlreadyRegistered = errors.New("extension already registered")
)
View Source 
var ResponseCodeToName = map[ResponseStatusCode]string{
	RequestAcknowledged:          "RequestAcknowledged",
	AdditionalPeers:              "AdditionalPeers",
	NotEnoughGas:                 "NotEnoughGas",
	OtherProtocol:                "OtherProtocol",
	PartialResponse:              "PartialResponse",
	RequestPaused:                "RequestPaused",
	RequestCompletedFull:         "RequestCompletedFull",
	RequestCompletedPartial:      "RequestCompletedPartial",
	RequestRejected:              "RequestRejected",
	RequestFailedBusy:            "RequestFailedBusy",
	RequestFailedUnknown:         "RequestFailedUnknown",
	RequestFailedLegal:           "RequestFailedLegal",
	RequestFailedContentNotFound: "RequestFailedContentNotFound",
	RequestCancelled:             "RequestCancelled",
}

Functions

This section is empty.

Types

type BlockData 

type BlockData interface {
	// Link is the link/cid for the block
	Link() ld.Link

	// BlockSize specifies the size of the block
	BlockSize() uint64

	// BlockSize specifies the amount of data actually transmitted over the network
	BlockSizeOnWire() uint64
}

BlockData gives information about a block included in a graphsync response

type ExtensionData 

type ExtensionData struct {
	Name ExtensionName
	Data []byte
}

ExtensionData is a name/data pair for a graphsync extension

type ExtensionName 

type ExtensionName string

ExtensionName is a name for a GraphSync extension

type GraphExchange 

type GraphExchange interface {
	// Request initiates a new GraphSync request to the given peer using the given selector spec.
	Request(ctx context.Context, p peer.ID, root ld.Link, selector ld.Node, extensions ...ExtensionData) (<-chan ResponseProgress, <-chan error)

	// RegisterPersistenceOption registers an alternate loader/storer combo that can be substituted for the default
	RegisterPersistenceOption(name string, lsys ld.LinkSystem) error

	// UnregisterPersistenceOption unregisters an alternate loader/storer combo
	UnregisterPersistenceOption(name string) error

	// RegisterIncomingRequestHook adds a hook that runs when a request is received
	RegisterIncomingRequestHook(hook OnIncomingRequestHook) UnregisterHookFunc

	// RegisterIncomingResponseHook adds a hook that runs when a response is received
	RegisterIncomingResponseHook(OnIncomingResponseHook) UnregisterHookFunc

	// RegisterIncomingBlockHook adds a hook that runs when a block is received and validated (put in block store)
	RegisterIncomingBlockHook(OnIncomingBlockHook) UnregisterHookFunc

	// RegisterOutgoingRequestHook adds a hook that runs immediately prior to sending a new request
	RegisterOutgoingRequestHook(hook OnOutgoingRequestHook) UnregisterHookFunc

	// RegisterOutgoingBlockHook adds a hook that runs every time a block is sent from a responder
	RegisterOutgoingBlockHook(hook OnOutgoingBlockHook) UnregisterHookFunc

	// RegisterRequestUpdatedHook adds a hook that runs every time an update to a request is received
	RegisterRequestUpdatedHook(hook OnRequestUpdatedHook) UnregisterHookFunc

	// RegisterCompletedResponseListener adds a listener on the responder for completed responses
	RegisterCompletedResponseListener(listener OnResponseCompletedListener) UnregisterHookFunc

	// RegisterRequestorCancelledListener adds a listener on the responder for
	// responses cancelled by the requestor
	RegisterRequestorCancelledListener(listener OnRequestorCancelledListener) UnregisterHookFunc

	// RegisterBlockSentListener adds a listener for when blocks are actually sent over the wire
	RegisterBlockSentListener(listener OnBlockSentListener) UnregisterHookFunc

	// RegisterNetworkErrorListener adds a listener for when errors occur sending data over the wire
	RegisterNetworkErrorListener(listener OnNetworkErrorListener) UnregisterHookFunc

	// RegisterReceiverNetworkErrorListener adds a listener for when errors occur receiving data over the wire
	RegisterReceiverNetworkErrorListener(listener OnReceiverNetworkErrorListener) UnregisterHookFunc

	// UnpauseRequest unpauses a request that was paused in a block hook based request ID
	// Can also send extensions with unpause
	UnpauseRequest(RequestID, ...ExtensionData) error

	// PauseRequest pauses an in progress request (may take 1 or more blocks to process)
	PauseRequest(RequestID) error

	// UnpauseResponse unpauses a response that was paused in a block hook based on peer ID and request ID
	// Can also send extensions with unpause
	UnpauseResponse(peer.ID, RequestID, ...ExtensionData) error

	// PauseResponse pauses an in progress response (may take 1 or more blocks to process)
	PauseResponse(peer.ID, RequestID) error

	// CancelResponse cancels an in progress response
	CancelResponse(peer.ID, RequestID) error
}

GraphExchange is a protocol that can exchange LD graphs based on a selector

type IncomingBlockHookActions 

type IncomingBlockHookActions interface {
	TerminateWithError(error)
	UpdateRequestWithExtensions(...ExtensionData)
	PauseRequest()
}

IncomingBlockHookActions are actions that incoming block hook can take to change the execution of a request

type IncomingRequestHookActions 

type IncomingRequestHookActions interface {
	SendExtensionData(ExtensionData)
	UsePersistenceOption(name string)
	UseLinkTargetNodePrototypeChooser(traversal.LinkTargetNodePrototypeChooser)
	TerminateWithError(error)
	ValidateRequest()
	PauseResponse()
}

IncomingRequestHookActions are actions that a request hook can take to change behavior for the response

type IncomingResponseHookActions 

type IncomingResponseHookActions interface {
	TerminateWithError(error)
	UpdateRequestWithExtensions(...ExtensionData)
}

IncomingResponseHookActions are actions that incoming response hook can take to change the execution of a request

type OnBlockSentListener 

type OnBlockSentListener func(p peer.ID, request RequestData, block BlockData)

OnBlockSentListener runs when a block is sent over the wire

type OnIncomingBlockHook 

type OnIncomingBlockHook func(p peer.ID, responseData ResponseData, blockData BlockData, hookActions IncomingBlockHookActions)

OnIncomingBlockHook is a hook that runs each time a new block is validated as part of the response, regardless of whether it came locally or over the network It receives that sent the response, the most recent response, a link for the block received, and the size of the block received The difference between BlockSize & BlockSizeOnWire can be used to determine where the block came from (Local vs remote) It receives an interface for customizing how we handle the ongoing execution of the request

type OnIncomingRequestHook 

type OnIncomingRequestHook func(p peer.ID, request RequestData, hookActions IncomingRequestHookActions)

OnIncomingRequestHook is a hook that runs each time a new request is received. It receives the peer that sent the request and all data about the request. It receives an interface for customizing the response to this request

type OnIncomingResponseHook 

type OnIncomingResponseHook func(p peer.ID, responseData ResponseData, hookActions IncomingResponseHookActions)

OnIncomingResponseHook is a hook that runs each time a new response is received. It receives the peer that sent the response and all data about the response. It receives an interface for customizing how we handle the ongoing execution of the request

type OnNetworkErrorListener 

type OnNetworkErrorListener func(p peer.ID, request RequestData, err error)

OnNetworkErrorListener runs when queued data is not able to be sent

type OnOutgoingBlockHook 

type OnOutgoingBlockHook func(p peer.ID, request RequestData, block BlockData, hookActions OutgoingBlockHookActions)

OnOutgoingBlockHook is a hook that runs immediately after a requestor sends a new block on a response It receives the peer we're sending a request to, all the data aobut the request, a link for the block sent, and the size of the block sent It receives an interface for taking further action on the response

type OnOutgoingRequestHook 

type OnOutgoingRequestHook func(p peer.ID, request RequestData, hookActions OutgoingRequestHookActions)

OnOutgoingRequestHook is a hook that runs immediately prior to sending a request It receives the peer we're sending a request to and all the data aobut the request It receives an interface for customizing how we handle executing this request

type OnReceiverNetworkErrorListener 

type OnReceiverNetworkErrorListener func(p peer.ID, err error)

OnReceiverNetworkErrorListener runs when errors occur receiving data over the wire

type OnRequestUpdatedHook 

type OnRequestUpdatedHook func(p peer.ID, request RequestData, updateRequest RequestData, hookActions RequestUpdatedHookActions)

OnRequestUpdatedHook is a hook that runs when an update to a request is received It receives the peer we're sending to, the original request, the request update It receives an interface to taking further action on the response

type OnRequestorCancelledListener 

type OnRequestorCancelledListener func(p peer.ID, request RequestData)

OnRequestorCancelledListener provides a way to listen for responses the requestor canncels

type OnResponseCompletedListener 

type OnResponseCompletedListener func(p peer.ID, request RequestData, status ResponseStatusCode)

OnResponseCompletedListener provides a way to listen for when responder has finished serving a response

type OutgoingBlockHookActions 

type OutgoingBlockHookActions interface {
	SendExtensionData(ExtensionData)
	TerminateWithError(error)
	PauseResponse()
}

OutgoingBlockHookActions are actions that an outgoing block hook can take to change the execution of a request

type OutgoingRequestHookActions 

type OutgoingRequestHookActions interface {
	UsePersistenceOption(name string)
	UseLinkTargetNodePrototypeChooser(traversal.LinkTargetNodePrototypeChooser)
}

OutgoingRequestHookActions are actions that an outgoing request hook can take to change the execution of a request

type Priority 

type Priority int32

Priority a priority for a GraphSync request.

type RequestCancelledErr 

type RequestCancelledErr struct{}

RequestCancelledErr is an error message received on the error channel that indicates the responder cancelled a request

func (RequestCancelledErr) Error 

func (e RequestCancelledErr) Error() string

type RequestContextCancelledErr 

type RequestContextCancelledErr struct{}

RequestContextCancelledErr is an error message received on the error channel when the request context given by the user is cancelled/times out

func (RequestContextCancelledErr) Error 

func (e RequestContextCancelledErr) Error() string

type RequestData 

type RequestData interface {
	// ID Returns the request ID for this Request
	ID() RequestID

	// Root returns the CID to the root block of this request
	Root() cid.Cid

	// Selector returns the byte representation of the selector for this request
	Selector() ld.Node

	// Priority returns the priority of this request
	Priority() Priority

	// Extension returns the content for an extension on a response, or errors
	// if extension is not present
	Extension(name ExtensionName) ([]byte, bool)

	// IsCancel returns true if this particular request is being cancelled
	IsCancel() bool
}

RequestData describes a received graphsync request.

type RequestFailedBusyErr 

type RequestFailedBusyErr struct{}

RequestFailedBusyErr is an error message received on the error channel when the peer is busy

func (RequestFailedBusyErr) Error 

func (e RequestFailedBusyErr) Error() string

type RequestFailedContentNotFoundErr 

type RequestFailedContentNotFoundErr struct{}

RequestFailedContentNotFoundErr is an error message received on the error channel when the content is not found

func (RequestFailedContentNotFoundErr) Error 

func (e RequestFailedContentNotFoundErr) Error() string

type RequestFailedLegalErr 

type RequestFailedLegalErr struct{}

RequestFailedLegalErr is an error message received on the error channel when the request fails for legal reasons

func (RequestFailedLegalErr) Error 

func (e RequestFailedLegalErr) Error() string

type RequestFailedUnknownErr 

type RequestFailedUnknownErr struct{}

RequestFailedUnknownErr is an error message received on the error channel when the request fails for unknown reasons

func (RequestFailedUnknownErr) Error 

func (e RequestFailedUnknownErr) Error() string

type RequestID 

type RequestID int32

RequestID is a unique identifier for a GraphSync request.

type RequestUpdatedHookActions 

type RequestUpdatedHookActions interface {
	TerminateWithError(error)
	SendExtensionData(ExtensionData)
	UnpauseResponse()
}

RequestUpdatedHookActions are actions that can be taken in a request updated hook to change execution of the response

type ResponseData 

type ResponseData interface {
	// RequestID returns the request ID for this response
	RequestID() RequestID

	// Status returns the status for a response
	Status() ResponseStatusCode

	// Extension returns the content for an extension on a response, or errors
	// if extension is not present
	Extension(name ExtensionName) ([]byte, bool)
}

ResponseData describes a received Graphsync response

type ResponseProgress 

type ResponseProgress struct {
	Node      ld.Node // a node which matched the graphsync query
	Path      ld.Path // the path of that node relative to the traversal start
	LastBlock struct {
		Path ld.Path
		Link ld.Link
	}
}

ResponseProgress is the fundamental unit of responses making progress in Graphsync.

type ResponseStatusCode 

type ResponseStatusCode int32

ResponseStatusCode is a status returned for a GraphSync Request.

func (ResponseStatusCode) String 

func (c ResponseStatusCode) String() string

type UnregisterHookFunc 

type UnregisterHookFunc func()

UnregisterHookFunc is a function call to unregister a hook that was previously registered

Source Files

View all Source files

Directories

Path Synopsis
benchmarks
testinstance
testnet
pb
asyncloader
asyncloader/loadattemptqueue
asyncloader/responsecache
asyncloader/unverifiedblockstore
executor
hooks
testloader
types
hooks
persistenceoptions
responseassembler
Package responseassembler assembles responses that are queued for sending in outgoing messages The response assembler's Transaction method allows a caller to specify response actions that will go into a single p2p message.
 Package responseassembler assembles responses that are queued for sending in outgoing messages The response assembler's Transaction method allows a caller to specify response actions that will go into a single p2p message.
runtraversal
chaintypes
chaintypes/gen

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.