gateway

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 9, 2024 License: Apache-2.0, MIT Imports: 72 Imported by: 16

README

IPFS Gateway

A reference implementation of HTTP Gateway Specifications.

Documentation

Example

This example shows how you can start your own gateway, assuming you have an IPFSBackend implementation.

conf := gateway.Config{}

// Initialize an IPFSBackend interface for both an online and offline versions.
// The offline version should not make any network request for missing content.
ipfsBackend := ...

// Create http mux and setup path gateway handler.
mux := http.NewServeMux()
handler := gateway.NewHandler(conf, ipfsBackend)
handler = gateway.NewHeaders(nil).ApplyCors().Wrap(handler)
mux.Handle("/ipfs/", handler)
mux.Handle("/ipns/", handler)


// Start the server on :8080 and voilá! You have a basic IPFS gateway running
// in http://localhost:8080.
_ = http.ListenAndServe(":8080", mux)

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInternalServerError = NewErrorStatusCodeFromStatus(http.StatusInternalServerError)
	ErrGatewayTimeout      = NewErrorStatusCodeFromStatus(http.StatusGatewayTimeout)
	ErrBadGateway          = NewErrorStatusCodeFromStatus(http.StatusBadGateway)
	ErrServiceUnavailable  = NewErrorStatusCodeFromStatus(http.StatusServiceUnavailable)
	ErrTooManyRequests     = NewErrorStatusCodeFromStatus(http.StatusTooManyRequests)
)

Functions 

func InlineDNSLink(fqdn string) (dnsLabel string, err error)

Converts a FQDN to DNS-safe representation that fits in 63 characters: my.v-long.example.com → my-v--long-example-com InlineDNSLink implements specification from https://specs.ipfs.tech/http-gateways/subdomain-gateway/#host-request-header

func NewDNSResolver 

func NewDNSResolver(resolvers map[string]string, dohOpts ...doh.Option) (*madns.Resolver, error)

NewDNSResolver creates a new DNS resolver based on the default resolvers and the provided resolvers.

The argument 'resolvers' is a map of FQDNs to URLs for custom DNS resolution. URLs starting with "https://" indicate DoH endpoints. Support for other resolver types may be added in the future.

Example:

func NewHandler 

func NewHandler(c Config, backend IPFSBackend) http.Handler

NewHandler returns an http.Handler that provides the functionality of an IPFS HTTP Gateway based on a Config and IPFSBackend.

func NewHostnameHandler added in v0.10.0 

func NewHostnameHandler(c Config, backend IPFSBackend, next http.Handler) http.HandlerFunc

NewHostnameHandler is a middleware that wraps an http.Handler in order to parse the Host header and translate it into the content path. This is useful for creating Subdomain Gateways or DNSLink Gateways. 

func UninlineDNSLink(dnsLabel string) (fqdn string)

Converts a DNS-safe representation of DNSLink FQDN to real FQDN: my-v--long-example-com → my.v-long.example.com UninlineDNSLink implements specification from https://specs.ipfs.tech/http-gateways/subdomain-gateway/#host-request-header

Types

type BlocksBackend added in v0.10.0 

type BlocksBackend struct {
	// contains filtered or unexported fields
}

BlocksBackend is an IPFSBackend implementation based on a blockservice.BlockService.

func NewBlocksBackend added in v0.10.0 

func NewBlocksBackend(blockService blockservice.BlockService, opts ...BlocksBackendOption) (*BlocksBackend, error)

func (*BlocksBackend) Get added in v0.10.0 

func (bb *BlocksBackend) Get(ctx context.Context, path path.ImmutablePath, ranges ...ByteRange) (ContentPathMetadata, *GetResponse, error)

func (*BlocksBackend) GetAll added in v0.10.0 

func (bb *BlocksBackend) GetAll(ctx context.Context, path path.ImmutablePath) (ContentPathMetadata, files.Node, error)

func (*BlocksBackend) GetBlock added in v0.10.0 

func (bb *BlocksBackend) GetBlock(ctx context.Context, path path.ImmutablePath) (ContentPathMetadata, files.File, error)

func (*BlocksBackend) GetCAR added in v0.10.0 

func (bb *BlocksBackend) GetCAR(ctx context.Context, p path.ImmutablePath, params CarParams) (ContentPathMetadata, io.ReadCloser, error)

func (*BlocksBackend) GetDNSLinkRecord added in v0.10.0 

func (bb *BlocksBackend) GetDNSLinkRecord(ctx context.Context, hostname string) (path.Path, error)

func (*BlocksBackend) GetIPNSRecord added in v0.10.0 

func (bb *BlocksBackend) GetIPNSRecord(ctx context.Context, c cid.Cid) ([]byte, error)

func (*BlocksBackend) Head added in v0.10.0 

func (bb *BlocksBackend) Head(ctx context.Context, path path.ImmutablePath) (ContentPathMetadata, *HeadResponse, error)

func (*BlocksBackend) IsCached added in v0.10.0 

func (bb *BlocksBackend) IsCached(ctx context.Context, p path.Path) bool

func (*BlocksBackend) ResolveMutable added in v0.10.0 

func (bb *BlocksBackend) ResolveMutable(ctx context.Context, p path.Path) (path.ImmutablePath, time.Duration, time.Time, error)

func (*BlocksBackend) ResolvePath added in v0.10.0 

func (bb *BlocksBackend) ResolvePath(ctx context.Context, path path.ImmutablePath) (ContentPathMetadata, error)

func (*BlocksBackend) WrapContextForRequest added in v0.18.0 

func (bb *BlocksBackend) WrapContextForRequest(ctx context.Context) context.Context

type BlocksBackendOption added in v0.10.0 

type BlocksBackendOption func(options *blocksBackendOptions) error

func WithNameSystem 

func WithNameSystem(ns namesys.NameSystem) BlocksBackendOption

WithNameSystem sets the name system to use with the BlocksBackend. If not set it will use the default DNSLink resolver generated by NewDNSResolver along with any configured routing.ValueStore.

func WithResolver added in v0.14.0 

func WithResolver(r resolver.Resolver) BlocksBackendOption

WithResolver sets the resolver.Resolver to use with the BlocksBackend.

func WithValueStore 

func WithValueStore(vs routing.ValueStore) BlocksBackendOption

WithValueStore sets the routing.ValueStore to use with the BlocksBackend.

type ByteRange 

type ByteRange struct {
	From uint64
	To   *int64
}

ByteRange describes a range request within a UnixFS file. "From" and "To" mostly follow HTTP Byte Range Request semantics:

  • From >= 0 and To = nil: Get the file (From, Length)
  • From >= 0 and To >= 0: Get the range (From, To)
  • From >= 0 and To <0: Get the range (From, Length - To)

type CarParams added in v0.10.0 

type CarParams struct {
	Range      *DagByteRange
	Scope      DagScope
	Order      DagOrder
	Duplicates DuplicateBlocksPolicy
}

type Config 

type Config struct {
	// DeserializedResponses configures this gateway to support returning data
	// in deserialized format. By default, the gateway will only support
	// trustless, verifiable [application/vnd.ipld.raw] and
	// [application/vnd.ipld.car] responses, operating as a [Trustless Gateway].
	//
	// This global flag can be overridden per FQDN in PublicGateways map.
	//
	// [application/vnd.ipld.raw]: https://www.iana.org/assignments/media-types/application/vnd.ipld.raw
	// [application/vnd.ipld.car]: https://www.iana.org/assignments/media-types/application/vnd.ipld.car
	// [Trustless Gateway]: https://specs.ipfs.tech/http-gateways/trustless-gateway/
	DeserializedResponses bool

	// NoDNSLink configures the gateway to _not_ perform DNS TXT record lookups in
	// response to requests with values in `Host` HTTP header. This flag can be
	// overridden per FQDN in PublicGateways. To be used with WithHostname.
	NoDNSLink bool

	// DisableHTMLErrors disables pretty HTML pages when an error occurs. Instead, a `text/plain`
	// page will be sent with the raw error message. This can be useful if this gateway
	// is being proxied by other service, which wants to use the error message.
	DisableHTMLErrors bool

	// PublicGateways configures the behavior of known public gateways. Each key is
	// a fully qualified domain name (FQDN). To be used with WithHostname.
	PublicGateways map[string]*PublicGateway

	// Menu adds items to the gateway menu that are shown in pages, such as
	// directory listings, DAG previews and errors. These will be displayed to the
	// right of "About IPFS" and "Install IPFS".
	Menu []assets.MenuItem
}

Config is the configuration used when creating a new gateway handler.

type ContentPathMetadata 

type ContentPathMetadata struct {
	PathSegmentRoots     []cid.Cid
	LastSegment          path.ImmutablePath
	LastSegmentRemainder []string
	ContentType          string // Only used for UnixFS requests
}

type DagByteRange added in v0.10.0 

type DagByteRange struct {
	From int64
	To   *int64
}

DagByteRange describes a range request within a UnixFS file. "From" and "To" mostly follow the HTTP Byte Range Request semantics:

  • From >= 0 and To = nil: Get the file (From, Length)
  • From >= 0 and To >= 0: Get the range (From, To)
  • From >= 0 and To <0: Get the range (From, Length - To)
  • From < 0 and To = nil: Get the file (Length - From, Length)
  • From < 0 and To >= 0: Get the range (Length - From, To)
  • From < 0 and To <0: Get the range (Length - From, Length - To)

func NewDagByteRange added in v0.10.0 

func NewDagByteRange(rangeStr string) (DagByteRange, error)

type DagOrder added in v0.11.0 

type DagOrder string
const (
	DagOrderUnspecified DagOrder = ""
	DagOrderUnknown     DagOrder = "unk"
	DagOrderDFS         DagOrder = "dfs"
)

type DagScope added in v0.10.0 

type DagScope string

DagScope describes the scope of the requested DAG, as per the Trustless Gateway specification. 

const (
	DagScopeAll    DagScope = "all"
	DagScopeEntity DagScope = "entity"
	DagScopeBlock  DagScope = "block"
)

type DuplicateBlocksPolicy added in v0.11.0 

type DuplicateBlocksPolicy uint8

DuplicateBlocksPolicy represents the content type parameter 'dups' (IPIP-412) 

const (
	DuplicateBlocksUnspecified DuplicateBlocksPolicy = iota // 0 - implicit default
	DuplicateBlocksIncluded                                 // 1 - explicitly include duplicates
	DuplicateBlocksExcluded                                 // 2 - explicitly NOT include duplicates
)

func NewDuplicateBlocksPolicy added in v0.11.0 

func NewDuplicateBlocksPolicy(dupsValue string) (DuplicateBlocksPolicy, error)

NewDuplicateBlocksPolicy returns DuplicateBlocksPolicy based on the content type parameter 'dups' (IPIP-412)

func (DuplicateBlocksPolicy) Bool added in v0.11.0 

func (d DuplicateBlocksPolicy) Bool() bool

func (DuplicateBlocksPolicy) String added in v0.11.0 

func (d DuplicateBlocksPolicy) String() string

type ErrorRetryAfter 

type ErrorRetryAfter struct {
	Err        error
	RetryAfter time.Duration
}

ErrorRetryAfter wraps any error with "retry after" hint. When an error of this type returned to the gateway handler by an IPFSBackend, the retry after value will be passed to the HTTP client in a Retry-After HTTP header.

func NewErrorRetryAfter 

func NewErrorRetryAfter(err error, retryAfter time.Duration) *ErrorRetryAfter

func (*ErrorRetryAfter) Error 

func (e *ErrorRetryAfter) Error() string

func (*ErrorRetryAfter) Is 

func (e *ErrorRetryAfter) Is(err error) bool

func (*ErrorRetryAfter) RetryAfterHeader added in v0.10.0 

func (e *ErrorRetryAfter) RetryAfterHeader() string

RetryAfterHeader returns the Retry-After header value as a string, representing the number of seconds to wait before making a new request, rounded to the nearest second. This function follows the Retry-After header definition as specified in RFC 9110.

func (*ErrorRetryAfter) Unwrap 

func (e *ErrorRetryAfter) Unwrap() error

type ErrorStatusCode added in v0.10.0 

type ErrorStatusCode struct {
	StatusCode int
	Err        error
}

ErrorStatusCode wraps any error with a specific HTTP status code. When an error of this type is returned to the gateway handler by an IPFSBackend, the status code will be used for the response status.

func NewErrorStatusCode added in v0.10.0 

func NewErrorStatusCode(err error, statusCode int) *ErrorStatusCode

func NewErrorStatusCodeFromStatus added in v0.10.0 

func NewErrorStatusCodeFromStatus(statusCode int) *ErrorStatusCode

func (*ErrorStatusCode) Error added in v0.10.0 

func (e *ErrorStatusCode) Error() string

func (*ErrorStatusCode) Is added in v0.10.0 

func (e *ErrorStatusCode) Is(err error) bool

func (*ErrorStatusCode) Unwrap added in v0.10.0 

func (e *ErrorStatusCode) Unwrap() error

type GetResponse 

type GetResponse struct {
	// contains filtered or unexported fields
}

func NewGetResponseFromDirectoryListing 

func NewGetResponseFromDirectoryListing(dagSize uint64, entries <-chan unixfs.LinkResult, closeFn func() error) *GetResponse

func NewGetResponseFromReader added in v0.14.0 

func NewGetResponseFromReader(file io.ReadCloser, fullFileSize int64) *GetResponse
func NewGetResponseFromSymlink(symlink *files.Symlink, size int64) *GetResponse

func (*GetResponse) Close added in v0.14.0 

func (r *GetResponse) Close() error

type HeadResponse added in v0.14.0 

type HeadResponse struct {
	// contains filtered or unexported fields
}

func NewHeadResponseForDirectory added in v0.14.0 

func NewHeadResponseForDirectory(dagSize int64) *HeadResponse

func NewHeadResponseForFile added in v0.14.0 

func NewHeadResponseForFile(startingBytes io.ReadCloser, size int64) *HeadResponse
func NewHeadResponseForSymlink(symlinkSize int64) *HeadResponse

func (*HeadResponse) Close added in v0.14.0 

func (r *HeadResponse) Close() error

type Headers added in v0.18.0 

type Headers struct {
	// contains filtered or unexported fields
}

Headers is an HTTP middleware that sets the configured headers in all requests.

func NewHeaders added in v0.18.0 

func NewHeaders(headers map[string][]string) *Headers

NewHeaders creates a new Headers middleware that applies the given headers to all requests. If you call Headers.ApplyCors, the default CORS configuration will also be applied, if any of the CORS headers is missing.

func (*Headers) ApplyCors added in v0.18.0 

func (h *Headers) ApplyCors() *Headers

ApplyCors applies safe default HTTP headers for controlling cross-origin requests. This function adds several values to the Access-Control-Allow-Headers and Access-Control-Expose-Headers entries to be exposed on GET and OPTIONS responses, including CORS Preflight.

If the Access-Control-Allow-Origin entry is missing, a default value of '*' is added, indicating that browsers should allow requesting code from any origin to access the resource.

If the Access-Control-Allow-Methods entry is missing a value, 'GET, HEAD, OPTIONS' is added, indicating that browsers may use them when issuing cross origin requests.

func (*Headers) Wrap added in v0.18.0 

func (h *Headers) Wrap(next http.Handler) http.Handler

Wrap wraps the given http.Handler with the headers middleware.

type IPFSBackend 

type IPFSBackend interface {
	// Get returns a [GetResponse] with UnixFS file, directory or a block in IPLD
	// format, e.g. (DAG-)CBOR/JSON.
	//
	// Returned Directories are preferably a minimum info required for enumeration: Name, Size, and Cid.
	//
	// Optional ranges follow [HTTP Byte Ranges] notation and can be used for
	// pre-fetching specific sections of a file or a block.
	//
	// Range notes:
	//   - Generating response to a range request may require additional data
	//     beyond the passed ranges (e.g. a single byte range from the middle of a
	//     file will still need magic bytes from the very beginning for content
	//     type sniffing).
	//   - A range request for a directory currently holds no semantic meaning.
	//   - For non-UnixFS (and non-raw data) such as terminal IPLD dag-cbor/json, etc. blocks the returned response
	//     bytes should be the complete block and returned as an [io.ReadSeekCloser] starting at the beginning of the
	//     block rather than as an [io.ReadCloser] that starts at the beginning of the range request.
	//
	// [HTTP Byte Ranges]: https://httpwg.org/specs/rfc9110.html#rfc.section.14.1.2
	Get(context.Context, path.ImmutablePath, ...ByteRange) (ContentPathMetadata, *GetResponse, error)

	// GetAll returns a UnixFS file or directory depending on what the path is that has been requested. Directories should
	// include all content recursively.
	GetAll(context.Context, path.ImmutablePath) (ContentPathMetadata, files.Node, error)

	// GetBlock returns a single block of data
	GetBlock(context.Context, path.ImmutablePath) (ContentPathMetadata, files.File, error)

	// Head returns a [HeadResponse] depending on what the path is that has been requested.
	// For UnixFS files (and raw blocks) should return the size of the file and either set the ContentType in
	// ContentPathMetadata or send back a reader from the beginning of the file with enough data (e.g. 3kiB) such that
	// the content type can be determined by sniffing.
	//
	// For UnixFS directories and symlinks only setting the size and type are necessary.
	//
	// For all other data types (e.g. (DAG-)CBOR/JSON blocks) returning the size information as a file while setting
	// the content-type is sufficient.
	Head(context.Context, path.ImmutablePath) (ContentPathMetadata, *HeadResponse, error)

	// ResolvePath resolves the path using UnixFS resolver. If the path does not
	// exist due to a missing link, it should return an error of type:
	// NewErrorResponse(fmt.Errorf("no link named %q under %s", name, cid), http.StatusNotFound)
	ResolvePath(context.Context, path.ImmutablePath) (ContentPathMetadata, error)

	// GetCAR returns a CAR file for the given immutable path. It returns an error
	// if there was an issue before the CAR streaming begins.
	GetCAR(context.Context, path.ImmutablePath, CarParams) (ContentPathMetadata, io.ReadCloser, error)

	// IsCached returns whether or not the path exists locally.
	IsCached(context.Context, path.Path) bool

	// GetIPNSRecord retrieves the best IPNS record for a given CID (libp2p-key)
	// from the routing system.
	GetIPNSRecord(context.Context, cid.Cid) ([]byte, error)

	// ResolveMutable takes a mutable path and resolves it into an immutable one. This means recursively resolving any
	// DNSLink or IPNS records. It should also return a TTL. If the TTL is unknown, 0 should be returned.
	//
	// For example, given a mapping from `/ipns/dnslink.tld -> /ipns/ipns-id/mydirectory` and `/ipns/ipns-id` to
	// `/ipfs/some-cid`, the result of passing `/ipns/dnslink.tld/myfile` would be `/ipfs/some-cid/mydirectory/myfile`.
	ResolveMutable(context.Context, path.Path) (path.ImmutablePath, time.Duration, time.Time, error)

	// GetDNSLinkRecord returns the DNSLink TXT record for the provided FQDN.
	// Unlike ResolvePath, it does not perform recursive resolution. It only
	// checks for the existence of a DNSLink TXT record with path starting with
	// /ipfs/ or /ipns/ and returns the path as-is.
	GetDNSLinkRecord(context.Context, string) (path.Path, error)
}

IPFSBackend is the required set of functionality used to implement the IPFS HTTP Gateway specification.

The error returned by the implementer influences the status code that the user receives. By default, the gateway is able to handle a few error types, such as context.DeadlineExceeded, cid.ErrInvalidCid and various IPLD-pathing related errors.

To signal custom error types to the gateway, such that not everything is an 500 Internal Server Error, implementers can return errors wrapped in either ErrorRetryAfter or ErrorStatusCode.

type PublicGateway added in v0.10.0 

type PublicGateway struct {
	// Paths is explicit list of path prefixes that should be handled by
	// this gateway. Example: `["/ipfs", "/ipns"]`
	// Useful if you only want to support immutable `/ipfs`.
	Paths []string

	// UseSubdomains indicates whether or not this is a [Subdomain Gateway].
	//
	// If this flag is set, any `/ipns/$id` and/or `/ipfs/$id` paths in Paths
	// will be permanently redirected to `http(s)://$id.[ipns|ipfs].$gateway/`.
	//
	// We do not support using both paths and subdomains for a single domain
	// for security reasons ([Origin isolation]).
	//
	// [Subdomain Gateway]: https://specs.ipfs.tech/http-gateways/subdomain-gateway/
	// [Origin isolation]: https://en.wikipedia.org/wiki/Same-origin_policy
	UseSubdomains bool

	// NoDNSLink configures this gateway to _not_ resolve DNSLink for the
	// specific FQDN provided in `Host` HTTP header. Useful when you want to
	// explicitly allow or refuse hosting a single hostname. To refuse all
	// DNSLinks in `Host` processing, set NoDNSLink in Config instead. This setting
	// overrides the global setting.
	NoDNSLink bool

	// InlineDNSLink configures this gateway to always inline DNSLink names
	// (FQDN) into a single DNS label in order to interop with wildcard TLS certs
	// and Origin per CID isolation provided by rules like https://publicsuffix.org
	//
	// This should be set to true if you use HTTPS.
	InlineDNSLink bool

	// DeserializedResponses configures this gateway to support returning data
	// in deserialized format. This setting overrides the global setting.
	DeserializedResponses bool
}

PublicGateway is the specification of an IPFS Public Gateway.

type RequestContextKey 

type RequestContextKey string

RequestContextKey is a type representing a context.Context value key. 

const (
	// GatewayHostnameKey is the key for the hostname at which the gateway is
	// operating. It may be a DNSLink, Subdomain or Regular gateway.
	GatewayHostnameKey RequestContextKey = "gw-hostname"

	// DNSLinkHostnameKey is the key for the hostname of a [DNSLink Gateway].
	//
	// [DNSLink Gateway]: https://specs.ipfs.tech/http-gateways/dnslink-gateway/
	DNSLinkHostnameKey RequestContextKey = "dnslink-hostname"

	// SubdomainHostnameKey is the key for the hostname of a [Subdomain Gateway].
	//
	// [Subdomain Gateway]: https://specs.ipfs.tech/http-gateways/subdomain-gateway/
	SubdomainHostnameKey RequestContextKey = "subdomain-hostname"

	// ContentPathKey is the key for the original [http.Request] URL Path, as an [ipath.Path].
	ContentPathKey RequestContextKey = "content-path"
)

type WithContextHint added in v0.18.0 

type WithContextHint interface {
	// WrapContextForRequest allows the backend to add request scopped modifications to the context, like debug values or value caches.
	// There are no promises on actual usage in consumers.
	WrapContextForRequest(context.Context) context.Context
}

WithContextHint allows an IPFSBackend to inject custom context.Context configurations. This should be considered optional, consumers might only make a best effort attempt at calling WrapContextForRequest on requests.

Source Files

View all Source files

Directories

Path Synopsis
test

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.