repo

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 26, 2019 License: GPL-3.0 Imports: 16 Imported by: 2

Documentation

Overview

Package repo represents a repository of qri information Analogous to a git repository, repo expects a rigid structure filled with rich types specific to qri.

Index

Constants

View Source 
const (
	// ETDsCreated represents a peer creating a dataset (either updating an existing or a new dataset)
	ETDsCreated = EventType("ds_created")
	// ETDsDeleted represents destroying a dataset. Peers should respect this and remove locally as well
	ETDsDeleted = EventType("ds_deleted")
	// ETDsRenamed represents changing a dataset's name. Peers should update their refstore
	ETDsRenamed = EventType("ds_renamed")
	// ETDsPinned represents a peer pinning a dataset to their local storage
	ETDsPinned = EventType("ds_pinned")
	// ETDsUnpinned represents a peer unpinnning a dataset from local storage
	ETDsUnpinned = EventType("ds_unpinned")
	// ETDsAdded represents adding a reference to another peer's dataset to their node
	ETDsAdded = EventType("ds_added")
	// ETTransformExecuted represents running a transformation
	ETTransformExecuted = EventType("tf_executed")
	// ETCronJobRan is a cron job that's executed
	ETCronJobRan = EventType("cron_job_ran")
)

Variables

View Source 
var (
	// DefaultQriLocation is where qri data defaults to storing. The keyword $HOME
	// (and only $HOME) will be replaced with the current user home directory
	DefaultQriLocation = "$HOME/.qri"

	// ErrNotFound is the err implementers should return when stuff isn't found
	ErrNotFound = fmt.Errorf("repo: not found")
	// ErrNoHistory is the err implementers should return when no versions exist in history
	ErrNoHistory = fmt.Errorf("repo: no history")
	// ErrPeerIDRequired is for when a peerID is missing-but-expected
	ErrPeerIDRequired = fmt.Errorf("repo: peerID is required")
	// ErrPeernameRequired is for when a peername is missing-but-expected
	ErrPeernameRequired = fmt.Errorf("repo: peername is required")
	// ErrNameRequired is for when a name is missing-but-expected
	ErrNameRequired = fmt.Errorf("repo: name is required")
	// ErrPathRequired is for when a path is missing-but-expected
	ErrPathRequired = fmt.Errorf("repo: path is required")
	// ErrNameTaken is for when a name name is already taken
	ErrNameTaken = fmt.Errorf("repo: name already in use")
	// ErrRepoEmpty is for when the repo has no datasets
	ErrRepoEmpty = fmt.Errorf("repo: this repo contains no datasets")
	// ErrNotPinner is for when the repo doesn't have the concept of pinning as a feature
	ErrNotPinner = fmt.Errorf("repo: backing store doesn't support pinning")
	// ErrNoRegistry indicates no regsitry is currently configured
	ErrNoRegistry = fmt.Errorf("no configured registry")
	// ErrEmptyRef indicates that the given reference is empty
	ErrEmptyRef = fmt.Errorf("repo: empty dataset reference")
)

Functions

func CanonicalizeDatasetRef added in v0.1.1 

func CanonicalizeDatasetRef(r Repo, ref *DatasetRef) error

CanonicalizeDatasetRef uses the user's repo to turn any local aliases into full dataset references using known canonical peernames and paths. If the provided reference is not in the local repo, still do the work of handling aliases, but return a repo.ErrNotFound error, which callers can respond to by possibly contacting remote repos.

func CanonicalizeProfile added in v0.3.0 

func CanonicalizeProfile(r Repo, ref *DatasetRef) error

CanonicalizeProfile populates dataset DatasetRef ProfileID and Peername properties, changing aliases to known names, and adding ProfileID from a peerstore

func CompareDatasetRef 

func CompareDatasetRef(a, b DatasetRef) error

CompareDatasetRef compares two Dataset References, returning an error describing any difference between the two references

func IsRefString added in v0.7.1 

func IsRefString(path string) bool

IsRefString checks to see if a reference is a valid dataset ref string

Types

type DatasetRef 

type DatasetRef struct {
	// Peername of dataset owner
	Peername string `json:"peername,omitempty"`
	// ProfileID of dataset owner
	ProfileID profile.ID `json:"profileID,omitempty"`
	// Unique name reference for this dataset
	Name string `json:"name,omitempty"`
	// Content-addressed path for this dataset
	Path string `json:"path,omitempty"`
	// FSIPath is this dataset's link to the local filesystem if one exists
	FSIPath string `json:"fsiPath,omitempty"`
	// Dataset is a pointer to the dataset being referenced
	Dataset *dataset.Dataset `json:"dataset,omitempty"`
	// Published indicates whether this reference is listed as an available dataset
	Published bool `json:"published"`
	// If true, this reference doesn't exist locally
	Foreign bool `json:"foreign,omitempty"`
}

DatasetRef encapsulates a reference to a dataset. This needs to exist to bind ways of referring to a dataset to a dataset itself, as datasets can't easily contain their own hash information, and names are unique on a per-repository basis. It's tempting to think this needs to be "bigger", supporting more fields, keep in mind that if the information is important at all, it should be stored as metadata within the dataset itself.

func MustParseDatasetRef added in v0.5.2 

func MustParseDatasetRef(refstr string) DatasetRef

MustParseDatasetRef panics if the reference is invalid. Useful for testing

func ParseDatasetRef 

func ParseDatasetRef(ref string) (DatasetRef, error)

ParseDatasetRef decodes a dataset reference from a string value It’s possible to refer to a dataset in a number of ways. The full definition of a dataset reference is as follows: 

dataset_reference = peer_name/dataset_name@peer_id/network/hash

we swap in defaults as follows, all of which are represented as empty strings: 

network - defaults to /ipfs/
hash - tip of version history (latest known commit)

these defaults are currently enforced by convention. TODO - make Dataset Ref parsing the responisiblity of the Repo interface, replacing empty strings with actual defaults

dataset names & hashes are disambiguated by checking if the input parses to a valid multihash after base58 decoding. through defaults & base58 checking the following should all parse: 

peer_name/dataset_name
/network/hash
peername
peer_id
@peer_id
@peer_id/network/hash

see tests for more exmples

TODO - add validation that prevents peernames from being valid base58 multihashes and makes sure hashes are actually valid base58 multihashes TODO - figure out how IPFS CID's play into this

func (DatasetRef) AliasString added in v0.2.0 

func (r DatasetRef) AliasString() (s string)

AliasString returns the alias components of a DatasetRef as a string

func (DatasetRef) Complete added in v0.6.1 

func (r DatasetRef) Complete() bool

Complete returns true if a dataset has Peername, Name, ProfileID and Path properties set

func (DatasetRef) Equal added in v0.1.1 

func (r DatasetRef) Equal(b DatasetRef) bool

Equal returns true only if Peername Name and Path are equal

func (DatasetRef) FlatbufferBytes added in v0.9.0 

func (r DatasetRef) FlatbufferBytes() []byte

FlatbufferBytes formats a ref as a flatbuffer byte slice

func (DatasetRef) IsEmpty added in v0.1.1 

func (r DatasetRef) IsEmpty() bool

IsEmpty returns true if none of it's fields are set

func (DatasetRef) IsPeerRef 

func (r DatasetRef) IsPeerRef() bool

IsPeerRef returns true if only Peername is set

func (DatasetRef) MarshalFlatbuffer added in v0.9.0 

func (r DatasetRef) MarshalFlatbuffer(builder *flatbuffers.Builder) flatbuffers.UOffsetT

MarshalFlatbuffer writes a ref to a builder

func (DatasetRef) Match added in v0.1.1 

func (r DatasetRef) Match(b DatasetRef) bool

Match checks returns true if Peername and Name are equal, and/or path is equal

func (DatasetRef) String 

func (r DatasetRef) String() (s string)

String implements the Stringer interface for DatasetRef

func (*DatasetRef) UnmarshalFlatbuffer added in v0.9.0 

func (r *DatasetRef) UnmarshalFlatbuffer(rfb *repofb.DatasetRef) (err error)

UnmarshalFlatbuffer decodes a job from a flatbuffer

type Event added in v0.3.0 

type Event struct {
	Time         time.Time
	Type         EventType
	Ref          DatasetRef
	PeerIDString string
	Params       interface{}
}

Event is a list of details for logging a query

type EventLog added in v0.3.0 

type EventLog interface {
	LogEvent(t EventType, ref DatasetRef) error
	Events(limit, offset int) ([]*Event, error)
	EventsSince(time.Time) ([]*Event, error)
}

EventLog keeps logs

type EventType added in v0.3.0 

type EventType string

EventType classifies types of events that can be logged

type MemEventLog added in v0.3.0 

type MemEventLog []*Event

MemEventLog is an in-memory implementation of the EventLog interface

func (MemEventLog) Events added in v0.3.0 

func (log MemEventLog) Events(limit, offset int) ([]*Event, error)

Events grabs a set of Events from the store

func (MemEventLog) EventsSince added in v0.3.0 

func (log MemEventLog) EventsSince(t time.Time) ([]*Event, error)

EventsSince produces a slice of all events since a given time

func (*MemEventLog) LogEvent added in v0.3.0 

func (log *MemEventLog) LogEvent(t EventType, ref DatasetRef) error

LogEvent adds a query entry to the store

func (*MemEventLog) LogEventDetails added in v0.3.2 

func (log *MemEventLog) LogEventDetails(t EventType, when int64, peerID peer.ID, ref DatasetRef, params interface{}) error

LogEventDetails adds an entry to the log TODO: Update LogEvent to work like this, update callers.

type MemRefstore added in v0.1.1 

type MemRefstore []DatasetRef

MemRefstore is an in-memory implementation of the Namestore interface

func (*MemRefstore) DeleteRef added in v0.1.1 

func (r *MemRefstore) DeleteRef(del DatasetRef) error

DeleteRef removes a name from the store

func (MemRefstore) GetRef added in v0.1.1 

func (r MemRefstore) GetRef(get DatasetRef) (ref DatasetRef, err error)

GetRef completes a reference with , refs can have either Path or Peername & Name specified, GetRef should fill out the missing pieces

func (*MemRefstore) PutRef added in v0.1.1 

func (r *MemRefstore) PutRef(put DatasetRef) error

PutRef adds a reference to the namestore. Only complete references may be added

func (MemRefstore) RefCount added in v0.1.1 

func (r MemRefstore) RefCount() (int, error)

RefCount returns the total number of names in the store

func (MemRefstore) References added in v0.1.1 

func (r MemRefstore) References(offset, limit int) ([]DatasetRef, error)

References grabs a set of names from the Store's namespace

type MemRepo 

type MemRepo struct {
	*MemRefstore
	*MemEventLog
	// contains filtered or unexported fields
}

MemRepo is an in-memory implementation of the Repo interface

func NewMemRepo 

func NewMemRepo(p *profile.Profile, store cafs.Filestore, fsys qfs.Filesystem, ps profile.Store) (*MemRepo, error)

NewMemRepo creates a new in-memory repository

func (*MemRepo) Filesystem added in v0.7.0 

func (r *MemRepo) Filesystem() qfs.Filesystem

Filesystem gives access to the underlying filesystem

func (*MemRepo) PrivateKey added in v0.3.0 

func (r *MemRepo) PrivateKey() crypto.PrivKey

PrivateKey returns this repo's private key

func (*MemRepo) Profile 

func (r *MemRepo) Profile() (*profile.Profile, error)

Profile returns the peer profile for this repository

func (*MemRepo) Profiles added in v0.3.0 

func (r *MemRepo) Profiles() profile.Store

Profiles gives this repo's Peer interface implementation

func (*MemRepo) RefCache added in v0.3.0 

func (r *MemRepo) RefCache() Refstore

RefCache gives access to the ephemeral Refstore

func (*MemRepo) SetFilesystem added in v0.8.0 

func (r *MemRepo) SetFilesystem(fs qfs.Filesystem)

SetFilesystem implements QFSSetter, currently used during lib contstruction

func (*MemRepo) SetProfile added in v0.3.0 

func (r *MemRepo) SetProfile(p *profile.Profile) error

SetProfile updates this repo's profile

func (*MemRepo) Store 

func (r *MemRepo) Store() cafs.Filestore

Store returns the underlying cafs.Filestore for this repo

type QFSSetter added in v0.8.0 

type QFSSetter interface {
	SetFilesystem(qfs.Filesystem)
}

QFSSetter sets a qfs.Filesystem the whole interface is a short-term hack that should only need to be called in one context: when lib is setting up a Repo TODO (b5): either decouple repo & qfs completely, or merge them

type Refs added in v0.9.0 

type Refs []DatasetRef

Refs is a list of refs

func UnmarshalRefsFlatbuffer added in v0.9.0 

func UnmarshalRefsFlatbuffer(data []byte) (ls Refs, err error)

UnmarshalRefsFlatbuffer turns a repo flatbuffer into a list of refs

func (Refs) FlatbufferBytes added in v0.9.0 

func (rs Refs) FlatbufferBytes() []byte

FlatbufferBytes turns refs into a byte slice of flatbuffer data

func (Refs) Len added in v0.9.0 

func (rs Refs) Len() int

Len returns the length of refs

func (Refs) Less added in v0.9.0 

func (rs Refs) Less(i, j int) bool

Less returns true if i comes before j

func (Refs) Remove added in v0.9.0 

func (rs Refs) Remove(i int) Refs

Remove deletes an entry from the list of refs at an index

func (Refs) Swap added in v0.9.0 

func (rs Refs) Swap(i, j int)

Swap flips the positions of i and j

type Refstore added in v0.1.1 

type Refstore interface {
	// PutRef adds a reference to the store. References must be complete with
	// Peername, Name, and Path specified
	PutRef(ref DatasetRef) error
	// GetRef "completes" a passed in alias (DatasetRef with at least Peername
	// and Name field specified), filling in missing fields with a stored ref
	// TODO - should we rename this to "CompleteRef"?
	GetRef(ref DatasetRef) (DatasetRef, error)
	// DeleteRef removes a reference from the store
	DeleteRef(ref DatasetRef) error
	// References returns a set of references from the store
	References(offset, limit int) ([]DatasetRef, error)
	// RefCount returns the number of references in the store
	RefCount() (int, error)
}

Refstore keeps a collection of dataset references, Refstores require complete references (with both alias and identifiers), and can carry only one of a given alias eg: putting peer/dataset@a/ipfs/b when a ref with alias peer/dataset is already in the store will overwrite the stored reference

type Repo 

type Repo interface {
	// All repositories wraps a content-addressed filestore as the cannonical
	// record of this repository's data. Store gives direct access to the
	// cafs.Filestore instance any given repo is using.
	Store() cafs.Filestore

	// Filesystem is currently a read-only source of Filesystem-like data
	// Filestores can multiplex to read from multiple sources like the local
	// filesystem, over http, or content-addressed filesystems.
	// the long term-plan is to merge Filestore & Store
	Filesystem() qfs.Filesystem

	// All Repos must keep a Refstore, defining a store of known datasets
	Refstore
	// EventLog keeps a log of Profile activity for this repo
	EventLog

	// A repository must maintain profile information about the owner of this dataset.
	// The value returned by Profile() should represent the peer.
	Profile() (*profile.Profile, error)
	// SetProfile sets this repo's current profile. Profiles must contain a private key
	SetProfile(*profile.Profile) error
	// PrivateKey hands over this repo's private key
	PrivateKey() crypto.PrivKey
	// A repository must maintain profile information about encountered peers.
	// Decsisions regarding retentaion of peers is left to the the implementation
	Profiles() profile.Store
}

Repo is the interface for working with a qri repository qri repos are stored graph of resources:datasets, known peers, analytics data, change requests, etc. Repos are connected to a single peer profile. Repos must wrap an underlying cafs.Filestore, which is intended to act as the canonical store of state across all peers that this repo may interact with.

type SearchParams 

type SearchParams struct {
	Q             string
	Limit, Offset int
}

SearchParams encapsulates parameters provided to Searchable.Search

type Searchable 

type Searchable interface {
	Search(p SearchParams) ([]DatasetRef, error)
}

Searchable is an opt-in interface for supporting repository search

Source Files

View all Source files

Directories

Path Synopsis
Package fsrepo is a file-system implementation of repo
 Package fsrepo is a file-system implementation of repo
Package gen contains routines that perform expensive cryptographic operations.
 Package gen contains routines that perform expensive cryptographic operations.
Package profile defines a qri peer profile
 Package profile defines a qri peer profile
Package search creates a local search index of a repo using the bleve search engine
 Package search creates a local search index of a repo using the bleve search engine
Package test contains a set of tests to ensure a repo implementation conforms to expected behaviors, calling RunRepoTests on a given repo implementation should pass all checks in order to properly work with Qri.
 Package test contains a set of tests to ensure a repo implementation conforms to expected behaviors, calling RunRepoTests on a given repo implementation should pass all checks in order to properly work with Qri.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.