client

package
v1.10.0-rc4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 6, 2020 License: Apache-2.0 Imports: 49 Imported by: 351

Documentation

Index

Constants

View Source 
const (
	// MaxListItemsLog specifies the maximum number of items we log in response to a List* API
	MaxListItemsLog = 10
	// StorageSecretName is the name of the Kubernetes secret in which
	// storage credentials are stored.
	StorageSecretName = "pachyderm-storage-secret"
)
View Source 
const (
	// PPSEtcdPrefixEnv is the environment variable that specifies the etcd
	// prefix that PPS uses.
	PPSEtcdPrefixEnv = "PPS_ETCD_PREFIX"
	// PPSWorkerIPEnv is the environment variable that a worker can use to
	// see its own IP.  The IP address is made available through the
	// Kubernetes downward API.
	PPSWorkerIPEnv = "PPS_WORKER_IP"
	// PPSPodNameEnv is the environment variable that a pod can use to
	// see its own name.  The pod name is made available through the
	// Kubernetes downward API.
	PPSPodNameEnv = "PPS_POD_NAME"
	// PPSPipelineNameEnv is the env var that sets the name of the pipeline
	// that the workers are running.
	PPSPipelineNameEnv = "PPS_PIPELINE_NAME"
	// PPSNamespaceEnv is the namespace in which pachyderm is deployed
	PPSNamespaceEnv = "PPS_NAMESPACE"
	// PPSJobIDEnv is the env var that sets the ID of the job that the
	// workers are running (if the workers belong to an orphan job, rather than a
	// pipeline).
	PPSJobIDEnv = "PPS_JOB_ID"
	// PPSSpecCommitEnv is the namespace in which pachyderm is deployed
	PPSSpecCommitEnv = "PPS_SPEC_COMMIT"
	// PPSInputPrefix is the prefix of the path where datums are downloaded
	// to.  A datum of an input named `XXX` is downloaded to `/pfs/XXX/`.
	PPSInputPrefix = "/pfs"
	// PPSScratchSpace is where pps workers store data while it's waiting to be
	// processed.
	PPSScratchSpace = ".scratch"
	// PPSWorkerPortEnv is environment variable name for the port that workers
	// use for their gRPC server
	PPSWorkerPortEnv = "PPS_WORKER_GRPC_PORT"
	// PPSWorkerVolume is the name of the volume in which workers store
	// data.
	PPSWorkerVolume = "pachyderm-worker"
	// PPSWorkerUserContainerName is the name of the container that runs
	// the user code to process data.
	PPSWorkerUserContainerName = "user"
	// PPSWorkerSidecarContainerName is the name of the sidecar container
	// that runs alongside of each worker container.
	PPSWorkerSidecarContainerName = "storage"
	// GCGenerationKey is the etcd key that stores a counter that the
	// GC utility increments when it runs, so as to invalidate all cache.
	GCGenerationKey = "gc-generation"
	// JobIDEnv is an env var that is added to the environment of user pipeline
	// code and indicates the id of the job currently being run.
	JobIDEnv = "PACH_JOB_ID"
	// OutputCommitIDEnv is an env var that is added to the environment of user
	// pipelined code and indicates the id of the output commit.
	OutputCommitIDEnv = "PACH_OUTPUT_COMMIT_ID"
	// PeerPortEnv is the env var that sets a custom peer port
	PeerPortEnv = "PEER_PORT"
)
View Source 
const DefaultDialTimeout = 30 * time.Second

DefaultDialTimeout is the max amount of time APIClient.connect() will wait for a connection to be established unless overridden by WithDialTimeout()

View Source 
const DefaultMaxConcurrentStreams = 100

DefaultMaxConcurrentStreams defines the max number of Putfiles or Getfiles happening simultaneously

Variables

This section is empty.

Functions

func DatumTagPrefix added in v1.5.1 

func DatumTagPrefix(salt string) string

DatumTagPrefix hashes a pipeline salt to a string of a fixed size for use as the prefix for datum output trees. This prefix allows us to do garbage collection correctly.

func DefaultDialOptions added in v1.7.4 

func DefaultDialOptions() []grpc.DialOption

DefaultDialOptions is a helper returning a slice of grpc.Dial options such that grpc.Dial() is synchronous: the call doesn't return until the connection has been established and it's safe to send RPCs

func GetDatumTotalTime added in v1.5.2 

func GetDatumTotalTime(s *pps.ProcessStats) time.Duration

GetDatumTotalTime sums the timing stats from a DatumInfo

func GetTransaction added in v1.8.8 

func GetTransaction(ctx context.Context) (*transaction.Transaction, error)

GetTransaction (should be run from the server-side) loads the active transaction from the grpc metadata and returns the associated transaction object - or `nil` if no transaction is set.

func NewBlock 

func NewBlock(hash string) *pfs.Block

NewBlock creates a pfs.Block.

func NewBlockRef added in v1.9.6 

func NewBlockRef(hash string, lower, upper uint64) *pfs.BlockRef

NewBlockRef creates a pfs.BlockRef.

func NewBranch added in v1.7.0 

func NewBranch(repoName string, branchName string) *pfs.Branch

NewBranch creates a pfs.Branch

func NewCommit 

func NewCommit(repoName string, commitID string) *pfs.Commit

NewCommit creates a pfs.Commit.

func NewCommitProvenance added in v1.8.8 

func NewCommitProvenance(repoName string, branchName string, commitID string) *pfs.CommitProvenance

NewCommitProvenance creates a pfs.CommitProvenance.

func NewCommitResponse added in v1.8.8 

func NewCommitResponse(commit *pfs.Commit) *transaction.TransactionResponse

NewCommitResponse is a helper function to instantiate a TransactionResponse for a transaction item that returns a Commit ID.

func NewCronInput added in v1.5.3 

func NewCronInput(name string, spec string) *pps.Input

NewCronInput returns an input which will trigger based on a timed schedule. It uses cron syntax to specify the schedule. The input will be exposed to jobs as `/pfs/<name>/<timestamp>`. The timestamp uses the RFC 3339 format, e.g. `2006-01-02T15:04:05Z07:00`. It only takes required options.

func NewCronInputOpts added in v1.9.12 

func NewCronInputOpts(name string, repo string, spec string, overwrite bool) *pps.Input

NewCronInputOpts returns an input which will trigger based on a timed schedule. It uses cron syntax to specify the schedule. The input will be exposed to jobs as `/pfs/<name>/<timestamp>`. The timestamp uses the RFC 3339 format, e.g. `2006-01-02T15:04:05Z07:00`. It includes all the options.

func NewCrossInput added in v1.4.6 

func NewCrossInput(input ...*pps.Input) *pps.Input

NewCrossInput returns an input which is the cross product of other inputs. That means that all combination of datums will be seen by the job / pipeline.

func NewFile 

func NewFile(repoName string, commitID string, path string) *pfs.File

NewFile creates a pfs.File.

func NewJob 

func NewJob(jobID string) *pps.Job

NewJob creates a pps.Job.

func NewJobInput 

func NewJobInput(repoName string, commitID string, glob string) *pps.JobInput

NewJobInput creates a pps.JobInput.

func NewJoinInput added in v1.8.8 

func NewJoinInput(input ...*pps.Input) *pps.Input

NewJoinInput returns an input which is the join of other inputs. That means that all combination of datums which match on `joinOn` will be seen by the job / pipeline.

func NewObject added in v1.8.1 

func NewObject(hash string) *pfs.Object

NewObject creates a pfs.Object.

func NewPFSInput added in v1.8.1 

func NewPFSInput(repo string, glob string) *pps.Input

NewPFSInput returns a new PFS input. It only includes required options.

func NewPFSInputOpts added in v1.8.1 

func NewPFSInputOpts(name string, repo string, branch string, glob string, joinOn string, lazy bool) *pps.Input

NewPFSInputOpts returns a new PFS input. It includes all options.

func NewPipeline 

func NewPipeline(pipelineName string) *pps.Pipeline

NewPipeline creates a pps.Pipeline.

func NewRepo 

func NewRepo(repoName string) *pfs.Repo

NewRepo creates a pfs.Repo.

func NewS3PFSInput 

func NewS3PFSInput(name string, repo string, branch string) *pps.Input

NewS3PFSInput returns a new PFS input with 'S3' set.

func NewTag added in v1.8.1 

func NewTag(name string) *pfs.Tag

NewTag creates a pfs.Tag.

func NewUnionInput added in v1.4.6 

func NewUnionInput(input ...*pps.Input) *pps.Input

NewUnionInput returns an input which is the union of other inputs. That means that all datums from any of the inputs will be seen individually by the job / pipeline.

func WithSystemCAs added in v1.9.6 

func WithSystemCAs(settings *clientSettings) error

WithSystemCAs uses the system certs for client creatin.

Types

type APIClient 

type APIClient struct {
	PfsAPIClient
	PpsAPIClient
	ObjectAPIClient
	AuthAPIClient
	VersionAPIClient
	AdminAPIClient
	TransactionAPIClient
	DebugClient
	Enterprise enterprise.APIClient // not embedded--method name conflicts with AuthAPIClient
	// contains filtered or unexported fields
}

An APIClient is a wrapper around pfs, pps and block APIClients.

func NewForTest added in v1.8.8 

func NewForTest() (*APIClient, error)

NewForTest constructs a new APIClient for tests.

func NewFromAddress 

func NewFromAddress(addr string, options ...Option) (*APIClient, error)

NewFromAddress constructs a new APIClient for the server at addr.

func NewInCluster 

func NewInCluster(options ...Option) (*APIClient, error)

NewInCluster constructs a new APIClient using env vars that Kubernetes creates. This should be used to access Pachyderm from within a Kubernetes cluster with Pachyderm running on it.

func NewOnUserMachine added in v1.5.0 

func NewOnUserMachine(prefix string, options ...Option) (*APIClient, error)

NewOnUserMachine constructs a new APIClient using $HOME/.pachyderm/config if it exists. This is intended to be used in the pachctl binary.

TODO(msteffen) this logic is fairly linux/unix specific, and makes the pachyderm client library incompatible with Windows. We may want to move this (and similar) logic into src/server and have it call a NewFromOptions() constructor.

func (*APIClient) AddMetadata added in v1.5.1 

func (c *APIClient) AddMetadata(ctx context.Context) context.Context

AddMetadata adds necessary metadata (including authentication credentials) to the context 'ctx', preserving any metadata that is present in either the incoming or outgoing metadata of 'ctx'.

func (APIClient) Binary added in v1.8.6 

func (c APIClient) Binary(w io.Writer) error

Binary writes the running pachd binary to w.

func (APIClient) BlockCommit added in v1.7.0 

func (c APIClient) BlockCommit(repoName string, commitID string) (*pfs.CommitInfo, error)

BlockCommit returns info about a specific Commit, but blocks until that commit has been finished.

func (APIClient) BuildCommit added in v1.6.0 

func (c APIClient) BuildCommit(repoName string, branch string, parent string, treeObject string, sizeBytes uint64) (*pfs.Commit, error)

BuildCommit builds a commit in a single call from an existing HashTree that has already been written to the object store. Note this is a more advanced pattern for creating commits that's mostly used internally.

func (*APIClient) Close added in v1.2.0 

func (c *APIClient) Close() error

Close the connection to gRPC

func (APIClient) Compact added in v1.3.9 

func (c APIClient) Compact() error

Compact forces compaction of objects.

func (APIClient) CopyFile added in v1.6.0 

func (c APIClient) CopyFile(srcRepo, srcCommit, srcPath, dstRepo, dstCommit, dstPath string, overwrite bool) error

CopyFile copys a file from one pfs location to another. It can be used on directories or regular files.

func (APIClient) CreateBranch added in v1.7.0 

func (c APIClient) CreateBranch(repoName string, branch string, commit string, provenance []*pfs.Branch) error

CreateBranch creates a new branch

func (APIClient) CreateJob 

func (c APIClient) CreateJob(pipeline string, outputCommit, statsCommit *pfs.Commit) (*pps.Job, error)

CreateJob creates and runs a job in PPS. This function is mostly useful internally, users should generally run work by creating pipelines as well.

func (APIClient) CreateObject added in v1.9.6 

func (c APIClient) CreateObject(hash, block string, lower, upper uint64) error

CreateObject creates an object with hash, referencing the range [lower,upper] in block. The block should already exist.

func (APIClient) CreatePipeline 

func (c APIClient) CreatePipeline(
	name string,
	image string,
	cmd []string,
	stdin []string,
	parallelismSpec *pps.ParallelismSpec,
	input *pps.Input,
	outputBranch string,
	update bool,
) error

CreatePipeline creates a new pipeline, pipelines are the main computation object in PPS they create a flow of data from a set of input Repos to an output Repo (which has the same name as the pipeline). Whenever new data is committed to one of the input repos the pipelines will create jobs to bring the output Repo up to data. image is the Docker image to run the jobs in. cmd is the command passed to the Docker run invocation. NOTE as with Docker cmd is not run inside a shell that means that things like wildcard globbing (*), pipes (|) and file redirects (> and >>) will not work. To get that behavior you should have your command be a shell of your choice and pass a shell script to stdin. stdin is a slice of lines that are sent to your command on stdin. Lines need not end in newline characters. parallelism is how many copies of your container should run in parallel. You may pass 0 for parallelism in which case PPS will set the parallelism based on available resources. input specifies a set of Repos that will be visible to the jobs during runtime. commits to these repos will cause the pipeline to create new jobs to process them. update indicates that you want to update an existing pipeline

func (APIClient) CreatePipelineService added in v1.6.0 

func (c APIClient) CreatePipelineService(
	name string,
	image string,
	cmd []string,
	stdin []string,
	parallelismSpec *pps.ParallelismSpec,
	input *pps.Input,
	update bool,
	internalPort int32,
	externalPort int32,
	annotations map[string]string,
) error

CreatePipelineService creates a new pipeline service.

func (APIClient) CreateRepo 

func (c APIClient) CreateRepo(repoName string) error

CreateRepo creates a new Repo object in pfs with the given name. Repos are the top level data object in pfs and should be used to store data of a similar type. For example rather than having a single Repo for an entire project you might have separate Repos for logs, metrics, database dumps etc.

func (APIClient) CreateSecret added in v1.9.12 

func (c APIClient) CreateSecret(file []byte) error

CreateSecret creates a secret on the cluster.

func (*APIClient) Ctx added in v1.5.1 

func (c *APIClient) Ctx() context.Context

Ctx is a convenience function that returns adds Pachyderm authn metadata to context.Background().

func (APIClient) DeleteAll 

func (c APIClient) DeleteAll() error

DeleteAll deletes everything in the cluster. Use with caution, there is no undo. TODO: rewrite this to use transactions

func (APIClient) DeleteBranch added in v1.3.19 

func (c APIClient) DeleteBranch(repoName string, branch string, force bool) error

DeleteBranch deletes a branch, but leaves the commits themselves intact. In other words, those commits can still be accessed via commit IDs and other branches they happen to be on.

func (APIClient) DeleteCommit 

func (c APIClient) DeleteCommit(repoName string, commitID string) error

DeleteCommit deletes a commit.

func (APIClient) DeleteFile 

func (c APIClient) DeleteFile(repoName string, commitID string, path string) error

DeleteFile deletes a file from a Commit. DeleteFile leaves a tombstone in the Commit, assuming the file isn't written to later attempting to get the file from the finished commit will result in not found error. The file will of course remain intact in the Commit's parent.

func (APIClient) DeleteJob added in v1.3.0 

func (c APIClient) DeleteJob(jobID string) error

DeleteJob deletes a job.

func (APIClient) DeletePipeline 

func (c APIClient) DeletePipeline(name string, force bool) error

DeletePipeline deletes a pipeline along with its output Repo.

func (APIClient) DeleteRepo 

func (c APIClient) DeleteRepo(repoName string, force bool) error

DeleteRepo deletes a repo and reclaims the storage space it was using. Note that as of 1.0 we do not reclaim the blocks that the Repo was referencing, this is because they may also be referenced by other Repos and deleting them would make those Repos inaccessible. This will be resolved in later versions. If "force" is set to true, the repo will be removed regardless of errors. This argument should be used with care.

func (APIClient) DeleteSecret added in v1.9.12 

func (c APIClient) DeleteSecret(secret string) error

DeleteSecret deletes a secret from the cluster.

func (APIClient) DeleteTransaction added in v1.8.8 

func (c APIClient) DeleteTransaction(txn *transaction.Transaction) error

DeleteTransaction is an RPC that aborts an existing transaction in the Pachyderm cluster and removes it from the cluster.

func (APIClient) DiffFile added in v1.4.8 

func (c APIClient) DiffFile(newRepoName, newCommitID, newPath, oldRepoName,
	oldCommitID, oldPath string, shallow bool) ([]*pfs.FileInfo, []*pfs.FileInfo, error)

DiffFile returns the difference between 2 paths, old path may be omitted in which case the parent of the new path will be used. DiffFile return 2 values (unless it returns an error) the first value is files present under new path, the second is files present under old path, files which are under both paths and have identical content are omitted.

func (APIClient) Dump added in v1.7.5 

func (c APIClient) Dump(w io.Writer) error

Dump writes debug information from the server to w.

func (APIClient) ExecuteInTransaction added in v1.8.8 

func (c APIClient) ExecuteInTransaction(f func(c *APIClient) error) (*transaction.TransactionInfo, error)

ExecuteInTransaction executes a callback within a transaction. The callback should use the passed in APIClient. If the callback returns a nil error, then the transaction will be finished. If the callback returns a non-nil error, then the transaction will be deleted.

func (APIClient) Extract added in v1.6.9 

func (c APIClient) Extract(objects bool, f func(op *admin.Op) error) error

Extract all cluster state, call f with each operation.

func (APIClient) ExtractAll added in v1.6.9 

func (c APIClient) ExtractAll(objects bool) ([]*admin.Op, error)

ExtractAll cluster state as a slice of operations.

func (APIClient) ExtractPipeline added in v1.7.0 

func (c APIClient) ExtractPipeline(pipelineName string) (*pps.CreatePipelineRequest, error)

ExtractPipeline extracts a single pipeline.

func (APIClient) ExtractURL added in v1.6.9 

func (c APIClient) ExtractURL(url string) error

ExtractURL extracts all cluster state and marshalls it to object storage.

func (APIClient) ExtractWriter added in v1.6.9 

func (c APIClient) ExtractWriter(objects bool, w io.Writer) error

ExtractWriter extracts all cluster state and marshals it to w.

func (APIClient) FinishCommit 

func (c APIClient) FinishCommit(repoName string, commitID string) error

FinishCommit ends the process of committing data to a Repo and persists the Commit. Once a Commit is finished the data becomes immutable and future attempts to write to it with PutFile will error.

func (APIClient) FinishTransaction added in v1.8.8 

func (c APIClient) FinishTransaction(txn *transaction.Transaction) (*transaction.TransactionInfo, error)

FinishTransaction is an RPC that closes an existing transaction in the Pachyderm cluster and commits its changes to the persisted cluster metadata transactionally.

func (APIClient) FlushCommit 

func (c APIClient) FlushCommit(commits []*pfs.Commit, toRepos []*pfs.Repo) (CommitInfoIterator, error)

FlushCommit returns an iterator that returns commits that have the specified `commits` as provenance. Note that the iterator can block if jobs have not successfully completed. This in effect waits for all of the jobs that are triggered by a set of commits to complete.

If toRepos is not nil then only the commits up to and including those repos will be considered, otherwise all repos are considered.

Note that it's never necessary to call FlushCommit to run jobs, they'll run no matter what, FlushCommit just allows you to wait for them to complete and see their output once they do.

func (APIClient) FlushCommitAll added in v1.8.8 

func (c APIClient) FlushCommitAll(commits []*pfs.Commit, toRepos []*pfs.Repo) ([]*pfs.CommitInfo, error)

FlushCommitAll returns commits that have the specified `commits` as provenance. Note that it can block if jobs have not successfully completed. This in effect waits for all of the jobs that are triggered by a set of commits to complete.

If toRepos is not nil then only the commits up to and including those repos will be considered, otherwise all repos are considered.

Note that it's never necessary to call FlushCommit to run jobs, they'll run no matter what, FlushCommitAll just allows you to wait for them to complete and see their output once they do.

func (APIClient) FlushCommitF added in v1.7.0 

func (c APIClient) FlushCommitF(commits []*pfs.Commit, toRepos []*pfs.Repo, f func(*pfs.CommitInfo) error) error

FlushCommitF calls f with commits that have the specified `commits` as provenance. Note that it can block if jobs have not successfully completed. This in effect waits for all of the jobs that are triggered by a set of commits to complete.

If toRepos is not nil then only the commits up to and including those repos will be considered, otherwise all repos are considered.

Note that it's never necessary to call FlushCommit to run jobs, they'll run no matter what, FlushCommitF just allows you to wait for them to complete and see their output once they do.

func (APIClient) FlushJob added in v1.7.0 

func (c APIClient) FlushJob(commits []*pfs.Commit, toPipelines []string, f func(*pps.JobInfo) error) error

FlushJob calls f with all the jobs which were triggered by commits. If toPipelines is non-nil then only the jobs between commits and those pipelines in the DAG will be returned.

func (APIClient) FlushJobAll added in v1.7.0 

func (c APIClient) FlushJobAll(commits []*pfs.Commit, toPipelines []string) ([]*pps.JobInfo, error)

FlushJobAll returns all the jobs which were triggered by commits. If toPipelines is non-nil then only the jobs between commits and those pipelines in the DAG will be returned.

func (APIClient) Fsck added in v1.8.8 

func (c APIClient) Fsck(fix bool, cb func(*pfs.FsckResponse) error) error

Fsck performs checks on pfs. Errors that are encountered will be passed onError. These aren't errors in the traditional sense, in that they don't prevent the completion of fsck. Errors that do prevent completion will be returned from the function.

func (APIClient) FsckFastExit added in v1.8.8 

func (c APIClient) FsckFastExit() error

FsckFastExit performs checks on pfs, similar to Fsck, except that it returns the first fsck error it encounters and exits.

func (APIClient) GarbageCollect added in v1.4.7 

func (c APIClient) GarbageCollect(memoryBytes int64) error

GarbageCollect garbage collects unused data. Currently GC needs to be run while no data is being added or removed (which, among other things, implies that there shouldn't be jobs actively running). Pfs Garbage collection uses bloom filters to keep track of live objects because it can store more objects than can be indexed in memory. This means that there is a chance for unreferenced objects to not be GCed, this chance increases as the number of objects in the system increases. You can tradeoff using more memory to get a lower chance of collisions, the default value is 10 MB and collisions should be unlikely until you have 10 million objects.

func (*APIClient) GetAddress added in v1.5.0 

func (c *APIClient) GetAddress() string

GetAddress returns the pachd host:port with which 'c' is communicating. If 'c' was created using NewInCluster or NewOnUserMachine then this is how the address may be retrieved from the environment.

func (APIClient) GetBlock 

func (c APIClient) GetBlock(hash string, w io.Writer) error

GetBlock gets the content of a block.

func (APIClient) GetFile 

func (c APIClient) GetFile(repoName string, commitID string, path string, offset int64, size int64, writer io.Writer) error

GetFile returns the contents of a file at a specific Commit. offset specifies a number of bytes that should be skipped in the beginning of the file. size limits the total amount of data returned, note you will get fewer bytes than size if you pass a value larger than the size of the file. If size is set to 0 then all of the data will be returned.

func (APIClient) GetFileReadSeeker added in v1.7.0 

func (c APIClient) GetFileReadSeeker(repoName string, commitID string, path string) (io.ReadSeeker, error)

GetFileReadSeeker returns a reader for the contents of a file at a specific Commit that permits Seeking to different points in the file.

func (APIClient) GetFileReader added in v1.3.19 

func (c APIClient) GetFileReader(repoName string, commitID string, path string, offset int64, size int64) (io.Reader, error)

GetFileReader returns a reader for the contents of a file at a specific Commit. offset specifies a number of bytes that should be skipped in the beginning of the file. size limits the total amount of data returned, note you will get fewer bytes than size if you pass a value larger than the size of the file. If size is set to 0 then all of the data will be returned.

func (APIClient) GetLogs 

func (c APIClient) GetLogs(
	pipelineName string,
	jobID string,
	data []string,
	datumID string,
	master bool,
	follow bool,
	tail int64,
) *LogsIter

GetLogs gets logs from a job (logs includes stdout and stderr). 'pipelineName', 'jobID', 'data', and 'datumID', are all filters. To forego any filter, simply pass an empty value, though one of 'pipelineName' and 'jobID' must be set. Responses are written to 'messages'

func (APIClient) GetObject added in v1.3.9 

func (c APIClient) GetObject(hash string, writer io.Writer) error

GetObject gets an object out of the object store by hash.

func (APIClient) GetObjectReader added in v1.8.0 

func (c APIClient) GetObjectReader(hash string) (io.ReadCloser, error)

GetObjectReader returns a reader for an object in object store by hash.

func (APIClient) GetObjects added in v1.3.19 

func (c APIClient) GetObjects(hashes []string, offset uint64, size uint64, totalSize uint64, writer io.Writer) error

GetObjects gets several objects out of the object store by hash.

func (APIClient) GetTag added in v1.3.9 

func (c APIClient) GetTag(tag string, writer io.Writer) error

GetTag gets an object out of the object store by tag.

func (APIClient) GetTagReader added in v1.8.0 

func (c APIClient) GetTagReader(tag string) (io.ReadCloser, error)

GetTagReader returns a reader for an object in object store by tag.

func (APIClient) GetTar 

func (c APIClient) GetTar(repo, commit, path string, w io.Writer) error

GetTar gets a tar stream out of PFS. Note: this should only be used for testing the new storage layer.

func (APIClient) GetTransaction added in v1.8.8 

func (c APIClient) GetTransaction() (*transaction.Transaction, error)

GetTransaction is a helper function to get the active transaction from the client's context metadata.

func (APIClient) GlobFile added in v1.4.6 

func (c APIClient) GlobFile(repoName string, commitID string, pattern string) ([]*pfs.FileInfo, error)

GlobFile returns files that match a given glob pattern in a given commit. The pattern is documented here: https://golang.org/pkg/path/filepath/#Match

func (APIClient) GlobFileF added in v1.8.8 

func (c APIClient) GlobFileF(repoName string, commitID string, pattern string, f func(fi *pfs.FileInfo) error) error

GlobFileF returns files that match a given glob pattern in a given commit, calling f with each FileInfo. The pattern is documented here: https://golang.org/pkg/path/filepath/#Match

func (APIClient) Health added in v1.7.0 

func (c APIClient) Health() error

Health health checks pachd, it returns an error if pachd isn't healthy.

func (APIClient) InspectBranch added in v1.7.0 

func (c APIClient) InspectBranch(repoName string, branch string) (*pfs.BranchInfo, error)

InspectBranch returns information on a specific PFS branch

func (APIClient) InspectCluster added in v1.7.0 

func (c APIClient) InspectCluster() (*admin.ClusterInfo, error)

InspectCluster retrieves cluster state

func (APIClient) InspectCommit 

func (c APIClient) InspectCommit(repoName string, commitID string) (*pfs.CommitInfo, error)

InspectCommit returns info about a specific Commit.

func (APIClient) InspectDatum added in v1.5.2 

func (c APIClient) InspectDatum(jobID string, datumID string) (*pps.DatumInfo, error)

InspectDatum returns info about a single datum

func (APIClient) InspectFile 

func (c APIClient) InspectFile(repoName string, commitID string, path string) (*pfs.FileInfo, error)

InspectFile returns info about a specific file.

func (APIClient) InspectJob 

func (c APIClient) InspectJob(jobID string, blockState bool) (*pps.JobInfo, error)

InspectJob returns info about a specific job. blockState will cause the call to block until the job reaches a terminal state (failure or success).

func (APIClient) InspectJobOutputCommit added in v1.7.1 

func (c APIClient) InspectJobOutputCommit(repoName, commitID string, blockState bool) (*pps.JobInfo, error)

InspectJobOutputCommit returns info about a job that created a commit. blockState will cause the call to block until the job reaches a terminal state (failure or success).

func (APIClient) InspectObject added in v1.3.9 

func (c APIClient) InspectObject(hash string) (*pfs.ObjectInfo, error)

InspectObject returns info about an Object.

func (APIClient) InspectPipeline 

func (c APIClient) InspectPipeline(pipelineName string) (*pps.PipelineInfo, error)

InspectPipeline returns info about a specific pipeline.

func (APIClient) InspectRepo 

func (c APIClient) InspectRepo(repoName string) (*pfs.RepoInfo, error)

InspectRepo returns info about a specific Repo.

func (APIClient) InspectSecret added in v1.9.12 

func (c APIClient) InspectSecret(secret string) (*pps.SecretInfo, error)

InspectSecret returns info about a specific secret.

func (APIClient) InspectTransaction added in v1.8.8 

func (c APIClient) InspectTransaction(txn *transaction.Transaction) (*transaction.TransactionInfo, error)

InspectTransaction is an RPC that fetches the detailed information for an existing transaction in the Pachyderm cluster.

func (APIClient) IsAuthActive added in v1.8.8 

func (c APIClient) IsAuthActive() (bool, error)

IsAuthActive returns whether auth is activated on the cluster

func (APIClient) ListBlock 

func (c APIClient) ListBlock(f func(*pfs.Block) error) error

ListBlock lists blocks stored in pfs.

func (APIClient) ListBranch 

func (c APIClient) ListBranch(repoName string) ([]*pfs.BranchInfo, error)

ListBranch lists the active branches on a Repo.

func (APIClient) ListCommit 

func (c APIClient) ListCommit(repoName string, to string, from string, number uint64) ([]*pfs.CommitInfo, error)

ListCommit lists commits. If only `repo` is given, all commits in the repo are returned. If `to` is given, only the ancestors of `to`, including `to` itself, are considered. If `from` is given, only the descendents of `from`, including `from` itself, are considered. If `to` and `from` are the same commit, no commits will be returned. `number` determines how many commits are returned. If `number` is 0, all commits that match the aforementioned criteria are returned.

func (APIClient) ListCommitByRepo added in v1.2.4 

func (c APIClient) ListCommitByRepo(repoName string) ([]*pfs.CommitInfo, error)

ListCommitByRepo lists all commits in a repo.

func (APIClient) ListCommitF added in v1.7.2 

func (c APIClient) ListCommitF(repoName string, to string, from string, number uint64, reverse bool, f func(*pfs.CommitInfo) error) error

ListCommitF lists commits, calling f with each commit. If only `repo` is given, all commits in the repo are returned. If `to` is given, only the ancestors of `to`, including `to` itself, are considered. If `from` is given, only the descendents of `from`, including `from` itself, are considered. If `to` and `from` are the same commit, no commits will be returned. `number` determines how many commits are returned. If `number` is 0, `reverse` lists the commits from oldest to newest, rather than newest to oldest all commits that match the aforementioned criteria are passed to f.

func (APIClient) ListDatum added in v1.5.2 

func (c APIClient) ListDatum(jobID string, pageSize int64, page int64) (*pps.ListDatumResponse, error)

ListDatum returns info about all datums in a Job

func (APIClient) ListDatumF added in v1.7.2 

func (c APIClient) ListDatumF(jobID string, pageSize int64, page int64, f func(di *pps.DatumInfo) error) error

ListDatumF returns info about all datums in a Job, calling f with each datum info.

func (APIClient) ListFile 

func (c APIClient) ListFile(repoName string, commitID string, path string) ([]*pfs.FileInfo, error)

ListFile returns info about all files in a Commit under path.

func (APIClient) ListFileF added in v1.7.2 

func (c APIClient) ListFileF(repoName string, commitID string, path string, history int64, f func(fi *pfs.FileInfo) error) error

ListFileF returns info about all files in a Commit under path, calling f with each FileInfo.

func (APIClient) ListFileHistory added in v1.8.1 

func (c APIClient) ListFileHistory(repoName string, commitID string, path string, history int64) ([]*pfs.FileInfo, error)

ListFileHistory returns info about all files and their history in a Commit under path.

func (APIClient) ListJob 

func (c APIClient) ListJob(pipelineName string, inputCommit []*pfs.Commit, outputCommit *pfs.Commit, history int64, includePipelineInfo bool) ([]*pps.JobInfo, error)

ListJob returns info about all jobs. If pipelineName is non empty then only jobs that were started by the named pipeline will be returned If inputCommit is non-nil then only jobs which took the specific commits as inputs will be returned. The order of the inputCommits doesn't matter. If outputCommit is non-nil then only the job which created that commit as output will be returned. 'history' controls whether jobs from historical versions of pipelines are returned, it has the following semantics: 0: Return jobs from the current version of the pipeline or pipelines. 1: Return the above and jobs from the next most recent version 2: etc. -1: Return jobs from all historical versions. 'includePipelineInfo' controls whether the JobInfo passed to 'f' includes details fromt the pipeline spec (e.g. the transform). Leaving this 'false' can improve performance.

func (APIClient) ListJobF added in v1.7.2 

func (c APIClient) ListJobF(pipelineName string, inputCommit []*pfs.Commit,
	outputCommit *pfs.Commit, history int64, includePipelineInfo bool,
	f func(*pps.JobInfo) error) error

ListJobF returns info about all jobs, calling f with each JobInfo. If f returns an error iteration of jobs will stop and ListJobF will return that error, unless the error is errutil.ErrBreak in which case it will return nil. If pipelineName is non empty then only jobs that were started by the named pipeline will be returned If inputCommit is non-nil then only jobs which took the specific commits as inputs will be returned. The order of the inputCommits doesn't matter. If outputCommit is non-nil then only the job which created that commit as output will be returned. 'history' controls whether jobs from historical versions of pipelines are returned, it has the following semantics: 0: Return jobs from the current version of the pipeline or pipelines. 1: Return the above and jobs from the next most recent version 2: etc. -1: Return jobs from all historical versions. 'includePipelineInfo' controls whether the JobInfo passed to 'f' includes details fromt the pipeline spec--setting this to 'false' can improve performance.

func (APIClient) ListObject added in v1.6.9 

func (c APIClient) ListObject(f func(*pfs.ObjectInfo) error) error

ListObject lists objects stored in pfs.

func (APIClient) ListPipeline 

func (c APIClient) ListPipeline() ([]*pps.PipelineInfo, error)

ListPipeline returns info about all pipelines.

func (APIClient) ListPipelineHistory added in v1.8.8 

func (c APIClient) ListPipelineHistory(pipeline string, history int64) ([]*pps.PipelineInfo, error)

ListPipelineHistory returns historical information about pipelines. `pipeline` specifies which pipeline to return history about, if it's equal to "" then ListPipelineHistory returns historical information about all pipelines. `history` specifies how many historical revisions to return: 0: Return the current version of the pipeline or pipelines. 1: Return the above and the next most recent version 2: etc. -1: Return all historical versions.

func (APIClient) ListRepo 

func (c APIClient) ListRepo() ([]*pfs.RepoInfo, error)

ListRepo returns info about all Repos. provenance specifies a set of provenance repos, only repos which have ALL of the specified repos as provenance will be returned unless provenance is nil in which case it is ignored.

func (APIClient) ListSecret added in v1.9.12 

func (c APIClient) ListSecret() ([]*pps.SecretInfo, error)

ListSecret returns info about all Pachyderm secrets.

func (APIClient) ListTag added in v1.6.9 

func (c APIClient) ListTag(f func(*pfs.ListTagsResponse) error) error

ListTag lists tags stored in pfs.

func (APIClient) ListTransaction added in v1.8.8 

func (c APIClient) ListTransaction() ([]*transaction.TransactionInfo, error)

ListTransaction is an RPC that fetches a list of all open transactions in the Pachyderm cluster.

func (APIClient) NewPutFileClient added in v1.7.7 

func (c APIClient) NewPutFileClient() (PutFileClient, error)

NewPutFileClient returns a new client for putting files into pfs in a single request.

func (APIClient) Profile added in v1.8.6 

func (c APIClient) Profile(profile string, duration time.Duration, w io.Writer) error

Profile writes a pprof profile for pachd to w.

func (APIClient) PutBlock 

func (c APIClient) PutBlock(hash string, _r io.Reader) (_ int64, retErr error)

PutBlock puts a block.

func (APIClient) PutFile 

func (c APIClient) PutFile(repoName string, commitID string, path string, reader io.Reader) (_ int, retErr error)

PutFile writes a file to PFS from a reader.

func (APIClient) PutFileOverwrite added in v1.5.3 

func (c APIClient) PutFileOverwrite(repoName string, commitID string, path string, reader io.Reader, overwriteIndex int64) (_ int, retErr error)

PutFileOverwrite is like PutFile but it overwrites the file rather than appending to it. overwriteIndex allows you to specify the index of the object starting from which you'd like to overwrite. If you want to overwrite the entire file, specify an index of 0.

func (APIClient) PutFileSplit added in v1.3.19 

func (c APIClient) PutFileSplit(repoName string, commitID string, path string, delimiter pfs.Delimiter, targetFileDatums int64, targetFileBytes int64, headerRecords int64, overwrite bool, reader io.Reader) (_ int, retErr error)

PutFileSplit writes a file to PFS from a reader delimiter is used to tell PFS how to break the input into blocks

func (APIClient) PutFileSplitWriter added in v1.3.19 

func (c APIClient) PutFileSplitWriter(repoName string, commitID string, path string,
	delimiter pfs.Delimiter, targetFileDatums int64, targetFileBytes int64, headerRecords int64, overwrite bool) (io.WriteCloser, error)

PutFileSplitWriter writes a multiple files to PFS by splitting up the data that is written to it. NOTE: PutFileSplitWriter returns an io.WriteCloser you must call Close on it when you are done writing.

func (APIClient) PutFileURL added in v1.2.0 

func (c APIClient) PutFileURL(repoName string, commitID string, path string, url string, recursive bool, overwrite bool) (retErr error)

PutFileURL puts a file using the content found at a URL. The URL is sent to the server which performs the request. recursive allow for recursive scraping of some types URLs for example on s3:// urls.

func (APIClient) PutFileWriter 

func (c APIClient) PutFileWriter(repoName string, commitID string, path string) (io.WriteCloser, error)

PutFileWriter writes a file to PFS. NOTE: PutFileWriter returns an io.WriteCloser you must call Close on it when you are done writing.

func (APIClient) PutObject added in v1.3.9 

func (c APIClient) PutObject(_r io.Reader, tags ...string) (object *pfs.Object, _ int64, retErr error)

PutObject puts a value into the object store and tags it with 0 or more tags.

func (APIClient) PutObjectAsync added in v1.8.0 

func (c APIClient) PutObjectAsync(tags []*pfs.Tag) (*PutObjectWriteCloserAsync, error)

PutObjectAsync puts a value into the object store asynchronously.

func (APIClient) PutObjectSplit added in v1.6.0 

func (c APIClient) PutObjectSplit(_r io.Reader) (objects []*pfs.Object, _ int64, retErr error)

PutObjectSplit is the same as PutObject except that the data is splitted into several smaller objects. This is primarily useful if you'd like to be able to resume upload.

func (APIClient) PutTar 

func (c APIClient) PutTar(repo, commit string, r io.Reader) (retErr error)

PutTar puts a tar stream into PFS. Note: this should only be used for testing the new storage layer.

func (APIClient) ReadObject added in v1.3.9 

func (c APIClient) ReadObject(hash string) ([]byte, error)

ReadObject gets an object by hash and returns it directly as []byte.

func (APIClient) ReadObjects added in v1.3.19 

func (c APIClient) ReadObjects(hashes []string, offset uint64, size uint64) ([]byte, error)

ReadObjects gets several objects by hash and returns them directly as []byte.

func (APIClient) ReadTag added in v1.3.9 

func (c APIClient) ReadTag(tag string) ([]byte, error)

ReadTag gets an object by tag and returns it directly as []byte.

func (APIClient) RestartDatum added in v1.4.4 

func (c APIClient) RestartDatum(jobID string, datumFilter []string) error

RestartDatum restarts a datum that's being processed as part of a job. datumFilter is a slice of strings which are matched against either the Path or Hash of the datum, the order of the strings in datumFilter is irrelevant.

func (APIClient) Restore added in v1.6.9 

func (c APIClient) Restore(ops []*admin.Op) (retErr error)

Restore cluster state from an extract series of operations.

func (APIClient) RestoreFrom added in v1.7.0 

func (c APIClient) RestoreFrom(objects bool, otherC *APIClient) (retErr error)

RestoreFrom restores state from another cluster which can be access through otherC.

func (APIClient) RestoreReader added in v1.6.9 

func (c APIClient) RestoreReader(r io.Reader) (retErr error)

RestoreReader restores cluster state from a reader containing marshaled ops. Such as those written by ExtractWriter.

func (APIClient) RestoreURL added in v1.6.9 

func (c APIClient) RestoreURL(url string) (retErr error)

RestoreURL restures cluster state from object storage.

func (APIClient) RunBatchInTransaction added in v1.9.11 

func (c APIClient) RunBatchInTransaction(cb func(builder *TransactionBuilder) error) (*transaction.TransactionInfo, error)

RunBatchInTransaction will execute a batch of API calls in a single round-trip transactionally. The callback is used to build the request, which is executed when the callback returns.

func (APIClient) RunCron added in v1.9.10 

func (c APIClient) RunCron(name string) error

RunCron runs a pipeline. It can be passed a list of commit provenance. This will trigger a new job provenant on those commits, effectively running the pipeline on the data in those commits.

func (APIClient) RunPipeline added in v1.8.8 

func (c APIClient) RunPipeline(name string, provenance []*pfs.CommitProvenance, jobID string) error

RunPipeline runs a pipeline. It can be passed a list of commit provenance. This will trigger a new job provenant on those commits, effectively running the pipeline on the data in those commits.

func (*APIClient) SetAuthToken added in v1.5.1 

func (c *APIClient) SetAuthToken(token string)

SetAuthToken sets the authentication token that will be used for all API calls for this client.

func (APIClient) SetBranch added in v1.3.19 

func (c APIClient) SetBranch(repoName string, commit string, branch string) error

SetBranch sets a commit and its ancestors as a branch. SetBranch is deprecated in favor of CreateBranch.

func (APIClient) SetMaxConcurrentStreams added in v1.3.15 

func (c APIClient) SetMaxConcurrentStreams(n int)

SetMaxConcurrentStreams Sets the maximum number of concurrent streams the client can have. It is not safe to call this operations while operations are outstanding.

func (APIClient) StartCommit 

func (c APIClient) StartCommit(repoName string, branch string) (*pfs.Commit, error)

StartCommit begins the process of committing data to a Repo. Once started you can write to the Commit with PutFile and when all the data has been written you must finish the Commit with FinishCommit. NOTE, data is not persisted until FinishCommit is called. branch is a more convenient way to build linear chains of commits. When a commit is started with a non empty branch the value of branch becomes an alias for the created Commit. This enables a more intuitive access pattern. When the commit is started on a branch the previous head of the branch is used as the parent of the commit.

func (APIClient) StartCommitParent added in v1.3.19 

func (c APIClient) StartCommitParent(repoName string, branch string, parentCommit string) (*pfs.Commit, error)

StartCommitParent begins the process of committing data to a Repo. Once started you can write to the Commit with PutFile and when all the data has been written you must finish the Commit with FinishCommit. NOTE, data is not persisted until FinishCommit is called. branch is a more convenient way to build linear chains of commits. When a commit is started with a non empty branch the value of branch becomes an alias for the created Commit. This enables a more intuitive access pattern. When the commit is started on a branch the previous head of the branch is used as the parent of the commit. parentCommit specifies the parent Commit, upon creation the new Commit will appear identical to the parent Commit, data can safely be added to the new commit without affecting the contents of the parent Commit. You may pass "" as parentCommit in which case the new Commit will have no parent and will initially appear empty.

func (APIClient) StartPipeline added in v1.2.0 

func (c APIClient) StartPipeline(name string) error

StartPipeline restarts a stopped pipeline.

func (APIClient) StartTransaction added in v1.8.8 

func (c APIClient) StartTransaction() (*transaction.Transaction, error)

StartTransaction is an RPC that registers a new transaction with the Pachyderm cluster and returns the identifier of the new transaction.

func (APIClient) StopJob added in v1.4.4 

func (c APIClient) StopJob(jobID string) error

StopJob stops a job.

func (APIClient) StopPipeline added in v1.2.0 

func (c APIClient) StopPipeline(name string) error

StopPipeline prevents a pipeline from processing things, it can be restarted with StartPipeline.

func (APIClient) SubscribeCommit added in v1.3.19 

func (c APIClient) SubscribeCommit(repo, branch string, prov *pfs.CommitProvenance, from string, state pfs.CommitState) (CommitInfoIterator, error)

SubscribeCommit is like ListCommit but it keeps listening for commits as they come in.

func (APIClient) SubscribeCommitF added in v1.7.0 

func (c APIClient) SubscribeCommitF(repo, branch string, prov *pfs.CommitProvenance, from string, state pfs.CommitState, f func(*pfs.CommitInfo) error) error

SubscribeCommitF is like ListCommit but it calls a callback function with the results rather than returning an iterator.

func (APIClient) TagObject added in v1.3.9 

func (c APIClient) TagObject(hash string, tags ...string) error

TagObject applies a tag to an existing object.

func (APIClient) UpdateRepo added in v1.8.8 

func (c APIClient) UpdateRepo(repoName string) error

UpdateRepo upserts a repo with the given name.

func (APIClient) Version added in v1.6.9 

func (c APIClient) Version() (string, error)

Version returns the version of pachd as a string.

func (APIClient) Walk added in v1.3.2 

func (c APIClient) Walk(repoName string, commitID string, path string, f WalkFn) error

Walk walks the pfs filesystem rooted at path. walkFn will be called for each file found under path in lexicographical order. This includes both regular files and directories.

func (*APIClient) WithCtx added in v1.5.1 

func (c *APIClient) WithCtx(ctx context.Context) *APIClient

WithCtx returns a new APIClient that uses ctx for requests it sends. Note that the new APIClient will still use the authentication token and metrics metadata of this client, so this is only useful for propagating other context-associated metadata.

func (APIClient) WithTransaction added in v1.8.8 

func (c APIClient) WithTransaction(txn *transaction.Transaction) *APIClient

WithTransaction (client-side) returns a new APIClient that will run supported write operations within the specified transaction.

type AdminAPIClient added in v1.6.9 

type AdminAPIClient admin.APIClient

AdminAPIClient is an alias of admin.APIClient

type AuthAPIClient added in v1.5.1 

type AuthAPIClient auth.APIClient

AuthAPIClient is an alias of auth.APIClient

type CommitInfoIterator added in v1.3.19 

type CommitInfoIterator interface {
	Next() (*pfs.CommitInfo, error)
	Close()
}

CommitInfoIterator wraps a stream of commits and makes them easy to iterate.

type DebugClient added in v1.7.5 

type DebugClient debug.DebugClient

DebugClient is an alias of debug.DebugClient

type LogsIter added in v1.3.19 

type LogsIter struct {
	// contains filtered or unexported fields
}

LogsIter iterates through log messages returned from pps.GetLogs. Logs can be fetched with 'Next()'. The log message received can be examined with 'Message()', and any errors can be examined with 'Err()'.

func (*LogsIter) Err added in v1.3.19 

func (l *LogsIter) Err() error

Err retrieves any errors encountered in the course of calling 'Next()'.

func (*LogsIter) Message added in v1.3.19 

func (l *LogsIter) Message() *pps.LogMessage

Message returns the most recently retrieve log message (as an annotated log line, in the form of a pps.LogMessage)

func (*LogsIter) Next added in v1.3.19 

func (l *LogsIter) Next() bool

Next retrieves the next relevant log message from pachd

type ObjectAPIClient added in v1.3.9 

type ObjectAPIClient pfs.ObjectAPIClient

ObjectAPIClient is an alias for pfs.ObjectAPIClient

type Option added in v1.7.4 

type Option func(*clientSettings) error

Option is a client creation option that may be passed to NewOnUserMachine(), or NewInCluster()

func WithAdditionalPachdCert added in v1.7.4 

func WithAdditionalPachdCert() Option

WithAdditionalPachdCert instructs the New* functions to additionally trust the signed cert mounted in Pachd's cert volume. This is used by Pachd when connecting to itself (if no cert is present, the clients cert pool will not be modified, so that if no other options have been passed, pachd will connect to itself over an insecure connection)

func WithAdditionalRootCAs added in v1.7.4 

func WithAdditionalRootCAs(pemBytes []byte) Option

WithAdditionalRootCAs instructs the New* functions to additionally trust the given base64-encoded, signed x509 certificates as root certificates. Introduced to pass certs in the Pachyderm config

func WithDialTimeout added in v1.8.0 

func WithDialTimeout(t time.Duration) Option

WithDialTimeout instructs the New* functions to use 't' as the deadline to connect to pachd

func WithMaxConcurrentStreams added in v1.7.4 

func WithMaxConcurrentStreams(streams int) Option

WithMaxConcurrentStreams instructs the New* functions to create client that can have at most 'streams' concurrent streams open with pachd at a time

func WithRootCAs added in v1.7.4 

func WithRootCAs(path string) Option

WithRootCAs instructs the New* functions to create client that uses the given signed x509 certificates as the trusted root certificates (instead of the system certs). Introduced to pass certs provided via command-line flags

type PfsAPIClient 

type PfsAPIClient pfs.APIClient

PfsAPIClient is an alias for pfs.APIClient.

type PortForwarder added in v1.8.3 

type PortForwarder struct {
	// contains filtered or unexported fields
}

PortForwarder handles proxying local traffic to a kubernetes pod

func NewPortForwarder added in v1.8.3 

func NewPortForwarder(context *config.Context, namespace string) (*PortForwarder, error)

NewPortForwarder creates a new port forwarder

func (*PortForwarder) Close added in v1.8.3 

func (f *PortForwarder) Close()

Close shuts down port forwarding.

func (*PortForwarder) Run added in v1.8.3 

func (f *PortForwarder) Run(appName string, localPort, remotePort uint16) (uint16, error)

Run starts the port forwarder. Returns after initialization is begun with the locally bound port and any initialization errors.

func (*PortForwarder) RunForDaemon added in v1.8.3 

func (f *PortForwarder) RunForDaemon(localPort, remotePort uint16) (uint16, error)

RunForDaemon creates a port forwarder for the pachd daemon.

func (*PortForwarder) RunForDashUI added in v1.8.3 

func (f *PortForwarder) RunForDashUI(localPort uint16) (uint16, error)

RunForDashUI creates a port forwarder for the dash UI.

func (*PortForwarder) RunForDashWebSocket added in v1.8.3 

func (f *PortForwarder) RunForDashWebSocket(localPort uint16) (uint16, error)

RunForDashWebSocket creates a port forwarder for the dash websocket.

func (*PortForwarder) RunForPFS added in v1.8.3 

func (f *PortForwarder) RunForPFS(localPort uint16) (uint16, error)

RunForPFS creates a port forwarder for PFS over HTTP.

func (*PortForwarder) RunForS3Gateway added in v1.8.8 

func (f *PortForwarder) RunForS3Gateway(localPort uint16) (uint16, error)

RunForS3Gateway creates a port forwarder for the s3gateway.

func (*PortForwarder) RunForSAMLACS added in v1.8.3 

func (f *PortForwarder) RunForSAMLACS(localPort uint16) (uint16, error)

RunForSAMLACS creates a port forwarder for SAML ACS.

type PpsAPIClient 

type PpsAPIClient pps.APIClient

PpsAPIClient is an alias for pps.APIClient.

type PutFileClient added in v1.7.7 

type PutFileClient interface {
	// PutFileWriter writes a file to PFS.
	// NOTE: PutFileWriter returns an io.WriteCloser that you must call Close on when
	// you are done writing.
	PutFileWriter(repoName, commitID, path string) (io.WriteCloser, error)

	// PutFileSplitWriter writes multiple files to PFS by splitting up the data
	// that is written to it.
	// NOTE: PutFileSplitWriter returns an io.WriteCloser that you must call Close on when
	// you are done writing.
	PutFileSplitWriter(repoName string, commitID string, path string, delimiter pfs.Delimiter, targetFileDatums int64, targetFileBytes int64, headerRecords int64, overwrite bool) (io.WriteCloser, error)

	// PutFile writes a file to PFS from a reader.
	PutFile(repoName string, commitID string, path string, reader io.Reader) (_ int, retErr error)

	// PutFileOverwrite is like PutFile but it overwrites the file rather than
	// appending to it. overwriteIndex allows you to specify the index of the
	// object starting from which you'd like to overwrite. If you want to
	// overwrite the entire file, specify an index of 0.
	PutFileOverwrite(repoName string, commitID string, path string, reader io.Reader, overwriteIndex int64) (_ int, retErr error)

	// PutFileSplit writes a file to PFS from a reader.
	// delimiter is used to tell PFS how to break the input into blocks.
	PutFileSplit(repoName string, commitID string, path string, delimiter pfs.Delimiter, targetFileDatums int64, targetFileBytes int64, headerRecords int64, overwrite bool, reader io.Reader) (_ int, retErr error)

	// PutFileURL puts a file using the content found at a URL.
	// The URL is sent to the server which performs the request.
	// recursive allows for recursive scraping of some types URLs. For example on s3:// urls.
	PutFileURL(repoName string, commitID string, path string, url string, recursive bool, overwrite bool) error

	// Close must be called after you're done using a PutFileClient.
	// Further requests will throw errors.
	Close() error
}

PutFileClient is a client interface for putting files. There are 2 implementations, 1 that does each file as a seperate request and one that does them all together in the same request.

type PutObjectWriteCloserAsync added in v1.8.0 

type PutObjectWriteCloserAsync struct {
	// contains filtered or unexported fields
}

PutObjectWriteCloserAsync wraps a put object call in an asynchronous buffered writer.

func (*PutObjectWriteCloserAsync) Close added in v1.8.0 

func (w *PutObjectWriteCloserAsync) Close() error

Close closes the writer.

func (*PutObjectWriteCloserAsync) Object added in v1.8.0 

func (w *PutObjectWriteCloserAsync) Object() (*pfs.Object, error)

Object gets the pfs object for this writer. This can only be called when the writer is closed (the put object call is complete)

func (*PutObjectWriteCloserAsync) Write added in v1.8.0 

func (w *PutObjectWriteCloserAsync) Write(p []byte) (int, error)

Write performs a write.

type TransactionAPIClient added in v1.8.8 

type TransactionAPIClient transaction.APIClient

TransactionAPIClient is an alias of transaction.APIClient

type TransactionBuilder added in v1.9.11 

type TransactionBuilder struct {
	APIClient
	// contains filtered or unexported fields
}

TransactionBuilder presents the same interface as a pachyderm APIClient, but captures requests rather than sending to the server. If a request is not supported by the transaction system, it immediately errors.

func (*TransactionBuilder) Close added in v1.9.11 

func (tb *TransactionBuilder) Close() error

Close does not exist on a TransactionBuilder because it doesn't represent ownership of a connection to the API server. We need this to shadow the inherited Close, though.

func (*TransactionBuilder) GetAddress added in v1.9.11 

func (tb *TransactionBuilder) GetAddress() string

GetAddress should not exist on a TransactionBuilder because it doesn't represent ownership of a connection to the API server, but it also doesn't return an error, so we just passthrough to the parent client's implementation.

type VersionAPIClient added in v1.6.9 

type VersionAPIClient versionpb.APIClient

VersionAPIClient is an alias of versionpb.APIClient

type WalkFn added in v1.3.2 

type WalkFn func(*pfs.FileInfo) error

WalkFn is the type of the function called for each file in Walk. Returning a non-nil error from WalkFn will result in Walk aborting and returning said error.

Source Files

View all Source files

Directories

Path Synopsis
v1_7/auth
v1_7/enterprise
v1_7/hashtree
v1_7/pfs
v1_7/pps
v1_8/auth
v1_8/enterprise
v1_8/pfs
v1_8/pps
v1_9/auth
v1_9/pfs
v1_9/pps
Package limit provides primitives to limit concurrency.
 Package limit provides primitives to limit concurrency.
pkg
config
discovery
grpcutil
pbutil
require
shard
tls
tracing
tracing/extended
versionpb

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.