smart

package
v0.0.0-...-2386a5e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2015 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source 
const PersistentTransportBufferSize = int64(131072)

Variables

This section is empty.

Functions

func ExtractStructFromJsonRawMessage 

func ExtractStructFromJsonRawMessage(raw *json.RawMessage, out interface{}) error

Late-bind a method-specific structure from the raw message

func InitCoreProviders 

func InitCoreProviders()

Init core smart providers

func RegisterSshTransportFactory 

func RegisterSshTransportFactory()

func RegisterTransportFactory 

func RegisterTransportFactory(f TransportFactory)

Registers an instance of a SmartTransportFactory for creating connections Must only be called from the main thread, not thread safe Later factories registered will take precedence over earlier ones (including core)

Types

type ConnectionError 

type ConnectionError error

Just a specially identified persistent connection error so we can re-try

type DownloadDeltaPrepareRequest 

type DownloadDeltaPrepareRequest struct {
	BaseLobSHA   string
	TargetLobSHA string
}

type DownloadDeltaPrepareResponse 

type DownloadDeltaPrepareResponse struct {
	Size int64
}

type DownloadDeltaStartRequest 

type DownloadDeltaStartRequest struct {
	BaseLobSHA   string
	TargetLobSHA string
	Size         int64
}

type DownloadFilePrepareRequest 

type DownloadFilePrepareRequest struct {
	LobSHA   string
	Type     string
	ChunkIdx int
}

type DownloadFilePrepareResponse 

type DownloadFilePrepareResponse struct {
	Size int64
}

type DownloadFileStartRequest 

type DownloadFileStartRequest struct {
	LobSHA   string
	Type     string
	ChunkIdx int
	Size     int64
}

type ExitRequest 

type ExitRequest struct {
}

type ExitResponse 

type ExitResponse struct {
}

type FileExistsOfSizeRequest 

type FileExistsOfSizeRequest struct {
	LobSHA   string
	Type     string
	ChunkIdx int
	Size     int64
}

type FileExistsOfSizeResponse 

type FileExistsOfSizeResponse struct {
	Result bool
}

type FileExistsRequest 

type FileExistsRequest struct {
	LobSHA   string
	Type     string
	ChunkIdx int
}

type FileExistsResponse 

type FileExistsResponse struct {
	Exists bool
	Size   int64
}

type GetFirstCompleteLOBFromListRequest 

type GetFirstCompleteLOBFromListRequest struct {
	LobSHAs []string
}

type GetFirstCompleteLOBFromListResponse 

type GetFirstCompleteLOBFromListResponse struct {
	FirstSHA string
}

type JsonRequest 

type JsonRequest struct {
	// Go's JSON support only uses public fields but JSON-RPC requires lower case
	Id     int
	Method string
	// RawMessage allows us to store late-resolved, message-specific nested types
	// requires an extra couple of steps though; even though RawMessage is a []byte, it's not
	// JSON itself. You need to convert JSON to/from RawMessage as well as JSON to/from the structure
	// - see RawMessage's own UnmarshalJSON/MarshalJSON for this extra step
	Params *json.RawMessage
}

Note *not* using net/rpc and net/rpc/jsonrpc because we want more control golang's rpc requires a certain method format (Object.Method) and also doesn't easily support interleaving with raw byte streams like we need to. as per http://www.jsonrpc.org/specification

func NewJsonRequest 

func NewJsonRequest(method string, params interface{}) (*JsonRequest, error)

type JsonResponse 

type JsonResponse struct {
	Id    int
	Error interface{}
	// RawMessage allows us to store late-resolved, message-specific nested types
	// requires an extra couple of steps though; even though RawMessage is a []byte, it's not
	// JSON itself. You need to convert JSON to/from RawMessage as well as JSON to/from the structure
	// - see RawMessage's own UnmarshalJSON/MarshalJSON for this extra step
	Result *json.RawMessage
}

func NewJsonErrorResponse 

func NewJsonErrorResponse(id int, err interface{}) *JsonResponse

func NewJsonResponse 

func NewJsonResponse(id int, result interface{}) (*JsonResponse, error)

type LOBExistsRequest 

type LOBExistsRequest struct {
	LobSHA string
}

type LOBExistsResponse 

type LOBExistsResponse struct {
	Exists bool
	Size   int64
}

type PersistentTransport 

type PersistentTransport struct {
	// The persistent connection we're using (implemented by another class)
	Connection io.ReadWriteCloser
	// Buffered reader we use to scan for ends of JSON
	BufferedReader *bufio.Reader
}

Transport implementation that uses a persistent connection to perform many operations in serial, without having to initiate a connection each time Most common use of this is SSH, althouth the underlying connection streams are abstracted to allow other connections if required.

func NewPersistentTransport 

func NewPersistentTransport(conn io.ReadWriteCloser) *PersistentTransport

Create a new persistent transport & connect

func (*PersistentTransport) ChunkExists 

func (self *PersistentTransport) ChunkExists(lobsha string, chunk int) (bool, int64, error)

Return whether LOB chunk content exists on the server

func (*PersistentTransport) ChunkExistsAndIsOfSize 

func (self *PersistentTransport) ChunkExistsAndIsOfSize(lobsha string, chunk int, sz int64) (bool, error)

Return whether LOB chunk content exists on the server, and is of a specific size

func (*PersistentTransport) DownloadChunk 

func (self *PersistentTransport) DownloadChunk(lobsha string, chunk int, out io.Writer, callback TransportProgressCallback) error

Download chunk content for a LOB (from a stream); must call back progress This is a non-delta download operation, just provide entire chunk content

func (*PersistentTransport) DownloadDelta 

func (self *PersistentTransport) DownloadDelta(baseSHA, targetSHA string, sizeLimit int64, out io.Writer, callback TransportProgressCallback) (bool, error)

Generate and download a binary delta that the client can apply locally to generate a new LOB Deltas apply to whole LOB content and are not per-chunk The server should respect sizeLimit and if the delta is larger than that, abandon the process Return a bool to indicate whether the delta went ahead or not (client will fall back to non-delta on false)

func (*PersistentTransport) DownloadDeltaPrepare 

func (self *PersistentTransport) DownloadDeltaPrepare(baseSHA, targetSHA string) (int64, error)

Prepare a binary delta between 2 LOBs and report the size

func (*PersistentTransport) DownloadMetadata 

func (self *PersistentTransport) DownloadMetadata(lobsha string, out io.Writer) error

Download metadata for a LOB (to a stream); no progress callback as very small

func (*PersistentTransport) GetFirstCompleteLOBFromList 

func (self *PersistentTransport) GetFirstCompleteLOBFromList(candidateSHAs []string) (string, error)

Return the LOB which the server has a complete copy of, from a list of candidates Server must test in the order provided & return the earliest one which is complete on the server Server doesn't have to test full integrity of LOB, just completeness (check size against meta) Return a blank string if none are available

func (*PersistentTransport) LOBExists 

func (self *PersistentTransport) LOBExists(lobsha string) (bool, int64, error)

Return whether LOB exists in entirety on the server

func (*PersistentTransport) MetadataExists 

func (self *PersistentTransport) MetadataExists(lobsha string) (bool, int64, error)

Return whether LOB metadata exists on the server

func (*PersistentTransport) QueryCaps 

func (self *PersistentTransport) QueryCaps() ([]string, error)

Ask the server for a list of capabilities

func (*PersistentTransport) Release 

func (self *PersistentTransport) Release()

Release any resources associated with this transport (including any persostent connections)

func (*PersistentTransport) SetEnabledCaps 

func (self *PersistentTransport) SetEnabledCaps(caps []string) error

Request that the server enable capabilities for this exchange (note, non-persistent transports can store & send this with every request)

func (*PersistentTransport) UploadChunk 

func (self *PersistentTransport) UploadChunk(lobsha string, chunk int, sz int64, data io.Reader, callback TransportProgressCallback) error

Upload chunk content for a LOB (from a stream); must call back progress

func (*PersistentTransport) UploadDelta 

func (self *PersistentTransport) UploadDelta(baseSHA, targetSHA string, deltaSize int64, data io.Reader, callback TransportProgressCallback) (bool, error)

Upload a binary delta to apply against a LOB the server already has, to generate a new LOB Deltas apply to whole LOB content and are not per-chunk Returns a boolean to determine whether the upload was accepted or not (server may prefer not to accept, not an error) In the case of false return, client will fall back to non-delta upload. On true, server must return nil error only after data is fully received, applied, saved as targetSHA and the integrity confirmed by recalculating the SHA of the final patched data. This only does the content, not the metadata; you should have already called UploadMetadata beforehand.

func (*PersistentTransport) UploadMetadata 

func (self *PersistentTransport) UploadMetadata(lobsha string, sz int64, data io.Reader) error

Upload metadata for a LOB (from a stream); no progress callback as very small

type QueryCapsRequest 

type QueryCapsRequest struct {
}

type QueryCapsResponse 

type QueryCapsResponse struct {
	Caps []string
}

type SetEnabledCapsRequest 

type SetEnabledCapsRequest struct {
	EnableCaps []string
}

type SetEnabledCapsResponse 

type SetEnabledCapsResponse struct {
}

type SmartSyncProviderImpl 

type SmartSyncProviderImpl struct {
	// contains filtered or unexported fields
}

The smart sync provider implements everything the standard SyncProvider does, but in addition provides methods to exchange binary deltas rather than entire files (as chunks). It can operate in 2 modes; 'persistent' mode where a connection is re-used for many requests (only possible with options like SSH), or 'transient' mode where all requests & responses are separate round-trips (e.g. REST). The Transport interface provides the abstraction required for that.

func (*SmartSyncProviderImpl) Download 

func (self *SmartSyncProviderImpl) Download(remoteName string, filenames []string, toDir string,
	force bool, callback providers.SyncProgressCallback) error

This is the file-based download (i.e. a meta or a chunk) so no deltas here Client will use delta alts if it wants

func (*SmartSyncProviderImpl) DownloadDelta 

func (self *SmartSyncProviderImpl) DownloadDelta(remoteName, basesha, targetsha string, out io.Writer, callback providers.SyncProgressCallback) error

Download delta of LOB content (must be applied later)

func (*SmartSyncProviderImpl) FileExists 

func (self *SmartSyncProviderImpl) FileExists(remoteName, filename string) bool

func (*SmartSyncProviderImpl) FileExistsAndIsOfSize 

func (self *SmartSyncProviderImpl) FileExistsAndIsOfSize(remoteName, filename string, sz int64) bool

func (*SmartSyncProviderImpl) GetFirstCompleteLOBFromList 

func (self *SmartSyncProviderImpl) GetFirstCompleteLOBFromList(remoteName string, candidateSHAs []string) (string, error)

func (*SmartSyncProviderImpl) HelpTextDetail 

func (*SmartSyncProviderImpl) HelpTextDetail() string

func (*SmartSyncProviderImpl) HelpTextSummary 

func (*SmartSyncProviderImpl) HelpTextSummary() string

func (*SmartSyncProviderImpl) LOBExists 

func (self *SmartSyncProviderImpl) LOBExists(remoteName, sha string) (ex bool, sz int64)

Whether a LOB exists in full on the remote, and gets its size

func (*SmartSyncProviderImpl) PrepareDeltaForDownload 

func (self *SmartSyncProviderImpl) PrepareDeltaForDownload(remoteName, sha string, candidateBaseSHAs []string) (size int64, base string, e error)

func (*SmartSyncProviderImpl) Release 

func (self *SmartSyncProviderImpl) Release()

func (*SmartSyncProviderImpl) TypeID 

func (*SmartSyncProviderImpl) TypeID() string

func (*SmartSyncProviderImpl) Upload 

func (self *SmartSyncProviderImpl) Upload(remoteName string, filenames []string, fromDir string,
	force bool, callback providers.SyncProgressCallback) error

This is the file-based upload (i.e. a meta or a chunk) so no deltas here Client will use delta alts if it wants

func (*SmartSyncProviderImpl) UploadDelta 

func (self *SmartSyncProviderImpl) UploadDelta(remoteName, basesha, targetsha string, in io.Reader, size int64, callback providers.SyncProgressCallback) error

Upload delta of LOB content (must be calculated first)

func (*SmartSyncProviderImpl) ValidateConfig 

func (self *SmartSyncProviderImpl) ValidateConfig(remoteName string) error

type SshConnection 

type SshConnection struct {
	// contains filtered or unexported fields
}

Underlying SSH connection to smart server, for use with PersistentTransport Works by invoking ssh/plink/tortoise_plink and connecting stdout/stdin

func (*SshConnection) Close 

func (self *SshConnection) Close() error

func (*SshConnection) Read 

func (self *SshConnection) Read(p []byte) (n int, err error)

SSH Connection implementation

func (*SshConnection) Write 

func (self *SshConnection) Write(p []byte) (n int, err error)

type SshTransportFactory 

type SshTransportFactory struct {
}

factory for creating SSH connections

func (*SshTransportFactory) Connect 

func (self *SshTransportFactory) Connect(u *url.URL) (Transport, error)

func (*SshTransportFactory) WillHandleUrl 

func (self *SshTransportFactory) WillHandleUrl(u *url.URL) bool

type Transport 

type Transport interface {
	// Release any resources associated with this transport (including closing any persistent connections)
	Release()
	// Ask the server for a list of capabilities
	QueryCaps() ([]string, error)
	// Request that the server enable capabilities for this exchange (note, non-persistent transports can store & send this with every request)
	SetEnabledCaps(caps []string) error

	// Return whether LOB metadata exists on the server (also returns size)
	MetadataExists(lobsha string) (ex bool, sz int64, e error)
	// Return whether LOB chunk content exists on the server
	ChunkExists(lobsha string, chunk int) (ex bool, sz int64, e error)
	// Return whether LOB chunk content exists on the server, and is of a specific size
	ChunkExistsAndIsOfSize(lobsha string, chunk int, sz int64) (bool, error)
	// Entire LOB exists? Also returns entire content size
	LOBExists(lobsha string) (ex bool, sz int64, e error)

	// Upload metadata for a LOB (from a stream); no progress callback as very small
	UploadMetadata(lobsha string, sz int64, data io.Reader) error
	// Upload chunk content for a LOB (from a stream); must call back progress
	UploadChunk(lobsha string, chunk int, sz int64, data io.Reader, callback TransportProgressCallback) error
	// Download metadata for a LOB (to a stream); no progress callback as very small
	DownloadMetadata(lobsha string, out io.Writer) error
	// Download chunk content for a LOB (from a stream); must call back progress
	// This is a non-delta download operation, just provide entire chunk content
	DownloadChunk(lobsha string, chunk int, out io.Writer, callback TransportProgressCallback) error

	// Return the LOB which the server has a complete copy of, from a list of candidates
	// Server must test in the order provided & return the earliest one which is complete on the server
	// Server doesn't have to test full integrity of LOB, just completeness (check size against meta)
	// Return a blank string if none are available
	GetFirstCompleteLOBFromList(candidateSHAs []string) (string, error)
	// Upload a binary delta to apply against a LOB the server already has, to generate a new LOB
	// Deltas apply to whole LOB content and are not per-chunk
	// Returns a boolean to determine whether the upload was accepted or not (server may prefer not to accept, not an error)
	// In the case of false return, client will fall back to non-delta upload.
	// On true, server must return nil error only after data is fully received, applied, saved as targetSHA and the
	// integrity confirmed by recalculating the SHA of the final patched data.
	UploadDelta(baseSHA, targetSHA string, deltaSize int64, data io.Reader, callback TransportProgressCallback) (bool, error)
	// Prepare a binary delta between 2 LOBs and report the size
	DownloadDeltaPrepare(baseSHA, targetSHA string) (int64, error)
	// Generate (if not already cached) and download a binary delta that the client can apply locally to generate a new LOB
	// Deltas apply to whole LOB content and are not per-chunk
	// The server should respect sizeLimit and if the delta is larger than that, abandon the process
	// Return a bool to indicate whether the delta went ahead or not (client will fall back to non-delta on false)
	DownloadDelta(baseSHA, targetSHA string, sizeLimit int64, out io.Writer, callback TransportProgressCallback) (bool, error)
}

The transport interface abstracts away how the smart provider talks to the server It might do this over a persistent SSH connection, sending data across in/out streams, or it might process each request as a discrete request/response pair over REST Note each transport instance is stateful and associated with a server/connection, see SmartTransportFactory for how they are created

type TransportFactory 

type TransportFactory interface {
	// Does this factory want to handle the URL passed in?
	WillHandleUrl(u *url.URL) bool
	// Provide a new, connected (may not be persistent, but if not test connection/auth) transport for given URL
	Connect(u *url.URL) (Transport, error)
}

Interface for a factory which creates persistent transports for use by SmartSyncProvider

func GetTransportFactory 

func GetTransportFactory(u *url.URL) TransportFactory

Retrieve the best ConnectionFactory for a given URL (or nil)

type TransportProgressCallback 

type TransportProgressCallback func(bytesDone, totalBytes int64)

type UploadDeltaCompleteResponse 

type UploadDeltaCompleteResponse struct {
	ReceivedOK bool
}

type UploadDeltaRequest 

type UploadDeltaRequest struct {
	BaseLobSHA   string
	TargetLobSHA string
	Size         int64
}

type UploadDeltaStartResponse 

type UploadDeltaStartResponse struct {
	OKToSend bool
}

type UploadFileCompleteResponse 

type UploadFileCompleteResponse struct {
	ReceivedOK bool
}

type UploadFileRequest 

type UploadFileRequest struct {
	LobSHA   string
	Type     string
	ChunkIdx int
	Size     int64
}

type UploadFileStartResponse 

type UploadFileStartResponse struct {
	OKToSend bool
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.