scp

README

Secure Copy Protocol implemented in Go

Overview

Production-ready Secure Copy Protocol (aka: SCP) implemented in Go with well documentation and neat dependency.

Introduction

Secure Copy Protocol uses Secure Shell (SSH) to transfer files between host on a network.

There is no RFC that defines the specifics of the protocol. This package simply implements SCP against the OpenSSH's scp tool, thus you can directly transfer files to/from *uinx system within your Go code, as long as the remote host has OpenSSH installed.

Features

• Copy file from local to remote.
• Copy file from remote to local.
• Copy from buffer to remote file. (e.g: copy from bytes.Reader)
• Copy from remote file to buffer. (e.g: copy to os.Stdout)
• Recursively copy directory from local to remote.
• Recursively copy directory from remote to local.
• Set permission bits for transferred files.
• Set timeout/context for transfer.
• Preserve the permission bits and modification time at transfer.
• No resources leak. (e.g: goroutine, file descriptor)
• Low memory consuming for transferring huge files.
• TODO:
  • Transfer speed limit.
  • Performance benchmark/optimization for lots of small files.
• Won't support:
  • Copy file from remote to remote.

Install

go get github.com/povsister/scp

Example usage

This package leverages golang.org/x/crypto/ssh to establish a SSH connection to remote host.

Error handling are omitted in examples!

Copy a file to remote

// Build a SSH config from username/password
sshConf := scp.NewSSHConfigFromPassword("username", "password")

// Build a SSH config from private key
privPEM, err := ioutil.ReadFile("/path/to/privateKey")
// without passphrase
sshConf, err := scp.NewSSHConfigFromPrivateKey("username", privPEM)
// with passphrase
sshConf, err := scp.NewSSHConfigFromPrivateKey("username", privPEM, passphrase)

// Dial SSH to "my.server.com:22".
// If your SSH server does not listen on 22, simply suffix the address with port.
// e.g: "my.server.com:1234"
scpClient, err := scp.NewClient("my.server.com", sshConf, &scp.ClientOption{})

// Build a SCP client based on existing "golang.org/x/crypto/ssh.Client"
scpClient, err := scp.NewClientFromExistingSSH(existingSSHClient, &scp.ClientOption{})

defer scpClient.Close()


// Do the file transfer without timeout/context
err = scpClient.CopyFileToRemote("/path/to/local/file", "/path/at/remote", &scp.FileTransferOption{})

// Do the file copy with timeout, context and file properties preserved.
// Note that the context and timeout will both take effect.
fo := &scp.FileTransferOption{
    Context: yourCotext,
    Timeout: 30 * time.Second, 
    PreserveProp: true,
}
err = scpClient.CopyFileToRemote("/path/to/local/file", "/path/at/remote", fo)

Copy a file from remote

// Copy the file from remote and save it as "/path/to/local/file".
err = scpClient.CopyFileFromRemote("/path/to/remote/file", "/path/to/local/file", &scp.FileTransferOption{})

// Copy the remote file and print it in Stdout.
err = scpClient.CopyFromRemote("/path/to/remote/file", os.Stdout, &scp.FileTransferOption{})

Copy from buffer to remote as a file

// From buffer
buffer := []byte("something excited")
reader := bytes.NewReader(buffer)

// From fd
// Note that its YOUR responsibility to CLOSE the fd after transfer.
reader, err := os.Open("/path/to/local/file")
defer reader.Close()


// Note that the reader must implement "KnownSize" interface except os.File
// For the content length must be provided before transfer.
// The last part of remote location will be used as file name at remote.
err := scpClient.CopyToRemote(reader, "/path/to/remote/file", &scp.FileTransferOption{})

Recursively copy a directory to remote

// recursively copy to remote
err := scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", &scp.DirTransferOption{})

// recursively copy to remote with timeout, context and file properties.
// Note that the context and timeout will both take effect.
do := &scp.DirTransferOption{
    Context: yourContext,
    Timeout: 10 * time.Minute,
    PreserveProp: true,
}
err:= scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", do)

Recursively copy a directory from remote

// recursively copy from remote.
// The content of remote dir will be save under "/path/to/local".
err := scpClient.CopyDirFromRemote("/path/to/remote/dir", "/path/to/local", &scp.DirTransferOption{})

Something you need to know

SCP is a light-weighted protocol which implements file transfer only. It does not support advanced features like: directory listing, resume from break-point.

So, it's commonly used for transferring some small-size, temporary files. If you heavily depend on the file transfer, you may consider using SFTP instead.

Another thing you may notice is that I didn't put context.Context as the first argument in function signature. Instead, it's located in TransferOption. This is intentional because it makes the API also light-weighted.

License

MIT License

Documentation

Overview

Production-ready Secure Copy Protocol (aka: SCP) implemented in Go with well documentation and neat dependency.

Index

• Variables
• func NewSSHConfigFromPassword(username, password string) *ssh.ClientConfig
• func NewSSHConfigFromPrivateKey(username string, privPEM []byte, passphrase ...string) (cfg *ssh.ClientConfig, err error)
• type Client
  • func NewClient(serverAddr string, sshCfg *ssh.ClientConfig, scpOpt *ClientOption) (*Client, error)
  • func NewClientFromExistingSSH(existing *ssh.Client, scpOpt *ClientOption) (*Client, error)
  • func (c *Client) CopyDirFromRemote(remoteDir, localDir string, opt *DirTransferOption) error
  • func (c *Client) CopyDirToRemote(localDir string, remoteDir string, opt *DirTransferOption) error
  • func (c *Client) CopyFileFromRemote(remoteFile, localFile string, opt *FileTransferOption) error
  • func (c *Client) CopyFileToRemote(localFile string, remoteLoc string, opt *FileTransferOption) error
  • func (c *Client) CopyFromRemote(remoteFile string, buffer io.Writer, opt *FileTransferOption) error
  • func (c *Client) CopyToRemote(reader io.Reader, remoteTarget string, opt *FileTransferOption) error
• type ClientOption
• type DirTransferOption
• type FileTransferOption
• type KnownSize

Variables

var (
	// DefaultFilePerm holds default permission bits for transferred files.
	DefaultFilePerm = os.FileMode(0644)
	// DefaultDirPerm holds default permission bits for transferred directories.
	DefaultDirPerm = os.FileMode(0755)

	// ErrNoTransferOption indicate a non-nil TransferOption should be provided.
	ErrNoTransferOption = errors.New("scp: TransferOption is not provided")

	// DirectoryPreReads sets the num of pre-read files/directories for recursively transferring a directory.
	// Set it larger may speedup the transfer with lots of small files.
	// Do not set it too large or you will exceed the max open files limit.
	DirectoryPreReads = 10
)

var (
	// DebugMode controls the debug output.
	// If true, the debug information of remote scp server
	// will be printed in Stderr.
	DebugMode = false
)

var (
	// ErrNoClientOption indicates a non-nil ClientOption should be provided
	ErrNoClientOption = errors.New("scp: ClientOption is not provided")
)

Functions

func NewSSHConfigFromPassword

func NewSSHConfigFromPassword(username, password string) *ssh.ClientConfig

NewSSHConfigFromPassword returns a *ssh.ClientConfig with ssh.Password AuthMethod and 3 seconds timeout for connecting the server.

It *insecurely* ignores server's host key validation.

func NewSSHConfigFromPrivateKey

func NewSSHConfigFromPrivateKey(username string, privPEM []byte, passphrase ...string) (cfg *ssh.ClientConfig, err error)

NewSSHConfigFromPrivateKey returns a *ssh.ClientConfig with ssh.PublicKey AuthMethod and 3 seconds timeout for connecting the server.

The passphrase is optional. If multiple passphrase are provided, only the first will be used.

If the private key is encrypted, it will return a ssh.PassphraseMissingError.

It *insecurely* ignores server's host key validation.

Types

type Client

type Client struct {
	// The underlying ssh client
	*ssh.Client
	// contains filtered or unexported fields
}

Client has the "golang.org/x/crypto/ssh/Client" embedded, so it can be used as normal SSH client with additional SCP features.

func NewClient

func NewClient(serverAddr string, sshCfg *ssh.ClientConfig, scpOpt *ClientOption) (*Client, error)

NewClient returns a SSH client with SCP capability.

The serverAddr should be in "host" or "host:port" format. If no port is supplied, the default port 22 will be used.

IPv6 serverAddr must be enclosed in square brackets, as in "[::1]" or "[::1]:22"

func NewClientFromExistingSSH

func NewClientFromExistingSSH(existing *ssh.Client, scpOpt *ClientOption) (*Client, error)

NewClientFromExistingSSH returns a SSH client with SCP capability. It reuse the existing SSH connection without dialing

func (*Client) CopyDirFromRemote

func (c *Client) CopyDirFromRemote(remoteDir, localDir string, opt *DirTransferOption) error

CopyDirFromRemote recursively copies a remote directory into local directory. The localDir must exist before copying.

For example:

• CopyDirFromRemote("/remote/dir1", "/local/dir2", &DirTransferOption{})
• Results: "remote/dir1/<contents>" -> "/local/dir2/<contents>"

func (*Client) CopyDirToRemote

func (c *Client) CopyDirToRemote(localDir string, remoteDir string, opt *DirTransferOption) error

CopyDirToRemote recursively copies a directory to remoteDir.

func (*Client) CopyFileFromRemote

func (c *Client) CopyFileFromRemote(remoteFile, localFile string, opt *FileTransferOption) error

CopyFileFromRemote copies a remoteFile as localFile.

If localFile does not exist, it will be automatically created. If localFile already exists, it will be truncated for writing. If localFile is a directory, the name of remoteFile will be used.

For example:

• CopyFileFromRemote("/remote/file1", "/local/fileNotExist", &FileTransferOption)

• Result: "/remote/file1" -> "/local/file2" The "fileNotExist" will be created.

• CopyFileFromRemote("/remote/file1", "/local/fileExist", &FileTransferOption)

• Result: "/remote/file1" -> "/local/fileExist" The "fileExist" will be truncated for writing.

• CopyFileFromRemote("/remote/file1", "/local/dir", &FileTransferOption)

• Result: "/remote/file1" -> "/local/dir/file1" The "file1" will be used as filename and stored under "/local/dir" directory. Note that "/local/dir" must exist in this case.

func (*Client) CopyFileToRemote

func (c *Client) CopyFileToRemote(localFile string, remoteLoc string, opt *FileTransferOption) error

CopyFileToRemote copies a local file to remote location. It will automatically close the file after read.

func (*Client) CopyFromRemote

func (c *Client) CopyFromRemote(remoteFile string, buffer io.Writer, opt *FileTransferOption) error

CopyFromRemote copies a remote file into buffer.

Note that "PreserveProp" and "Perm" option does not take effect in this case.

func (*Client) CopyToRemote

func (c *Client) CopyToRemote(reader io.Reader, remoteTarget string, opt *FileTransferOption) error

CopyToRemote copies content from reader to remoteTarget. The reader must implement "KnownSize" interface except *os.File.

Currently, it supports following readers:

• *os.File
• *strings.Reader
• *bytes.Reader

Note that the last part of remoteTarget will be used as filename if unspecified.

It's CALLER'S responsibility to CLOSE the file if an *os.File is supplied.

type ClientOption

type ClientOption struct {
	// Use sudo to run remote scp server.
	// Default: false.
	Sudo bool
	// The scp remote server executable file.
	// Default: "scp".
	//
	// If your scp command is not in the default path,
	// specify it as "/path/to/scp".
	RemoteBinary string
}

ClientOption contains several configurable options for SCP client.

type DirTransferOption

type DirTransferOption struct {
	// Context for the directory transfer.
	// Can be both set with Timeout.
	// Default: no context
	Context context.Context
	// Timeout for transferring the whole directory.
	// Can be both set with Context.
	// Default: 0 (means no timeout)
	Timeout time.Duration
	// Preserve modification times and modes from the original file/directory.
	// Default: false
	PreserveProp bool
	// Limits the used bandwidth, specified in Kbit/s.
	// Default: 0 (Means no limit)
	// TODO: not implemented yet
	SpeedLimit int64
}

DirTransferOption holds the transfer options for directory.

type FileTransferOption

type FileTransferOption struct {
	// Context for the file transfer.
	// Can be both set with Timeout.
	// Default: no context
	Context context.Context
	// Timeout for transferring the file.
	// Can be both set with Context.
	// Default: 0 (Means no timeout)
	Timeout time.Duration
	// The permission bits for transferred file.
	// Override "PreserveProp" if specified.
	// Default: 0644
	Perm os.FileMode
	// Preserve modification times and permission bits from the original file.
	// Only valid for file transfer.
	// Default: false
	PreserveProp bool
	// Limits the used bandwidth, specified in Kbit/s.
	// Default: 0 (Means no limit)
	// TODO: not implemented yet
	SpeedLimit int64
}

FileTransferOption holds the transfer options for file.

type KnownSize

type KnownSize interface {
	// return num in bytes
	Size() int64
}

KnownSize is intended for reader whose size is already known before reading.