sftpsync

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 17, 2026 License: MIT Imports: 13 Imported by: 0

README

go-sftpsync

Status: pre-1.0 (v0.1.0), in development. The safety properties are tested but the library has not yet been used in anger. Expect API churn in minor versions; breaking changes are called out loudly in CHANGELOG.md. Bug reports welcome — issues are the fastest way to get something fixed.

Recursive directory transfer over an SFTP connection you already have. go-sftpsync walks a directory tree in either direction — local to remote or remote to local — carrying permission bits, refusing symlinks that point outside the sync root, writing every file to a temp name and renaming it into place, honouring context.Context between files and inside one, and returning a report of what happened rather than a bare error. It takes a live *sftp.Client or *ssh.Client and never dials, authenticates, or reads ~/.ssh/config: the connection is the caller's, and so are the decisions that made it trustworthy.

Documentation: pkg.go.dev/github.com/hollis-labs/go-sftpsync.

Install

go get github.com/hollis-labs/go-sftpsync

Usage

package main

import (
	"context"
	"fmt"
	"log"

	sftpsync "github.com/hollis-labs/go-sftpsync"
	"golang.org/x/crypto/ssh"
)

func deploy(ctx context.Context, client *ssh.Client) error {
	// Preview first. The dry run walks exactly what the transfer walks, so
	// this is the file set, not a second opinion about it.
	plan, err := sftpsync.UploadOverSSH(ctx, client, "./build", "/srv/app",
		sftpsync.WithDryRun(true))
	if err != nil {
		return err
	}
	fmt.Println(plan) // upload ./build -> /srv/app (dry run): 12 files, 4 dirs, 84213 bytes in 31ms
	for _, e := range plan.Entries {
		fmt.Printf("  %-8s %s\n", e.Action, e.Path)
	}

	result, err := sftpsync.UploadOverSSH(ctx, client, "./build", "/srv/app")
	if err != nil {
		// Errors name the side and the path: "sftpsync: remote create
		// /srv/app/.bin.3f2a.sftpsync-tmp: permission denied"
		return err
	}
	log.Println(result)
	return nil
}

For a version that runs with no host at all, against an in-process SFTP server:

go run ./examples/preview

examples/overssh is the same thing against a real host, and shows the caller doing its own dialling, key loading and known_hosts verification.

API

Symbol Purpose
Upload(ctx, *sftp.Client, localDir, remoteDir, ...Option) (*Result, error) Copy a local tree to the far end.
Download(ctx, *sftp.Client, remoteDir, localDir, ...Option) (*Result, error) The same walk, reversed.
UploadOverSSH(ctx, *ssh.Client, ...) / DownloadOverSSH(ctx, *ssh.Client, ...) Open an SFTP subsystem on a connection you own, transfer, close the subsystem. The *ssh.Client is left open.
WithPreserveMode(bool) Carry source permission bits. Default true.
WithPreserveModTime(bool) Carry source modification times. Default false.
WithSymlinkPolicy(SymlinkPolicy) SymlinkReplicate (default), SymlinkSkip, SymlinkDereferenceIncludingOutsideRoot.
WithDryRun(bool) Walk and report, write nothing.
WithFileMode(os.FileMode) / WithDirMode(os.FileMode) Modes used when WithPreserveMode is off. Default 0644 / 0755.
WithMaxDepth(int) Bound the descent. Default 64. Exceeding it is an error, never a silent truncation.
Result Direction, Source, Destination, DryRun, Entries, Files, Dirs, Symlinks, Skipped, Bytes, Duration, and a String() one-liner for operators.
Entry One thing the walk found: Path (relative, slash-separated), Action, Mode, Size, Target, Reason.
PathError Every error. Carries Op, Side (Local / Remote), Path, and unwraps to the cause.
ErrSymlinkEscape, ErrNotDirectory, ErrMaxDepthExceeded Sentinels for conditions a caller acts on.

The opinions

A general-purpose "copy a directory" helper usually lacks these. They are what makes it safe to point at a real host — and they are the contract, not incidental detail: changing any of them is a breaking change even where the function signature does not move.

Permission bits are carried across. An uploaded script that arrives 0644 does not fail at upload time; it fails an hour later, far from the transfer that caused it. Directory modes are applied after the directory's contents are written, so a source directory of 0500 arrives as 0500 instead of refusing its own files — a refinement over the SCP prior art, which has to send a directory's mode at creation because the protocol requires it.

Symlinks that leave the sync root are refused. A link inside the tree pointing at /etc/shadow does not exfiltrate it. The check is lexical, so it costs no I/O, cannot be raced by a target that changes after the check, and gives the same answer on the remote side where there is no EvalSymlinks to call. It is sound because nothing is ever read through a replicated link. Refusal is ErrSymlinkEscape, wrapped in a PathError that names the link and its target. Following links outside the root is possible, and the option is named SymlinkDereferenceIncludingOutsideRoot so that choosing it reads as a decision.

An absolute target counts as an escape even when it points inside the source root: the same absolute path on the other machine is a different place, and this library will not guess which one you meant.

Every file is written to a sibling temp name and renamed into place. An interrupted sync leaves the previous file intact rather than half of the new one. Mode and mtime are applied to the temp file before the rename, so the file arrives with its final attributes in the same instant it arrives with its contents — never complete but not yet executable. Where the server offers posix-rename@openssh.com the replacement is atomic; where it does not, the destination is removed first, which is the one window this technique cannot close.

Cancellation is honoured between files and inside one. io.Copy has no context, so a sync of ten thousand small files over a dropped VPN would otherwise run to completion long after the caller gave up. The reader is wrapped rather than the copy loop hand-rolled, which keeps pkg/sftp's pipelined writes — most of its throughput over a real link — while still checking on every chunk.

Transfers return a report. *Result says what moved, what was skipped and why, how many bytes and how long. It is returned even when the transfer fails, so a caller can tell an operator the difference between "nothing moved" and "half the tree moved".

Dry run first. WithDryRun(true) walks and reports without creating, writing or renaming anything. It is an option on the transfer functions rather than a separate Plan call on purpose: a preview is only worth showing an operator if it came from the same walk as the transfer.

Upload and download are one implementation. The two directions differ only in which end is the source. That is deliberate: the download path is where "it is only reading" quietly becomes "it wrote outside the destination", and one code path cannot drift into two sets of safety properties.

Limits

There is no delta transfer. Every byte is copied every time. There is no rolling checksum, no size-and-mtime skip, and no deletion of files that exist only at the destination. That is the right trade for config files, env files, compose stacks and agent definitions, and the wrong one for a 600MB image that changed in one place. If you are moving something large and mostly unchanged, use rsync. Should a tar-over-exec strategy ever land here it will be a second strategy, not a replacement for this one.

Also out of scope, and deliberately: compression, resume, bandwidth limiting, and anything that requires a binary on the remote host.

Every entry is lstated. SFTP's SSH_FXP_READDIR carries attributes, but which stat they come from is server-defined, and a server that reports a symlink's target instead of the link would let a link present itself as a regular file and walk straight past the escape check. One extra round trip per entry is the price of that check meaning anything. For a tree of thousands of tiny files over a high-latency link, this is the dominant cost.

Ownership is not carried. No chown, no uid/gid mapping. The far end's files belong to whoever the SFTP connection authenticated as.

Alternatives

A reader deciding between libraries is better served by an honest list than by a claim that this one is best.

Library When to prefer it
github.com/pkg/sftp Whenever it is enough. It is the foundation this package is built on, and its Walk, ReadDir, MkdirAll and Chtimes will serve you directly if you are moving a handful of files, do not need recursion, or want to make the safety decisions yourself. Depending on this package only pays for itself if you want these particular opinions.
github.com/bramvdbogaerde/go-scp A maintained, tagged, single-file SCP client with a good PassThru progress hook and a NewClientBySSH constructor that takes an existing connection. Prefer it if you need progress reporting on one large file and are content with SCP — which OpenSSH has deprecated and now implements over SFTP anyway. It does not do recursion.
github.com/povsister/scp The most complete recursive SCP implementation in Go, with careful directory traversal ordering and mode handling worth reading. Prefer it if you must speak SCP specifically. Its caveats are that it has no tagged release (v0.0.0 pseudo-version only) and that it, too, is SCP.
github.com/melbahja/goph A convenience wrapper over golang.org/x/crypto/ssh covering connection, commands and single-file transfer. Prefer it if you want one package to handle dialling as well — which is exactly the thing this package refuses to do. It does not solve recursion.
rsync over ssh.Session Prefer it for anything large, anything mostly unchanged, anything needing --delete, and anything where resuming matters. It requires rsync on both ends; this package requires only an SFTP subsystem.

Prior art

povsister/scp and bramvdbogaerde/go-scp were both read before this was written, and where an approach is borrowed from either, the comment at that line says so.

The clearest debts are povsister/scp's directory-creation ordering and mode handling — reading how it has to send a directory's mode at creation is what made applying the mode after contents the obvious improvement, and the comment in enterDir says as much — and pkg/sftp's own delayed-writer test trick, which is what lets the cancellation tests interrupt a transfer at a known point instead of sleeping and hoping.

Dependencies

Direct:

That is the whole list, and it is meant to stay that way.

Testing

go test -race ./...

Tests run against a real SFTP server in-process — the one that ships with pkg/sftp, wired to a client over a net.Pipe — rather than a mock, so they exercise SSH_FXP_LSTAT, SSH_FXP_SYMLINK, SSH_FXP_SETSTAT and posix-rename@openssh.com as the protocol actually implements them. No network, no ssh daemon, no external sftp-server binary, no environment variables and no fixtures.

License

MIT © Hollis Labs.

Documentation

Overview

Package sftpsync recursively transfers a directory tree over an SFTP connection that the caller already owns.

Status

Pre-1.0 (v0.1.0), in development. The safety properties are tested but the library has not yet been used in anger. Expect API churn in minor versions; breaking changes are called out loudly in CHANGELOG.md. Bug reports welcome — https://github.com/hollis-labs/go-sftpsync/issues is the fastest way to get something fixed.

It exists because github.com/pkg/sftp gives you the primitives — Walk, ReadDir, MkdirAll, Chtimes — and leaves recursion, safety and options to you, and because every recursive alternative speaks the SCP protocol that OpenSSH has deprecated. The value here is not the wrapping; it is the opinions:

  • Permission bits are carried across, so an uploaded script arrives executable.
  • A symlink whose target resolves outside the sync root is refused, not followed. See ErrSymlinkEscape.
  • Every file is written to a sibling temp name and renamed into place, so an interrupted transfer never replaces a good file with half of a new one.
  • Cancellation is honoured between files and inside one. An io.Copy over a dropped VPN stops when the context says so, not when TCP notices.
  • The return value is a Result — what moved, what was skipped, how many bytes, how long — because callers show this to operators.
  • WithDryRun walks and reports without writing, so a caller can preview an operation without reimplementing the walk.

Connections

This package takes a connection; it does not make one. Upload and Download accept a live *sftp.Client, UploadOverSSH and DownloadOverSSH a live *ssh.Client. Nothing here reads ~/.ssh/config, accepts credentials, or dials. Authentication, host-key policy and connection lifetime belong to the caller, who already has opinions about all three; a library that dialled again would be a second transport with different failure modes from the one in use.

No delta transfer

Every byte is copied every time. There is no rolling checksum, no size-and-mtime skip, and no deletion of files that exist only at the destination. That is the right trade for config files, env files, compose stacks and agent definitions, and the wrong one for a 600MB image. If you are moving something large and mostly unchanged, use rsync.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrSymlinkEscape reports a symlink whose target resolves outside the
	// sync root. It is a sentinel rather than a plain error because a caller
	// acts on it differently from an I/O failure: an escape is a property of
	// the tree that retrying will not fix, and is worth naming to an operator.
	//
	// Wrapped in a [*PathError] that names the link, so the offending path is
	// recoverable as well as the condition.
	ErrSymlinkEscape = errors.New("symlink target escapes the sync root")

	// ErrNotDirectory reports that a sync root is not a directory. This
	// package transfers trees; for a single file, pkg/sftp is enough on its own.
	ErrNotDirectory = errors.New("not a directory")

	// ErrMaxDepthExceeded reports that the walk descended past the configured
	// limit (see [WithMaxDepth]). Under [SymlinkDereferenceIncludingOutsideRoot]
	// this is also how a symlink loop terminates.
	ErrMaxDepthExceeded = errors.New("maximum directory depth exceeded")
)

Functions

This section is empty.

Types

type Action 

type Action string

Action is what happened to one entry in the tree. 

const (
	// ActionFile is a regular file whose contents were transferred.
	ActionFile Action = "file"
	// ActionDir is a directory that exists at the destination and was walked.
	ActionDir Action = "dir"
	// ActionSymlink is a symlink recreated at the destination.
	ActionSymlink Action = "symlink"
	// ActionSkip is an entry deliberately not transferred. [Entry.Reason]
	// says why.
	ActionSkip Action = "skip"
)

type Direction 

type Direction string

Direction records which way a transfer went. 

const (
	// DirectionUpload is local to remote.
	DirectionUpload Direction = "upload"
	// DirectionDownload is remote to local.
	DirectionDownload Direction = "download"
)

type Entry 

type Entry struct {
	// Path is relative to the sync root and always slash-separated, on both
	// sides and on every OS, so it is comparable between the two ends. The
	// root itself is ".".
	Path string
	// Action is what happened, or what would have happened under a dry run.
	Action Action
	// Mode is the permission bits the destination entry was given. It is zero
	// for [ActionSymlink], where nothing is chmodded — a link's own bits are
	// not portable and are not carried. For [ActionSkip] it is the source's,
	// for information.
	Mode os.FileMode
	// Size is the byte count, for regular files only.
	Size int64
	// Target is the symlink target as written, when Action is [ActionSymlink].
	Target string
	// Reason explains the skip, when Action is [ActionSkip]. Empty otherwise.
	Reason string
}

Entry is one thing the walk found. The set of entries is identical for a dry run and for the transfer it previews.

type Option 

type Option func(*config)

Option configures a transfer. Options are functional so that adding one does not break callers, and so that a call site names only what it changes.

func WithDirMode 

func WithDirMode(mode os.FileMode) Option

WithDirMode sets the permission bits given to created directories when WithPreserveMode is off. Default: 0755. Ignored while preserving.

func WithDryRun 

func WithDryRun(dry bool) Option

WithDryRun walks the source and reports what would happen without creating, writing, renaming or changing anything at the destination. Default: false.

It is an option on the transfer functions rather than a separate Plan call on purpose: a preview is only worth showing an operator if it came from the same walk as the transfer, and a second entry point is a second thing to keep in step.

func WithFileMode 

func WithFileMode(mode os.FileMode) Option

WithFileMode sets the permission bits given to transferred files when WithPreserveMode is off. Default: 0644. Ignored while preserving.

func WithMaxDepth 

func WithMaxDepth(depth int) Option

WithMaxDepth bounds how far below the sync root the walk descends, counting the root as depth 0. Default: 64. A value below 1 restores the default.

Exceeding it is ErrMaxDepthExceeded rather than a silent truncation: a tree deeper than the limit is a tree this package has not fully copied, and a caller must not be told otherwise.

func WithPreserveModTime 

func WithPreserveModTime(preserve bool) Option

WithPreserveModTime carries the source modification time to the destination. Default: false.

Off by default because a transferred file is new at the destination and most callers want its mtime to say so. Turn it on when something downstream compares timestamps across the two sides.

func WithPreserveMode 

func WithPreserveMode(preserve bool) Option

WithPreserveMode carries the source permission bits to the destination. Default: true.

It is on by default because the alternative surfaces an hour later: an uploaded script that arrives 0644 fails at the moment something tries to run it, far from the transfer that caused it. With it off, files get WithFileMode and directories WithDirMode.

func WithSymlinkPolicy 

func WithSymlinkPolicy(p SymlinkPolicy) Option

WithSymlinkPolicy chooses what symlinks inside the tree mean. Default: SymlinkReplicate.

type PathError 

type PathError struct {
	// Op is the operation that failed: "open", "create", "mkdir", "readdir",
	// "lstat", "stat", "copy", "close", "chmod", "chtimes", "symlink",
	// "rename" or "walk".
	Op string
	// Side is the machine Path lives on.
	Side Side
	// Path is the absolute path on Side, as this package addressed it.
	Path string
	// Err is the underlying cause.
	Err error
}

PathError is the error every operation in this package returns. It names the operation, the side it happened on and the path it happened to, and unwraps to the underlying cause — so errors.Is against ErrSymlinkEscape, context.Canceled or fs.ErrPermission all work through it.

func (*PathError) Error 

func (e *PathError) Error() string

func (*PathError) Unwrap 

func (e *PathError) Unwrap() error

Unwrap returns the underlying cause.

type Result 

type Result struct {
	// Direction is which way the transfer went.
	Direction Direction
	// Source and Destination are the two roots, cleaned.
	Source      string
	Destination string
	// DryRun reports whether anything was actually written.
	DryRun bool
	// Entries is every entry the walk reached, in the order it reached them:
	// a directory before its contents, and names within a directory sorted,
	// so two runs over an unchanged tree report the same thing.
	Entries []Entry
	// Files, Dirs, Symlinks and Skipped count Entries by action.
	Files    int
	Dirs     int
	Symlinks int
	Skipped  int
	// Bytes is the total size of the files transferred.
	Bytes int64
	// Duration is how long the walk took, including the failed attempt when
	// the transfer returned an error.
	Duration time.Duration
}

Result is what happened. Transfers return one even when they fail, so a caller can tell an operator how far the transfer got before it stopped — which is the difference between "nothing moved" and "half the tree moved".

func Download 

func Download(ctx context.Context, sc *sftp.Client, remoteDir, localDir string, opts ...Option) (*Result, error)

Download recursively copies remoteDir to localDir over sc, which must be a live client the caller opened and continues to own.

It is the same walk as Upload with the two ends exchanged — the same escape check, the same temp-and-rename writer, the same cancellation. That is deliberate: the download path is where "it is only reading" quietly becomes "it wrote outside the destination", and one implementation cannot drift into two sets of safety properties.

func DownloadOverSSH 

func DownloadOverSSH(ctx context.Context, c *ssh.Client, remoteDir, localDir string, opts ...Option) (*Result, error)

DownloadOverSSH opens an SFTP subsystem on c, runs Download across it and closes the subsystem again. c is left open; see UploadOverSSH.

func Upload 

func Upload(ctx context.Context, sc *sftp.Client, localDir, remoteDir string, opts ...Option) (*Result, error)

Upload recursively copies localDir to remoteDir over sc, which must be a live client the caller opened and continues to own. Nothing here dials, authenticates, or reads an SSH config.

remoteDir is created if it does not exist, taking localDir's mode; if it already exists its own mode is left alone. Everything below it is created as needed.

A *Result is returned even when err is non-nil, describing how far the transfer got. Errors are *PathError and name the side and the path.

func UploadOverSSH 

func UploadOverSSH(ctx context.Context, c *ssh.Client, localDir, remoteDir string, opts ...Option) (*Result, error)

UploadOverSSH opens an SFTP subsystem on c, runs Upload across it and closes the subsystem again. c itself is neither opened nor closed by this package — it stays exactly as the caller left it, still usable for sessions and for further transfers.

Use it when a transfer is a one-off. When several transfers share a connection, open one *sftp.Client yourself and call Upload, so they share a subsystem too.

func (*Result) String 

func (r *Result) String() string

String is a one-line operator-facing summary.

type Side 

type Side string

Side names which end of a transfer a path lives on. Every error this package returns carries one. "permission denied" without a side sends an operator to the wrong machine, which is the failure mode this package exists to avoid. 

const (
	// Local is the machine running this code.
	Local Side = "local"
	// Remote is the machine on the far end of the SFTP connection.
	Remote Side = "remote"
)

type SymlinkPolicy 

type SymlinkPolicy int

SymlinkPolicy decides what a symlink inside the tree means.

The default refuses to follow anything out of the sync root. The policy that lifts that refusal is named so that choosing it reads as a decision at the call site rather than as a tuning knob. 

const (
	// SymlinkReplicate recreates a symlink on the destination with the same
	// target, and refuses one whose target escapes the sync root with
	// [ErrSymlinkEscape]. Nothing is read through the link, so a link to a
	// 40GB file or to a device node costs nothing and a loop is impossible.
	//
	// An absolute target counts as an escape even when it happens to point
	// inside the source root: the same absolute path on the other machine is
	// a different place, and this package will not guess which one you meant.
	//
	// This is the default.
	SymlinkReplicate SymlinkPolicy = iota

	// SymlinkSkip transfers no symlinks at all and records each one in the
	// [Result] as skipped. Nothing is read and nothing is created, so an
	// escaping link is not an error under this policy — it is simply not
	// copied. Use it when the destination must contain only regular files.
	SymlinkSkip

	// SymlinkDereferenceIncludingOutsideRoot follows every symlink and copies
	// what it points at, including targets outside the sync root. A link to
	// /etc/shadow copies /etc/shadow.
	//
	// This is the escape hatch, and the name is the warning. Choose it only
	// when you control the tree and know that following is what you want.
	// Symlink loops terminate at [WithMaxDepth] with [ErrMaxDepthExceeded]
	// rather than running forever.
	SymlinkDereferenceIncludingOutsideRoot
)

func (SymlinkPolicy) String 

func (p SymlinkPolicy) String() string

String implements fmt.Stringer.

Source Files

View all Source files

Directories

Path Synopsis
examples
overssh command
Package main demonstrates the shape a real caller uses: you own the connection, this library only uses it.
 Package main demonstrates the shape a real caller uses: you own the connection, this library only uses it.
preview command
Package main is a runnable end-to-end demonstration of go-sftpsync: it previews a directory transfer, then performs it, and prints both reports.
 Package main is a runnable end-to-end demonstration of go-sftpsync: it previews a directory transfer, then performs it, and prints both reports.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.