uploadfun

README

uploadfun

A headless, config-driven uploader that fans one batch of files out to many FTP, FTPS, and SFTP endpoints at once, concurrently, with retries and post-upload verification. Built for unattended automation (cron, CI, folder-watchers), not interactive use. A single static binary and one YAML file.

You point it at some files and a list of endpoints; it uploads every file to every endpoint in parallel, verifies each transfer, retries transient failures, and exits non-zero if anything failed so a wrapping script can react.

$ uploadfun ./batch --config stocks.yml
[shutterstock] photo1.jpg: uploaded (verified: size) in 1.21s
[dreamstime] photo1.jpg: uploaded (verified: size) in 0.98s
[shutterstock] photo2.jpg: uploaded (verified: size) in 1.15s
[dreamstime] photo2.jpg: uploaded (verified: size) in 1.04s
[shutterstock] done: 2 succeeded, 0 failed in 3.10s
[dreamstime] done: 2 succeeded, 0 failed in 2.87s

Features

• One batch, many destinations. Declare endpoints in YAML; add or remove one without touching code. Every endpoint uploads in its own goroutine.
• Three protocols, one model: ftp, ftps (explicit AUTH TLS), and sftp (password or private-key auth, host keys checked against ~/.ssh/known_hosts).
• Verification on by default. Every upload's remote size is checked against the local file to catch truncation. Disable with --no-verify.
• Retries with reconnect. A fixed number of attempts per file with a fixed delay, each attempt reconnecting from scratch. Configurable globally and per endpoint.
• Fail-fast preflight (--dry-run): connect, authenticate, and prove the target directory is writable with a self-deleting probe file, without touching the files you are actually sending.
• Automation-friendly output: quiet / default / verbose text, or newline-delimited JSON (--json), with meaningful exit codes and errors always on stderr even under --quiet.
• Secrets via env vars: ${ENV_VAR} interpolation in any config field, resolved at load time. Literal values still allowed.
• Zero runtime dependencies: one static binary, no daemon, no config server.

Install

Homebrew (macOS / Linux)

brew install alexeyu/tap/uploadfun

go install

go install github.com/alexeyu/uploadfun/cmd/uploadfun@latest

Prebuilt binaries

Download the archive for your OS/architecture from the latest release, unpack it, and put uploadfun on your PATH. Linux, macOS, and Windows on amd64 and arm64 are built for every release; each archive is listed in checksums.txt.

Quick start

1. Write a config (stocks.yml) listing your endpoints. Start from config.yml.example, or see Configuration below for the full field reference.

2. Preflight it before you trust it to a cron job:

   $ export SHUTTERSTOCK_FTP_PASSWORD=...
   $ uploadfun ./batch --config stocks.yml --dry-run
   [shutterstock] dry-run ok: reachable and writable, would upload 12 files
   [dreamstime] dry-run ok: reachable and writable, would upload 12 files
   

3. Upload for real:

   $ uploadfun ./batch photo-extra.jpg --config stocks.yml
   

Positional arguments may be files or directories, mixed freely. Directories expand non-recursively: every regular file is included; subdirectories and dotfiles are skipped. Two inputs that would land under the same remote name (same basename) are rejected up front.

Configuration

Config is a YAML file with a list of endpoints plus optional global policy defaults. Every policy key can be overridden per endpoint.

endpoints:
  - name: shutterstock          # unique label, used in output and errors
    protocol: ftps              # ftp | ftps | sftp
    host: ftp.shutterstock.com
    port: 21                    # optional; defaults to the protocol's standard port
    username: myuser
    password: ${SHUTTERSTOCK_FTP_PASSWORD}
    overwrite: delete-first     # delete-first (default) | direct

  - name: dreamstime
    protocol: sftp
    host: sftp.dreamstime.com
    username: myuser
    private_key: ~/.ssh/id_ed25519   # ~ is expanded

  - name: legacy-agency
    protocol: ftp
    host: ftp.legacy-agency.example
    username: myuser
    password: hunter2           # literal secrets allowed, not forced to env vars
    attempts: 5                 # per-endpoint override of the global default below

# Global policy defaults (each overridable per endpoint):
attempts: 3                     # total tries per file
retry_delay: 2s
connect_timeout: 30s            # bounds the whole connect + login, not just the TCP dial
stall_timeout: 5m               # fail a transfer idle this long; 0 disables
max_consecutive_connect_failures: 3   # write an endpoint off after this many connect failures in a row

Endpoint fields

Field	Applies to	Notes
name	all	Required, unique. Shown in all output.
protocol	all	ftp, ftps, or sftp. Required.
host	all	Required.
port	all	Optional; defaults to the protocol's standard port.
username	all	Required.
password	ftp, ftps, sftp	Required for ftp/ftps. For sftp, an alternative or complement to private_key.
private_key	sftp	Path to an SSH private key (~ expanded). If both this and password are set, the key wins and password is used only as its passphrase when the key is encrypted.
overwrite	all	delete-first (default) deletes any existing remote file first; direct uploads straight over it.
insecure_skip_verify	ftps	Disables TLS certificate verification. For self-signed / test servers only; never use against a production endpoint.

Policy fields (global or per-endpoint)

Field	Default	Meaning
attempts	3	Total tries per file before it is counted as failed.
retry_delay	2s	Fixed wait between attempts (Go duration string).
connect_timeout	30s	Bounds the entire connect + authenticate sequence.
stall_timeout	5m	Fails a transfer that makes no forward progress for this long. 0 disables.
max_consecutive_connect_failures	3	After this many connect failures in a row, the endpoint's remaining files are skipped as unreachable instead of each retrying.

Secrets

${ENV_VAR} interpolation works in any string field and is resolved when the config loads. A referenced variable that is unset is a load-time error, so a typo surfaces immediately instead of as a confusing auth failure mid-run.

Literal secrets are allowed but discouraged. Because a config file can hold plaintext credentials, uploadfun prints a warning if the file is readable by group or others; chmod 600 stocks.yml and keep it out of version control.

Usage

uploadfun <path>... --config <file> [flags]

Flag	Effect
--config <file>	Path to the YAML config. Required.
--dry-run	Connect, authenticate, and probe each endpoint for writability, then report how many files would upload. No transfers, no deletes of real files.
--no-verify	Skip post-upload size verification.
--quiet	Suppress non-error stdout. Errors still print to stderr.
--verbose	Print the full event stream, including byte-level progress.
--json	Emit newline-delimited JSON instead of text (honors the verbosity level).
--version	Print version and exit.
--help	Print usage and exit.

--quiet and --verbose are mutually exclusive. Flags and paths may appear in any order.

Output modes

Mode	stdout	stderr
--quiet	nothing	errors
default	per-file success + per-endpoint summary	errors
--verbose	full event stream incl. byte-level progress	errors

Failure events always go to stderr, even under --quiet: an unattended job still needs to see what broke.

JSON output

--json reformats whatever the current verbosity level would print as one JSON object per line, keeping the same stdout/stderr split. Each line carries a type discriminator (file_start, progress, file_success, file_error, endpoint_unreachable, endpoint_given_up, endpoint_done, dry_run):

$ uploadfun ./batch --config stocks.yml --json
{"type":"file_success","endpoint":"shutterstock","file":"batch/photo1.jpg","verifyMethod":"size","durationSec":1.21}
{"type":"endpoint_done","endpoint":"shutterstock","succeeded":1,"failed":0,"durationSec":1.34}

Exit codes

Code	Meaning
0	Every file uploaded (and verified) on every endpoint.
1	Partial failure: at least one file failed after exhausting retries on at least one endpoint.
2	Configuration error (bad or invalid YAML, validation failures).
3	Usage error (bad flags, an input path that does not exist).

How it works

• Concurrency. One goroutine per endpoint, fully parallel. Within an endpoint, files upload sequentially over a single reused connection (connect once, upload all, disconnect), which amortizes login cost and respects servers that cap concurrent connections.
• Retries. A fixed attempt count with a fixed delay between tries. Each retry reconnects. Upload and verification are retried together: a verification failure is treated the same as an upload failure.
• Verification. After each upload the remote size is compared against the local file (SIZE / Stat), which catches truncation cheaply and works on all three protocols. Hash-based verification is planned but not in v1; SFTP has no standard equivalent. --no-verify turns it off.
• SSH host keys. SFTP endpoints are verified against ~/.ssh/known_hosts. There is no insecure-by-default fallback: a host you have not trusted yet fails with an error, the same trust-on-first-use flow as OpenSSH. Connect once with ssh (or ssh-keyscan) to record the key first.
• Cancellation. Ctrl-C / SIGTERM cancels cleanly: workers stop starting new transfers and retries promptly, though an in-flight blocking transfer runs to completion first. Each endpoint still reports its results.

Use as a Go library

The CLI is a thin wrapper over the uploadfun package. Upload returns a channel of typed events you range over:

package main

import (
	"context"
	"fmt"

	"github.com/alexeyu/uploadfun"
)

func main() {
	endpoints, err := uploadfun.LoadConfig("stocks.yml")
	if err != nil {
		panic(err)
	}

	files := []string{"photo1.jpg", "photo2.jpg"}
	events := uploadfun.Upload(context.Background(), files, endpoints, uploadfun.Options{})

	// You MUST keep receiving until the channel closes: workers send
	// unbuffered, so bailing out early would deadlock them.
	for ev := range events {
		switch e := ev.(type) {
		case uploadfun.FileSuccessEvent:
			fmt.Printf("%s -> %s ok\n", e.File, e.Endpoint)
		case uploadfun.FileErrorEvent:
			fmt.Printf("%s -> %s failed: %s\n", e.File, e.Endpoint, e.Reason)
		}
	}
}

You can also build []uploadfun.Endpoint yourself instead of loading YAML. See the package reference for the full event vocabulary and Options.

Building from source

Requires Go (see go.mod for the version). A Makefile wraps the common tasks:

make build              # build ./uploadfun with the version stamped in
make test               # unit tests, race detector on
make test-integration   # integration tests against real servers in Docker
make lint               # golangci-lint

Integration tests spin up real pure-ftpd and atmoz/sftp containers via the docker CLI and are gated behind the integration build tag, so a plain go test ./... never needs Docker. They skip themselves with a message if Docker is not reachable.

Releasing (maintainers)

Releases are cut by GoReleaser from a tag:

git tag v0.1.0
git push origin v0.1.0

The Release workflow then cross-compiles every target, attaches checksummed archives to the GitHub Release, and updates the Homebrew tap. Publishing the Homebrew cask requires a HOMEBREW_TAP_GITHUB_TOKEN repository secret: a token with write access to alexeyu/homebrew-tap. If that secret is absent the release still succeeds and only the cask push is skipped. Validate config changes locally with goreleaser check and dry-run a full build with goreleaser release --snapshot --clean.

Stability

Pre-1.0. The tool works and is tested, but the config schema and the Go library API may still change between minor versions. v1.0.0 will mark both as stable.

License

MIT.

Documentation

Overview

Package uploadfun fans out local files to multiple FTP/FTPS/SFTP endpoints concurrently, with retry and an event stream, for unattended automation rather than interactive use.

Index

• Constants
• func Upload(ctx context.Context, files []string, endpoints []Endpoint, opts Options) <-chan UploadEvent
• type DryRunEvent
• type Duration
  • func (d Duration) MarshalJSON() ([]byte, error)
• type Endpoint
  • func LoadConfig(path string) ([]Endpoint, error)
• type EndpointDoneEvent
• type EndpointGivenUpEvent
• type EndpointUnreachableEvent
• type FileErrorEvent
• type FileStartEvent
• type FileSuccessEvent
• type Options
• type OverwriteMode
• type ProgressEvent
• type Protocol
• type UploadEvent

Constants

const (
	DefaultAttempts       = 3
	DefaultRetryDelay     = 2 * time.Second
	DefaultConnectTimeout = 30 * time.Second
	DefaultStallTimeout   = 5 * time.Minute
	// DefaultMaxConsecutiveConnectFailures is a floor, not a multiple of
	// Attempts: even Attempts: 1 gets a few tries across the batch before
	// being written off, rather than quitting after one connect blip.
	DefaultMaxConsecutiveConnectFailures = 3
)

Defaults applied by LoadConfig to any endpoint (or the config file as a whole) that doesn't set its own value.

Functions

func Upload

func Upload(
	ctx context.Context, files []string, endpoints []Endpoint, opts Options,
) <-chan UploadEvent

Upload fans out files to endpoints, one goroutine per endpoint, retrying per Endpoint.Attempts/RetryDelay. The channel closes once every worker finishes; canceling ctx doesn't stop in-flight transfers. The caller must keep receiving from the channel until it closes - each worker sends on it unbuffered, so if the caller stops ranging early (e.g. returning on the first error), every worker blocks forever on its next send.

Types

type DryRunEvent

type DryRunEvent struct {
	Endpoint string `json:"endpoint"`
	// Files is how many files a real run would upload to this endpoint;
	// meaningful only when Err is nil.
	Files int `json:"files"`
	// Err is set if connecting, authenticating, or the write probe failed;
	// nil means the endpoint is reachable and writable and Files reflects
	// the planned upload.
	Err error `json:"-"`
}

DryRunEvent reports the outcome of a --dry-run preflight for one endpoint: connect and authenticate to prove it's reachable, probe the target directory for writability, then report how many files a real run would upload. Exactly one is sent per endpoint when Options.DryRun is set, replacing the per-file events.

type Duration

type Duration time.Duration

Duration wraps time.Duration so an event's timing field serializes to JSON as a seconds number rounded to two decimals - matching the text output and friendlier than the raw nanosecond count a plain time.Duration would marshal to.

func (Duration) MarshalJSON

func (d Duration) MarshalJSON() ([]byte, error)

type Endpoint

type Endpoint struct {
	Name     string
	Protocol Protocol
	Host     string
	Port     int
	Username string
	Password string
	// PrivateKey is a path to an SSH private key file, used by the sftp
	// protocol as an alternative to Password.
	PrivateKey string
	Overwrite  OverwriteMode
	// InsecureSkipVerify disables TLS certificate verification for the
	// ftps protocol. Only for self-signed/test servers - it accepts any
	// certificate, so never enable it against a production endpoint.
	InsecureSkipVerify bool

	Attempts       int
	RetryDelay     time.Duration
	ConnectTimeout time.Duration
	// StallTimeout bounds how long a transfer may go without forward
	// progress; zero disables idle-stall protection.
	StallTimeout time.Duration
	// MaxConsecutiveConnectFailures bounds how many connect failures in a
	// row this endpoint tolerates across the whole batch before the rest
	// of the files are skipped as unreachable, independent of Attempts.
	MaxConsecutiveConnectFailures int
}

Endpoint is one remote destination to upload to, fully resolved (global config defaults already merged in) by LoadConfig.

func LoadConfig

func LoadConfig(path string) ([]Endpoint, error)

LoadConfig reads and validates a YAML endpoint config, resolving ${ENV_VAR} interpolation and global-default/per-endpoint-override merging. It collects every error instead of stopping at the first.

type EndpointDoneEvent

type EndpointDoneEvent struct {
	Endpoint  string `json:"endpoint"`
	Succeeded int    `json:"succeeded"`
	Failed    int    `json:"failed"`
	// Elapsed is the endpoint worker's total wall-clock time across the
	// whole batch, including connects, retries, and backoff.
	Elapsed Duration `json:"durationSec"`
}

EndpointDoneEvent reports that one endpoint's worker has finished (uploaded or given up on) every file and disconnected.

type EndpointGivenUpEvent

type EndpointGivenUpEvent struct {
	Endpoint     string   `json:"endpoint"`
	Reason       string   `json:"reason"`
	SkippedFiles []string `json:"skippedFiles"`
}

EndpointGivenUpEvent reports that one endpoint's worker abandoned the rest of the batch after hitting an unrecoverable transfer error (for example, an SFTP permission-denied response), covering every skipped file in a single event.

type EndpointUnreachableEvent

type EndpointUnreachableEvent struct {
	Endpoint            string   `json:"endpoint"`
	ConsecutiveFailures int      `json:"consecutiveFailures"`
	SkippedFiles        []string `json:"skippedFiles"`
}

EndpointUnreachableEvent reports that, after ConsecutiveFailures connect failures in a row, one endpoint's worker gave up on the rest of the batch, covering every skipped file in a single event.

type FileErrorEvent

type FileErrorEvent struct {
	Endpoint string `json:"endpoint"`
	File     string `json:"file"`
	Attempt  int    `json:"attempt"`
	Reason   string `json:"reason"`
	// Err is the underlying error; excluded from JSON output since the
	// error interface carries no exported fields worth serializing (its
	// message is already captured in Reason).
	Err error `json:"-"`
}

FileErrorEvent reports a single failed attempt (upload or verification) for a file on one endpoint. Attempt is 1-based; further attempts follow up to the endpoint's Attempts budget before the file is given up on.

type FileStartEvent

type FileStartEvent struct {
	Endpoint string `json:"endpoint"`
	File     string `json:"file"`
	Attempt  int    `json:"attempt"`
}

FileStartEvent reports that an endpoint worker is about to attempt one file - emitted once per attempt, before delete/upload/verify, so consumers know a (possibly long) transfer is underway before the first ProgressEvent arrives.

type FileSuccessEvent

type FileSuccessEvent struct {
	Endpoint string `json:"endpoint"`
	File     string `json:"file"`
	// VerifyMethod describes what verification was performed ("size",
	// "size+hash"), or "" if disabled (NoVerify).
	VerifyMethod string `json:"verifyMethod,omitempty"`
	// Elapsed is the wall-clock time of the successful attempt's work -
	// delete (if any), upload, and verification - excluding earlier failed
	// attempts and their retry backoff.
	Elapsed Duration `json:"durationSec"`
}

FileSuccessEvent reports that a file was uploaded (and, unless NoVerify, verified) successfully on one endpoint.

type Options

type Options struct {
	// NoVerify disables the post-upload size/hash verification that's on
	// by default.
	NoVerify bool
	// DryRun connects and authenticates per endpoint, verifies the target
	// directory is writable by round-tripping a throwaway probe file, and
	// reports how many files a real run would upload - without touching any
	// of the actual files being sent.
	DryRun bool
}

Options controls behavior of Upload that isn't per-endpoint config.

type OverwriteMode

type OverwriteMode string

OverwriteMode controls how an existing remote file with the same name is handled before upload.

const (
	// OverwriteDeleteFirst deletes any existing remote file before
	// uploading. It's the default: it avoids servers that reject a PUT
	// over an existing file.
	OverwriteDeleteFirst OverwriteMode = "delete-first"
	// OverwriteDirect uploads straight over any existing remote file,
	// avoiding the brief window where the remote file doesn't exist at
	// all between delete and re-upload.
	OverwriteDirect OverwriteMode = "direct"
)

type ProgressEvent

type ProgressEvent struct {
	Endpoint   string `json:"endpoint"`
	File       string `json:"file"`
	BytesSent  int64  `json:"bytesSent"`
	TotalBytes int64  `json:"totalBytes"`
}

ProgressEvent reports byte-level upload progress for one file on one endpoint. It is always emitted; consumers that don't want progress detail (like the CLI's non-verbose modes) simply ignore it.

type Protocol

type Protocol string

Protocol identifies which transport an Endpoint uses.

const (
	ProtocolFTP  Protocol = "ftp"
	ProtocolFTPS Protocol = "ftps"
	ProtocolSFTP Protocol = "sftp"
)

type UploadEvent

type UploadEvent interface {
	// contains filtered or unexported methods
}

UploadEvent is the vocabulary of events sent on the channel returned by Upload; consumers type-switch on it to distinguish the event kinds below.