ddex_sftp

package module
v0.0.0-...-73dfddf Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 28, 2026 License: BSD-2-Clause Imports: 22 Imported by: 0

README

ddex-sftp

An SFTP server that doesn't make you want to throw your laptop out the window.

What is this?

ddex-sftp is a Go library for spinning up SFTP servers — either a plain vanilla one or a specialized server built for inbound DDEX music delivery ingestion. It's the kind of thing you reach for when you need an SFTP endpoint yesterday and don't want to duct-tape together SSH configs at 2 AM.

Why?

Because standing up an SFTP server in Go shouldn't require a PhD in SSH internals. And if you're in the music industry dealing with DDEX batch deliveries, you've already got enough problems.

Design Philosophy

Bring your own everything. This library is deliberately modular so it plays nicely with whatever you've already built:

  • Authentication — Plug in your own auth backend. Password auth ships out of the box, but the Auth interface is yours to implement however you like. LDAP, OAuth, a spreadsheet your ops team swears by — no judgment.

  • Storage drivers — Local filesystem and Google Cloud Storage are included. Need S3? Azure Blob? A network-attached tostrader from 2003? Implement the Driver interface and you're in business.

  • Logging — The generic Logger interface means you wire in your existing logging stack instead of adopting yet another opinionated logger that formats things slightly differently than the other five you're already using.

  • Batch handling — For DDEX workflows, the server automatically detects BatchComplete manifest uploads and triggers your BatchHandler with the manifest payload. You handle the business logic; the server handles the plumbing.

Quick Start

package main

import (
    "log"

    "golang.org/x/crypto/ssh"

    ddex "github.com/yongjohnlee80/ddex-sftp"
    "github.com/yongjohnlee80/ddex-sftp/auth"
)

func main() {
    // Parse your host key
    hostKey, _ := ssh.ParsePrivateKey([]byte("your-private-key"))

    // Set up a user
    user := auth.NewSimpleUser("alice", "s3cret")

    // Create the server
    server, err := ddex.NewServer(
        yourLogger,
        &ddex.ServerOptions{
            Auth:    auth.NewPasswordAuth(user),
            Port:    2022,
            HostKey: hostKey,
            Driver:  ddex.LocalDriverFactory("/var/sftp"),
        },
    )
    if err != nil {
        log.Fatal(err)
    }

    log.Fatal(server.ListenAndServe())
}

Storage Drivers

Driver Status Description
Local filesystem Included Files on disk. Old reliable.
Google Cloud Storage Included For when "the cloud" is actually the answer.
Your custom driver Driver interface Go wild.

Status

This is beta — the first public cut. The core is solid and functional, but there are more features on the roadmap:

  • Additional auth methods (public key registry, multi-user stores)
  • DDEX validation and batch handling (manifests, and resources)
  • More storage drivers
  • Rate limiting and connection management
  • Transfer hooks and event callbacks
  • Improved metrics and observability

Expect the API to evolve. If something breaks between versions, it probably got better.

License

This project is licensed under the BSD 2-Clause License.

In short: do whatever you want with it — use it, modify it, ship it in your own product, sell it on a USB stick at a flea market. Just keep the copyright notice and license text intact when you redistribute. No warranties, no guarantees, no strings attached beyond that.

Contributing

This is a personal project in its early days. Issues and ideas are welcome — pull requests even more so.

Documentation

Index

Constants

View Source 
const (
	// CmdRead represensts sftp.FileReader interface command, ReadAt
	CmdRead  = "Read"
	CmdWrite = "Write"
	CmdPut   = "Put"

	// CmdSetStat and others represenst sftp.FileCmder interface commands
	CmdSetStat = "Setstat"
	CmdRename  = "Rename"
	CmdRmdir   = "Rmdir"
	CmdRemove  = "Remove"
	CmdMkdir   = "Mkdir"
	CmdSymlink = "Symlink"

	// CmdList and others represenst sftp.FileLister interface commands
	CmdList     = "List"
	CmdStat     = "Stat"
	CmdReadlink = "Readlink"
)

Variables

This section is empty.

Functions

func GetBatchIDFromManifestMessage 

func GetBatchIDFromManifestMessage(filepath string) (string, bool)

GetBatchIDFromManifestMessage extracts the batch ID from a manifest filename if it matches the expected batch complete regex. Returns the ID and a success flag.

func GetMessagesFromManifestMessage 

func GetMessagesFromManifestMessage(ctx *Context, manifestFilepath string) ([]string, error)

GetMessagesFromManifestMessage discovers all XML message files in the same directory tree as the manifest file, excluding the manifest itself.

func LimitReader 

func LimitReader(r io.Reader, l RateLimiter) io.Reader

LimitReader returns a reader with limiter

func LimitWriter 

func LimitWriter(w io.Writer, l RateLimiter) io.Writer

LimitWriter returns a writer with limiter

func NewSftpContextual 

func NewSftpContextual[T logger.Severity](
	_l logger.Logger[T],
	name string,
	sessionID string,
	args ...any,
) logger.Logger[T]

NewSftpContextual creates a contextual logger with additional contextual data such as session ID and user information.

Types

type BatchHandler 

type BatchHandler interface {
	BatchComplete(ctx *Context, msg []byte) error
}

BatchHandler processes batches of messages and commits them upon successful execution or returns an error if failed.

type Context 

type Context struct {

	// Sess stores a reference to the current session associated with the context.
	Sess *Session

	// Cmd stores the command associated with the current context execution.
	Cmd string

	// Param holds the command parameter string associated with the current
	// request in the context.
	Param string
	// contains filtered or unexported fields
}

Context represents the execution context carrying request-specific data and a base context for cancellations or deadlines.

func NewContext 

func NewContext(
	ctx context.Context,
	sess *Session,
	cmd, param string,
) *Context

NewContext creates a new Context with the provided base context, session, command, parameter, and additional data.

func (*Context) Deadline 

func (c *Context) Deadline() (deadline time.Time, ok bool)

Deadline returns the time when the context will be canceled, if a deadline is set, along with a boolean indicating its presence.

func (*Context) Done 

func (c *Context) Done() <-chan struct{}

Done returns a channel that's closed when the context is canceled or its deadline expires.

func (*Context) Err 

func (c *Context) Err() error

Err returns a non-nil error if the context is canceled or exceeds its deadline; otherwise, it returns nil.

func (*Context) Value 

func (c *Context) Value(key any) any

Value retrieves the value associated with the provided key from the underlying context.

type DataLimiter 

type DataLimiter struct {
	// contains filtered or unexported fields
}

DataLimiter is a struct used to enforce rate limiting by tracking the rate, count, and time of operations.

func (*DataLimiter) Wait 

func (l *DataLimiter) Wait(count int64)

Wait enforces rate limiting by pausing execution if the permitted rate is exceeded based on the given count.

type Driver 

type Driver interface {
	// Stat retrieves the metadata of the object or directory at the specified
	// path within the storage and returns it.
	Stat(*Context, string) (os.FileInfo, error)

	// ListDir traverses the specified directory and applies the callback
	// function to each encountered file or directory.
	ListDir(*Context, string, func(os.FileInfo) error) error

	// DeleteDir removes a directory and all its contents recursively from the
	// storage associated with the driver.
	DeleteDir(*Context, string) error

	// DeleteFile deletes the specified file identified by its path.
	// Returns an error if the operation fails.
	DeleteFile(*Context, string) error

	// Rename changes the name of a resource identified by the given old name to
	// the specified new name within the provided context.
	Rename(*Context, string, string) error

	// MakeDir creates a new directory at the specified path within the storage
	// system. Returns an error if creation fails.
	MakeDir(*Context, string) error

	// GetFile retrieves a file starting from the specified offset in bytes and
	// returns the remaining size, reader, and any error.
	GetFile(*Context, string, int64) (int64, io.ReadCloser, error)

	// PutFile uploads data to the specified destination path.
	// Returns the number of bytes written and an error if the operation fails.
	PutFile(*Context, string, io.Reader) (int64, error)
}

Driver represents an interface for interacting with a storage system and performing file and directory operations.

func NewGCSDriver 

func NewGCSDriver(client *storage.Client, bucket, rootDir string) Driver

NewGCSDriver initializes a new GCSDriver instance with the given storage client and bucket name.

func NewLocalDriver 

func NewLocalDriver(rootPath string) (Driver, error)

NewLocalDriver creates a new instance of LocalDriver with the specified rootPath after verifying its absolute path.

type DriverFactory 

type DriverFactory func(u auth.User) (Driver, error)

DriverFactory is a function type that creates and returns a Driver instance for a given User or an error if creation fails.

func GCSDriverFactory 

func GCSDriverFactory(client *storage.Client, bucket string) DriverFactory

func LocalDriverFactory 

func LocalDriverFactory(rootPath string) DriverFactory

LocalDriverFactory creates a DriverFactory configured to produce LocalDriver instances with a user-specific root path for sandboxed environments.

type FileInfo 

type FileInfo interface {
	os.FileInfo

	// Owner returns the name of the user who owns the file.
	Owner() string

	// Group returns the group ownership of the file as a string.
	Group() string
}

FileInfo represents a file metadata interface that extends os.FileInfo with additional owner and group information.

type GCSDriver 

type GCSDriver struct {
	// contains filtered or unexported fields
}

GCSDriver implements the Driver interface for Google Cloud Storage. It provides methods for file and directory operations within a specific GCS bucket.

func (*GCSDriver) DeleteDir 

func (d *GCSDriver) DeleteDir(c *Context, p string) error

DeleteDir removes all objects within the specified directory path in the GCS bucket, including the directory itself.

func (*GCSDriver) DeleteFile 

func (d *GCSDriver) DeleteFile(c *Context, p string) error

DeleteFile deletes the file located at the specified path within the GCS bucket. Returns an error if the operation fails.

func (*GCSDriver) GetFile 

func (d *GCSDriver) GetFile(
	c *Context,
	p string,
	offset int64,
) (int64, io.ReadCloser, error)

GetFile retrieves a file from the specified path in the GCS bucket with an optional byte offset to start reading from.

func (*GCSDriver) ListDir 

func (d *GCSDriver) ListDir(
	c *Context,
	p string,
	callback func(os.FileInfo) error,
) error

ListDir retrieves and processes the contents of a given directory in GCS using a callback function.

func (*GCSDriver) MakeDir 

func (d *GCSDriver) MakeDir(c *Context, p string) error

MakeDir creates a directory at the specified path within the GCS bucket by writing a zero-byte object with a trailing slash.

func (*GCSDriver) PutFile 

func (d *GCSDriver) PutFile(
	c *Context,
	dest string,
	data io.Reader,
) (int64, error)

PutFile writes data from the provided reader to the specified destination path in the GCS bucket. Returns the number of bytes written or an error if the operation fails.

func (*GCSDriver) Rename 

func (d *GCSDriver) Rename(c *Context, from, to string) error

Rename moves an object from the `from` path to the `to` path within the GCS bucket and deletes the source on success. GCS does not support moving directories, so we need to copy and delete them as follows: 1. List objects with the source prefix. 2. Calculate the new path for each object. 3. Copy and then Delete the original objects.

func (*GCSDriver) Stat 

func (d *GCSDriver) Stat(c *Context, p string) (os.FileInfo, error)

Stat retrieves metadata for a file or directory at the given path in the GCS bucket. Returns os.FileInfo or an error.

type LocalDriver 

type LocalDriver struct {
	RootPath string
}

LocalDriver represents a driver implementation for local file system operations, rooted at the specified directory.

func (*LocalDriver) DeleteDir 

func (d *LocalDriver) DeleteDir(c *Context, path string) error

DeleteDir deletes the specified directory and all its contents. Returns an error if the path is not a directory.

func (*LocalDriver) DeleteFile 

func (d *LocalDriver) DeleteFile(c *Context, path string) error

DeleteFile removes the file at the specified path. Returns an error if the path does not exist or is a directory.

func (*LocalDriver) GetFile 

func (d *LocalDriver) GetFile(
	c *Context,
	path string,
	offset int64,
) (int64, io.ReadCloser, error)

GetFile retrieves a file at the specified path starting from the provided offset, returning its size, a ReadCloser, and an error.

func (*LocalDriver) ListDir 

func (d *LocalDriver) ListDir(
	c *Context,
	path string,
	callback func(os.FileInfo) error) error

ListDir traverses the specified directory and invokes the callback function for each file or directory encountered.

func (*LocalDriver) MakeDir 

func (d *LocalDriver) MakeDir(c *Context, path string) error

MakeDir creates a directory and any necessary parent directories at the specified path relative to the root directory.

func (*LocalDriver) PutFile 

func (d *LocalDriver) PutFile(
	c *Context,
	dest string,
	data io.Reader,
) (int64, error)

PutFile writes data from the provided reader to a file at the given destination path. Returns the number of bytes written and an error, if any.

func (*LocalDriver) Rename 

func (d *LocalDriver) Rename(c *Context, from string, to string) error

Rename renames a file or directory from the specified source path to the target path within the LocalDriver root directory.

func (*LocalDriver) Stat 

func (d *LocalDriver) Stat(c *Context, path string) (os.FileInfo, error)

Stat retrieves the file information for a specified path within the root directory of the LocalDriver instance.

type RateLimiter 

type RateLimiter interface {
	Wait(count int64)
}

RateLimiter defines an interface for controlling the rate of some actions over time.

func NewLimiter 

func NewLimiter(rate int64) RateLimiter

NewLimiter creates and returns a new RateLimiter instance configured with the specified rate.

type Server 

type Server struct {
	// Options defines a set of configuration settings for customizing the
	// behavior of a component or application.
	Options *ServerOptions
	// contains filtered or unexported fields
}

Server represents an SFTP server.

func NewServer 

func NewServer(
	logger Logger[Severity],
	opts *ServerOptions,
) (*Server, error)

NewServer creates a new SFTP server with the provided options.

func (*Server) BatchHandler 

func (s *Server) BatchHandler() BatchHandler

BatchHandler returns the BatchHandler implementation defined in the server options.

func (*Server) Close 

func (s *Server) Close() error

Close shuts down the server by canceling operations and closing the underlying network listener.

func (*Server) Driver 

func (s *Server) Driver() DriverFactory

Driver returns the DriverFactory implementation used as the storage backend for the server.

func (*Server) ListenAndServe 

func (s *Server) ListenAndServe() error

ListenAndServe starts the SFTP server and listens for connections.

type ServerOptions 

type ServerOptions struct {
	// Auth is the authentication mechanism used by the server.
	Auth auth.Auth

	// Port is the port the server will listen on.
	Port int

	// HostKey is the SSH private key for the server.
	HostKey ssh.Signer

	// Driver is the storage backend.
	Driver DriverFactory

	// BatchHandler processes batches of messages and commits them upon
	// successful execution or returns an error if failed.
	BatchHandler BatchHandler
}

ServerOptions defines the configuration for the SFTP server.

type Session 

type Session struct {
	Logger[Severity]
	// contains filtered or unexported fields
}

Session represents an SFTP session.

func NewSession 

func NewSession(server *Server, user, rootPath string) (*Session, error)

NewSession initializes and returns a new SFTP session with the specified server, user, and rootPath details.

func (*Session) Filecmd 

func (s *Session) Filecmd(r *sftp.Request) error

Filecmd implements sftp.FileCmder interface.

func (*Session) Filelist 

func (s *Session) Filelist(r *sftp.Request) (sftp.ListerAt, error)

Filelist implements sftp.FileLister interface.

func (*Session) Fileput 

func (s *Session) Fileput(r *sftp.Request) (io.WriterAt, error)

Fileput is called by the SFTP server for uploads. It satisfies the sftp.Handlers interface if we use the RequestServer.

func (*Session) Fileread 

func (s *Session) Fileread(r *sftp.Request) (io.ReaderAt, error)

Fileread implements sftp.FileReader interface.

func (*Session) Filewrite 

func (s *Session) Filewrite(r *sftp.Request) (io.WriterAt, error)

Filewrite implements sftp.FileWriter interface.

func (*Session) RootPath 

func (s *Session) RootPath() string

RootPath returns the root path associated with the current SFTP session.

func (*Session) User 

func (s *Session) User() string

User returns the username associated with the current SFTP session.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.