README
¶
go-rsyncfs
The rsync daemon protocol, as a Go library. Client presents a remote rsync module as an io/fs.FS; Server serves local io/fs.FS modules over the protocol. Library-first: no CLI, no sockets, no encryption -- callers wire up transport (TCP, SSH, pipes, etc) themselves by handing each operation an io.ReadWriter.
Protocol versions 20 through 32 (headroom to 40), with optional pinning to a specific version. Instances are constructed once and reused across operations; the library never spawns goroutines.
// Serve a local directory as rsync module "stuff"
srv, _ := rsyncfs.NewServer(&rsyncfs.ServerModule{
Name: "stuff",
FS: os.DirFS("/path/to/stuff"),
})
conn := netConn // caller-owned transport
srv.HandleConnection(conn)
// Browse a remote module as a filesystem
sess, _ := rsyncfs.Client{Module: "stuff"}.Connect(conn2) // or set ConnectFunc for lazy dialing
f, _ := sess.Open("subdir/file.txt")
Testing is trivial: rsyncfs.BufPipe() connects a Client directly to a Server in memory (plain net.Pipe/io.Pipe deadlock on the handshake, hence the bounded variant).
Layout
- root package
rsyncfs--Client,Server,Session, the transport-agnostic API protocol-- wire protocol: greeting exchange, file list encoding, delta streamprotocol/mux-- rsync's multiplexed I/O framing
Status
v1: read-only transfers, metadata within what io/fs.FS covers (sizes, mtimes, symlinks), optional auth/ACL via callbacks. Writability (push), hardlinks/xattrs/device files, partial-transfer caching, and on-the-wire compression are planned future work -- see goals.md.
Integration tests run against the real upstream rsync binaries (including old versions covering protocols 27-32 from .upstream/old_versions/); they skip under go test -short.
Documentation
¶
Overview ¶
Package rsyncfs implements the rsync daemon protocol as a Go library. Client presents a remote rsync module as an io/fs.FS, and Server serves local io/fs.FS modules over the protocol. The library is transport-agnostic -- no CLI, no sockets, no encryption: callers wire up the transport (TCP, SSH, pipes, etc) themselves by handing each operation an io.ReadWriter.
It speaks protocol versions 20 through 32 (with headroom to 40), with optional pinning to a specific version. Instances are constructed once and reused across operations, and the library never spawns goroutines.
// serve a local directory as rsync module "stuff"
srv, _ := rsyncfs.NewServer(&rsyncfs.ServerModule{
Name: "stuff",
FS: os.DirFS("/path/to/stuff"),
})
srv.HandleConnection(conn)
// browse a remote module as a filesystem
sess, _ := rsyncfs.Client{Module: "stuff"}.Connect(conn2)
f, _ := sess.Open("subdir/file.txt")
BufPipe connects a Client directly to a Server in memory, which makes tests trivial (plain net.Pipe and io.Pipe deadlock on the handshake, hence the bounded variant).
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BufPipe ¶
func BufPipe() (a, b io.ReadWriteCloser)
BufPipe returns two connected, in-memory io.ReadWriteCloser ends, each backed by a bounded buffer.
Unlike net.Pipe or io.Pipe -- which are zero-capacity and block a write until the peer reads -- each direction here buffers, so a writer can make progress without the reader simultaneously consuming. The rsync handshake depends on exactly that: both sides write their greeting before reading the peer's, and the vstring algorithm negotiation has both sides send before receiving. A real transport (TCP, Unix socket, etc) supplies that capacity via its kernel buffer; BufPipe supplies it in-process so a Client and a Server can be wired together with no network stack. The bound mirrors what those kernel buffers do -- apply backpressure when the receiver falls behind and cap memory -- rather than growing without limit.
Types ¶
type Client ¶
type Client struct {
// Module is the rsync module name to connect to. Empty
// string enables root mode (all modules as top-level directories).
Module string
// Greeting is the greeting sent to the server. Zero-value fields
// are filled by [protocol.Greeting.ApplyDefaults].
Greeting protocol.Greeting
// AuthUser is the username for module authentication. Empty
// string means anonymous access.
AuthUser string
// AuthResponse computes the auth response hash given
// the server's challenge and the negotiated digest algorithm.
// The returned raw hash bytes are base64-encoded by the
// library before sending. Nil means anonymous access (AuthUser
// must also be empty). Use [PasswordAuth] for the standard
// password+challenge digest flow.
AuthResponse func(digest string, challenge []byte) ([]byte, error)
// LogSink, if set, receives every log-level control frame the daemon
// emits except MSG_ERROR_XFER -- those are accumulated and attached to
// the transfer error instead, so a truncated stream names the daemon's
// actual diagnostic. Used for observability only; it does not affect the
// transfer outcome. The code is a protocol/mux message code (MsgInfo,
// MsgError, MsgWarning, the sibling log codes); MsgNoop is not delivered.
// The text is the wire payload verbatim (typically already
// newline-terminated), so log it with `log.Print` or `fmt.Fprint` rather
// than a `*ln` variant, which would add a second newline.
LogSink func(code uint8, text string)
// ConnectFunc creates a new connection to the rsync server.
// Required for root mode (Module == ""). The moduleName argument
// is the target module name, or empty string for a #list request.
// The caller is responsible for closing the returned ReadWriter.
//
// In root mode, each FS operation (listing modules, opening a
// module) gets its own connection -- the server closes the
// connection after #list, so a single persistent connection is
// not possible.
//
// When used with [Client.Connect] and a nil io.ReadWriter,
// ConnectFunc is called with the configured Module name to create
// the connection.
ConnectFunc func(moduleName string) (io.ReadWriter, error)
}
Client configures a connection to an rsync daemon module. Construct directly with &Client{...}; zero-value fields use sensible defaults applied lazily during Client.Connect or Client.OpenRoot.
Example ¶
ExampleClient pulls a file end-to-end over a TCP connection: the Client.ConnectFunc dials the daemon, Client.Connect runs the handshake on the resulting connection, and Session.Open triggers the transfer.
// a single anonymous module, served over loopback TCP
// (see ExampleServer)
server, err := NewServer(&ServerModule{
Name: "data",
FS: fstest.MapFS{
"hello.txt": &fstest.MapFile{Data: []byte("hello, rsync!")},
},
})
if err != nil {
panic(err)
}
addr, stop := startExampleServer(server)
defer stop()
// the [Client] is plain config: the module to open, a ConnectFunc that
// produces a connection to the daemon (here a plain TCP dial, but an SSH
// session or any io.ReadWriter would work just as well), and a LogSink
// that forwards the daemon's log-level control frames to stderr
client := Client{
Module: "data",
LogSink: func(code uint8, text string) {
fmt.Fprint(os.Stderr, text)
},
ConnectFunc: func(string) (io.ReadWriter, error) {
return net.Dial("tcp", addr)
},
}
// Connect(nil) calls the ConnectFunc itself, then runs the full
// handshake
session, err := client.Connect(nil)
if err != nil {
panic(err)
}
// Open is a complete, self-contained rsync transfer of the file
f, err := session.Open("hello.txt")
if err != nil {
panic(err)
}
data, err := io.ReadAll(f)
if err != nil {
panic(err)
}
fmt.Println(string(data))
Output: hello, rsync!
Example (Auth) ¶
ExampleClient_auth uses PasswordAuth to answer the daemon's auth challenge with the "zerocool" / "hack the planet" credentials, then shows the wrong username and password being rejected.
// the module requires the "zerocool" username (the AuthCallback side
// is shown in ExampleServer_auth)
server, err := NewServer(&ServerModule{
Name: "data",
FS: fstest.MapFS{
"hello.txt": &fstest.MapFile{Data: []byte("hello, rsync!")},
},
AuthCallback: func(username string, challenge []byte) ([]byte, error) {
if username != "zerocool" {
return nil, fmt.Errorf("unknown user %q", username)
}
sum := md5.Sum(append([]byte("hack the planet"), challenge...))
return sum[:], nil
},
})
if err != nil {
panic(err)
}
addr, stop := startExampleServer(server)
defer stop()
// the client answers the challenge with digest(password + challenge):
// PasswordAuth wraps the password in an AuthResponse for it
client := Client{
Module: "data",
AuthUser: "zerocool",
AuthResponse: PasswordAuth("hack the planet"),
ConnectFunc: func(string) (io.ReadWriter, error) {
return net.Dial("tcp", addr)
},
}
session, err := client.Connect(nil)
if err != nil {
panic(err)
}
f, err := session.Open("hello.txt")
if err != nil {
panic(err)
}
data, err := io.ReadAll(f)
if err != nil {
panic(err)
}
fmt.Println(string(data))
// the wrong username and password: the server answers the auth
// response with an @ERROR and the connection is aborted
bad := Client{
Module: "data",
AuthUser: "werner",
AuthResponse: PasswordAuth("my voice is my passport"),
ConnectFunc: func(string) (io.ReadWriter, error) {
return net.Dial("tcp", addr)
},
}
if _, err = bad.Connect(nil); err != nil {
fmt.Println(err)
}
Output: hello, rsync! read server response: Authentication failed
Example (RootMode) ¶
ExampleClient_rootMode creates a Client in root mode (Module is empty), where the modules are top-level directories and every FS operation runs on its own connection: it lists the available modules via Session.Open on "." and then opens a file within a specific module.
// two modules over loopback TCP
server, err := NewServer(
&ServerModule{
Name: "public",
Comment: "public data",
FS: fstest.MapFS{
"welcome.txt": &fstest.MapFile{Data: []byte("welcome!")},
},
},
&ServerModule{
Name: "data",
Comment: "the example data",
FS: fstest.MapFS{
"hello.txt": &fstest.MapFile{Data: []byte("hello, rsync!")},
},
},
)
if err != nil {
panic(err)
}
addr, stop := startExampleServer(server)
defer stop()
// root mode: no module, so the modules themselves become top-level
// directories and each operation dials the daemon fresh
root, err := Client{
ConnectFunc: func(string) (io.ReadWriter, error) {
return net.Dial("tcp", addr)
},
}.OpenRoot()
if err != nil {
panic(err)
}
// list the available modules: each appears once as a directory and
// again as a hidden ".<module>\t<comment>" symlink that carries the
// module's comment (the hidden entries are dropped below)
d, err := root.Open(".")
if err != nil {
panic(err)
}
entries, err := d.(fs.ReadDirFile).ReadDir(-1)
if err != nil {
panic(err)
}
for _, e := range entries {
if !strings.HasPrefix(e.Name(), ".") {
fmt.Println(e.Name())
}
}
// open a file within a specific module: "data/hello.txt" dials the
// "data" module and pulls the file
f, err := root.Open("data/hello.txt")
if err != nil {
panic(err)
}
data, err := io.ReadAll(f)
if err != nil {
panic(err)
}
fmt.Println(string(data))
Output: data public hello, rsync!
func (Client) Connect ¶
func (c Client) Connect(rw io.ReadWriter) (*Session, error)
Connect runs the full pre-transfer handshake (greeting exchange, version negotiation, module selection, authentication, argument transmission, compat flags, algorithm negotiation, checksum seed, filter list) and returns an active session ready for filesystem operations. If rw is nil and Client.ConnectFunc is set, ConnectFunc is called with the configured module name to create the connection. For root mode (Module == ""), use Client.OpenRoot instead.
The session's input is always multiplexed (the daemon's output is muxed on every supported protocol version); its output is multiplexed for proto >= 30 and raw below. The file list itself is read by the first Open.
func (Client) OpenRoot ¶
OpenRoot returns a Session for root mode (modules as top-level directories). Unlike Client.Connect, it does not establish a live connection -- each FS operation gets its own connection via Client.ConnectFunc (the server closes the connection after #list, so a single persistent connection is not possible).
type Server ¶
type Server struct {
// Greeting is the greeting the server advertises on every connection.
// Zero-value fields are filled by [protocol.Greeting.ApplyDefaults].
Greeting protocol.Greeting
// LogSink, if set, receives every log-level control frame the peer emits
// except MSG_ERROR_XFER -- those are accumulated and attached to the
// transfer error instead, so a truncated stream names the peer's actual
// diagnostic. Used for observability only; it does not affect the
// transfer outcome. The code is a protocol/mux message code (MsgInfo,
// MsgError, MsgWarning, the sibling log codes); MsgNoop is not delivered.
// The text is the wire payload verbatim (typically already
// newline-terminated), so log it with `log.Print` or `fmt.Fprint` rather
// than a `*ln` variant, which would add a second newline.
LogSink func(code uint8, text string)
// contains filtered or unexported fields
}
Server represents an rsync daemon that serves one or more modules. Construct with NewServer. A single Server handles multiple connections.
Example ¶
ExampleServer creates a Server with a single module backed by [fstest.MapFS] and serves it over a loopback TCP socket, one Server.HandleConnection goroutine per connection, then pulls a file from it with a Client.
// a single module backed by a [fstest.MapFS]
server, err := NewServer(&ServerModule{
Name: "data",
Comment: "the example data",
FS: fstest.MapFS{
"hello.txt": &fstest.MapFile{Data: []byte("hello, rsync!")},
},
})
if err != nil {
panic(err)
}
// the server's LogSink receives the log-level control frames its peer
// (the client) emits and forwards them to stderr
server.LogSink = func(code uint8, text string) {
fmt.Fprint(os.Stderr, text)
}
// the library has no listener of its own -- the caller owns the
// transport: listen on an ephemeral loopback port and hand every
// accepted connection to [Server.HandleConnection] in its own goroutine
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
panic(err)
}
defer ln.Close()
go func() {
for {
conn, err := ln.Accept()
if err != nil {
return
}
go func() {
_ = server.HandleConnection(conn)
_ = conn.Close()
}()
}
}()
addr := ln.Addr().String()
// connect and pull a file
client := Client{
Module: "data",
ConnectFunc: func(string) (io.ReadWriter, error) {
return net.Dial("tcp", addr)
},
}
session, err := client.Connect(nil)
if err != nil {
panic(err)
}
f, err := session.Open("hello.txt")
if err != nil {
panic(err)
}
data, err := io.ReadAll(f)
if err != nil {
panic(err)
}
fmt.Println(string(data))
Output: hello, rsync!
Example (Auth) ¶
ExampleServer_auth serves a module whose ServerModule.AuthCallback requires the "zerocool" username -- the rsync secrets-file flow, where the server generates a random challenge and the callback verifies the digest(password + challenge) the client responds with.
// a module that requires the "zerocool" username: the callback gets
// the username the client sent and the challenge the server
// generated, and returns the expected digest bytes to compare against
// the client's response ("md5" being the auth digest the default
// greetings negotiate at the current protocol version)
server, err := NewServer(&ServerModule{
Name: "data",
Comment: "the example data",
FS: fstest.MapFS{
"hello.txt": &fstest.MapFile{Data: []byte("hello, rsync!")},
},
AuthCallback: func(username string, challenge []byte) ([]byte, error) {
if username != "zerocool" {
return nil, fmt.Errorf("unknown user %q", username)
}
sum := md5.Sum(append([]byte("hack the planet"), challenge...))
return sum[:], nil
},
})
if err != nil {
panic(err)
}
// serve the modules over loopback TCP (see ExampleServer)
addr, stop := startExampleServer(server)
defer stop()
// a client answering the challenge with [PasswordAuth] pulls the file
client := Client{
Module: "data",
AuthUser: "zerocool",
AuthResponse: PasswordAuth("hack the planet"),
ConnectFunc: func(string) (io.ReadWriter, error) {
return net.Dial("tcp", addr)
},
}
session, err := client.Connect(nil)
if err != nil {
panic(err)
}
f, err := session.Open("hello.txt")
if err != nil {
panic(err)
}
data, err := io.ReadAll(f)
if err != nil {
panic(err)
}
fmt.Println(string(data))
Output: hello, rsync!
func NewServer ¶
func NewServer(mods ...*ServerModule) (*Server, error)
NewServer creates a new rsync daemon server with the provided modules. Returns an error if any two modules share the same name.
func (*Server) HandleConnection ¶
func (s *Server) HandleConnection(rw io.ReadWriter) error
HandleConnection runs the full rsync daemon protocol on a single connection, sequentially: the pre-transfer handshake (greeting, module selection, auth, arguments, compat flags, algorithm negotiation, checksum seed), then the transfer phase in [(*Server).doServerSender].
The rw is the underlying transport (TCP socket, pipe, etc). The call blocks until the connection is complete; it returns nil for clean disconnects and #list requests, or an error that aborted the protocol.
type ServerModule ¶
type ServerModule struct {
Name string // module name
Comment string // displayed in #list
FS fs.FS // backing filesystem
ReadOnly bool // true = reject push operations
// AuthCallback verifies a username+challenge response for this module.
// Returns the expected raw digest bytes, or an error to reject.
// Nil means no authentication required for this module.
// Matches rsync's per-module secrets file model.
AuthCallback func(username string, challenge []byte) ([]byte, error)
}
ServerModule wraps a backing filesystem with rsync module configuration.
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session holds an active connection to an rsync daemon, ready for FS operations. In root mode (Module == ""), the Session is a config holder -- connectFunc creates fresh connections for each operation.
Session is not safe for concurrent use. The rsync protocol is sequential: selectors are sent one-at-a-time through a single-phase loop, and the compressed NDX encoder maintains shared delta state.
func (*Session) Lstat ¶
Lstat implements fs.ReadLinkFS: it returns a fs.FileInfo for the named entry without following symlinks, a complete operation in its own right.
func (*Session) Open ¶
Open implements fs.FS. In module mode (a Session from Client.Connect) it opens the named file or directory within the module; in root mode (a Session from Client.OpenRoot) the name routes to a module and each operation runs on its own connection.
A regular file open is a complete, self-contained rsync transfer: the connection's file list is read, one selector is sent, the data is pulled and verified, and the session is torn down (phase exchange, stats, final goodbye). The rsync protocol is a single batch session per connection, so the connection is consumed by the end of the open and the next open establishes a fresh one via Client.ConnectFunc (module mode) or the root-mode connect func.
Source Files
Directories