README
¶
s
zrepl
zrepl is a one-stop ZFS backup & replication solution.
User Documentation
User Documentation can be found at zrepl.github.io.
Bug Reports
- If the issue is reproducible, enable debug logging, reproduce and capture the log.
- Open an issue on GitHub, with logs pasted as GitHub gists / inline.
Feature Requests
- Does your feature request require default values / some kind of configuration? If so, think of an expressive configuration example.
- Think of at least one use case that generalizes from your concrete application.
- Open an issue on GitHub with example conf & use case attached.
The above does not apply if you already implemented everything. Check out the Coding Workflow section below for details.
Development
zrepl is written in Go and uses Go modules to manage dependencies. The documentation is written in ReStructured Text using the Sphinx framework.
Building
Go Code
Dependencies:
- Go 1.22 or newer
- GNU Make
- Git
- wget (
make generate) - unzip (
make generate)
Some Go code is generated, and generated code is committed to the source tree.
Therefore, building does not require having code generation tools set up.
When making changes that require code to be (re-)generated, run make generate.
I downloads and installs pinned versions of the code generation tools into ./build/install.
There is a CI check that ensures Git state is clean, i.e., code generation has been done by a PR and is deterministic.
Docs
Set up a Python environment that has docs/requirements.txt installed via pip.
Use a venv to avoid global state.
Testing
The test suite is split into pure Go tests (make test-go) and platform tests that interact with ZFS and thus generally require root privileges (sudo make test-platform).
Platform tests run on their own pool with the name zreplplatformtest, which is created using the file vdev in /tmp.
For a full code coverage profile, run make test-go COVER=1 && sudo make test-platform && make cover-merge.
An HTML report can be generated using make cover-html.
Circle CI
We use CircleCI for automated build & test pre- and post-merge.
There are two workflows:
-
ciruns for every commit / branch / tag pushed to GitHub. It is supposed to run very fast (<5min and provides quick feedback to developers). It runs formatting checks, lints and tests on the most important OSes / architectures. -
releaseruns- on manual triggers through the CircleCI API (in order to produce a release)
- periodically on
master
Artifacts are stored in CircleCI.
Releasing
All zrepl releases are git-tagged and then published as a GitHub Release.
There is a git tag for each zrepl release, usually vMAJOR.MINOR.0.
We don't move git tags once the release has been published.
The procedure to issue a release is as follows:
- Issue the source release:
- Git tag the release on the
masterbranch. - Push the tag.
- Run
./docs/publish.shto re-build & push zrepl.github.io.
- Git tag the release on the
- Issue the official binary release:
- Run the
releasepipeline (triggered via CircleCI API) - Download the artifacts to the release manager's machine.
- Create a GitHub release, edit the changelog, upload all the release artifacts, including .rpm and .deb files.
- Issue the GitHub release.
- Add the .rpm and .deb files to the official zrepl repos.
- Code for management of these repos: https://github.com/zrepl/package-repo-ops (private repo at this time)
- Run the
Patch releases, Go toolchain updates, APT/RPM Package rebuilds
vMAJOR.MINOR.0 is typically a tagged commit on master, because development velocity isn't high
and thus release branches for stabilization aren't necessary.
Occasionally though there is a need for patch changes to a release, e.g.
- security issue in a dependency
- Go toolchain update (e.g. security issue in standard library)
The procedure for this is the following
- create a branch off the release tag we need to patch, named
releases/MAJOR.MINOR.X - that branch will never be merged into
master, it'll be a dead-end for this specific patch - make changes in that branch
- make the final commit that bumps version numbers
- create the git tag
- follow general procedure for publishing the release (previous sectino
For Go toolchain updates and package rebuilds with no source code changes, leverage the APT/RPM package revision field.
Control via the ZREPL_PACKAGE_RELEASE Makefile variable, see origin/releases/0.6.1-2 for an example.
Notes to Distro Package Maintainers
- The
Makefilein this project is not suitable for builds in distros. - Run the platform tests (Docs -> Usage -> Platform Tests) on a test system to validate that zrepl's abstractions on top of ZFS work with the system ZFS.
- Ship a default config that adheres to your distro's
hierand logging system. - Ship a service manager file and please try to upstream it to this repository.
dist/systemdcontains a Systemd unit template.
- Ship other material provided in
./dist, e.g. in/usr/share/zrepl/. - Have a look at the
Makefile'sZREPL_VERSIONvariable and how it passed to Go'sldFlags. This is howzrepl versionknows what version number to show. Your build system should set theldFlagsflags appropriately and add a prefix or suffix that indicates that the given zrepl binary is a distro build, not an official one. - Make sure you are informed about new zrepl versions, e.g. by subscribing to GitHub's release RSS feed.
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
daemon/hooks
Package hooks implements pre- and post snapshot hooks.
|
Package hooks implements pre- and post snapshot hooks. |
|
daemon/logging/trace
package trace provides activity tracing via ctx through Tasks and Spans
|
package trace provides activity tracing via ctx through Tasks and Spans |
|
endpoint
Package endpoint implements replication endpoints for use with package replication.
|
Package endpoint implements replication endpoints for use with package replication. |
|
platformtest/harness
command
|
|
|
platformtest/tests/gen
command
|
|
|
replication
Package replication implements replication of filesystems with existing versions (snapshots) from a sender to a receiver.
|
Package replication implements replication of filesystems with existing versions (snapshots) from a sender to a receiver. |
|
rpc
Package rpc implements zrepl daemon-to-daemon RPC protocol on top of a transport provided by package transport.
|
Package rpc implements zrepl daemon-to-daemon RPC protocol on top of a transport provided by package transport. |
|
rpc/dataconn/microbenchmark
command
microbenchmark to manually test rpc/dataconn performance
|
microbenchmark to manually test rpc/dataconn performance |
|
rpc/dataconn/timeoutconn
package timeoutconn wraps a Wire to provide idle timeouts based on Set{Read,Write}Deadline.
|
package timeoutconn wraps a Wire to provide idle timeouts based on Set{Read,Write}Deadline. |
|
rpc/dataconn/timeoutconn/internal/wireevaluator
command
a tool to test whether a given transport implements the timeoutconn.Wire interface
|
a tool to test whether a given transport implements the timeoutconn.Wire interface |
|
rpc/grpcclientidentity
Package grpcclientidentity makes the client identity provided by github.com/LyingCak3/zrepl/daemon/transport/serve.{AuthenticatedListener,AuthConn} available to gRPC service handlers.
|
Package grpcclientidentity makes the client identity provided by github.com/LyingCak3/zrepl/daemon/transport/serve.{AuthenticatedListener,AuthConn} available to gRPC service handlers. |
|
rpc/grpcclientidentity/example
command
This package demonstrates how the grpcclientidentity package can be used to set up a gRPC greeter service.
|
This package demonstrates how the grpcclientidentity package can be used to set up a gRPC greeter service. |
|
rpc/grpcclientidentity/grpchelper
Package grpchelper wraps the adaptors implemented by package grpcclientidentity into a less flexible API which, however, ensures that the individual adaptor primitive's expectations are met and hence do not panic.
|
Package grpchelper wraps the adaptors implemented by package grpcclientidentity into a less flexible API which, however, ensures that the individual adaptor primitive's expectations are met and hence do not panic. |
|
rpc/netadaptor
Package netadaptor implements an adaptor from transport.AuthenticatedListener to net.Listener.
|
Package netadaptor implements an adaptor from transport.AuthenticatedListener to net.Listener. |
|
rpc/transportmux
Package transportmux wraps a transport.{Connecter,AuthenticatedListener} to distinguish different connection types based on a label sent from client to server on connection establishment.
|
Package transportmux wraps a transport.{Connecter,AuthenticatedListener} to distinguish different connection types based on a label sent from client to server on connection establishment. |
|
rpc/versionhandshake
Package versionhandshake wraps a transport.{Connecter,AuthenticatedListener} to add an exchange of protocol version information on connection establishment.
|
Package versionhandshake wraps a transport.{Connecter,AuthenticatedListener} to add an exchange of protocol version information on connection establishment. |
|
transport
Package transport defines a common interface for network connections that have an associated client identity.
|
Package transport defines a common interface for network connections that have an associated client identity. |
|
transport/fromconfig
Package fromconfig instantiates transports based on zrepl config structures (see package config).
|
Package fromconfig instantiates transports based on zrepl config structures (see package config). |
|
util/chainlock
package chainlock implements a mutex whose Lock and Unlock methods return the lock itself, to enable chaining.
|
package chainlock implements a mutex whose Lock and Unlock methods return the lock itself, to enable chaining. |
|
util/choices
Package choice implements a flag.Value type that accepts a set of choices.
|
Package choice implements a flag.Value type that accepts a set of choices. |
|
util/devnoop
package devnoop provides an io.ReadWriteCloser that never errors and always reports reads / writes to / from buffers as complete.
|
package devnoop provides an io.ReadWriteCloser that never errors and always reports reads / writes to / from buffers as complete. |
|
util/nodefault
Package nodefault provides newtypes around builtin Go types so that if the newtype is used as a field in a struct and that field is zero-initialized (https://golang.org/ref/spec#The_zero_value), accessing the field will cause a null pointer deref panic.
|
Package nodefault provides newtypes around builtin Go types so that if the newtype is used as a field in a struct and that field is zero-initialized (https://golang.org/ref/spec#The_zero_value), accessing the field will cause a null pointer deref panic. |
|
zfs/zfscmd
Package zfscmd provides a wrapper around packate os/exec.
|
Package zfscmd provides a wrapper around packate os/exec. |