README
¶
Resync
Resync is a layer on top of rsync and cron that adds some helpful functionality.
Resync aims to be a very simple replacement for people that are already familiar with Rsync and Cron. All you need to do is create the YAML file and you're ready to go.
Why Resync?
Rsync with Cron is a popular pattern. If that's working for you then you might not have a use for Resync. However, I ran into a few pain points that I wrote resync to address.
- Skips the next cron invocation if the previous rsync command is still running
- An optional maximum time on how long each rsync command is allowed to run
- Stores and rotates logs from each rsync invocation
- Provides historical data on each rsync invocation
- Send email notifications for failures and reports
How does it work?
Instead of using the system crontab resync runs cron jobs internally using the same syntax. A YAML file is used to configure resync and define how and when rsync is run.
- Download the binary
- Create a YAML file
- Run it
resync -conf resync.yaml
Configuration file
The YAML file defines how and when each rsync command is run.
Minimal config example
Just define the syncs you'd like to perform and use the defaults.
syncs:
backup:
command: -a /home/user /mnt/backup
schedule: 0 0 * * *
Full config example
rsync_path: rsync
log_path: /var/log/resync
log_level: error
lib_path: /var/lib/resync
time_format: Mon Jan 02 03:04:05 PM MST
retention: 7
seconds_field: false
time_limit: 0
email:
host: smtp.myserver.com
port: 587
user: myuser
pass: mypass
starttls: true
ssl: false
from: me@myserver.com
to:
- user1@myserver.com
- user2@myserver.com
schedule: "* * * * * *"
on_failure: false
syncs:
data:
command: -a /data/ /mnt/backup/data/
schedule: "0 0 * * *"
data2:
command: -a /data2/ /mnt/backup/data2/
schedule: "0 2 * * *"
Global Options
rsync_path
Path to the rsync binary. Defaults to rsync or rsync.exe on Windows.
log_path
Directory on disk where resync logs will be stored. Defaults to /var/log/resync.
log_level
Sets the log level. Valid levels are: panic, fatal, trace, debug, warn, info, and error. Defaults to error.
lib_path
Directory on disk where resync lib files will be stored. Defaults to /var/lib/resync.
time_format
The time format used when displaying sync stats. See formatting options in the go time.Time package. Defaults to Mon Jan 02 03:04:05 PM MST
retention
The number of logs and stats that are stored for each sync. Defaults to 7.
seconds_field
Enable the cron seconds field. This makes the first field in the cron expression handle seconds changes the expression to 6 fields. Defaults to false.
time_limit
The maximum amount of time that a sync job will run before being killed. TimeLimit must be a string that can be passed to the time.Duration.ParseDuration() function. Default is no time limit.
host
The hostname or IP of the SMTP server.
port
The port of the SMTP server.
user
The username used to authenticate.
pass
The password used to authenticate.
start_tls
StartTLS enables TLS security. If both StartTLS and SSL are true then StartTLS will be used.
ssl
SSL enables SSL security. If both StartTLS and SSL are true then StartTLS will be used.
from
The email address the email will be sent from.
to
An array of email addresses for which emails will be sent.
schedule
Defines a cron expression used to send scheduled reports. If set then an email with sync history will be sent based on the schedule.
on_failure
Send an email for each sync failure if true.
Syncs
command
The rsync command that will be run. It should be idential to an rsync command on the command line with just the rsync command itself omitted.
schedule
The cron expression that will be used. It should be quoted as * have a special meaning in YAML files.
time_limit
The maximum amount of time that a sync job will run before being killed. TimeLimit must be a string that can be passed to the time.Duration.ParseDuration() function. Default is no time limit.
Flags
-conf
Path to the resync configuration file
-debug
Log to STDOUT
Road Map
- Docker Image
- Systemd service file
- Create rpm
- Create deb
- More persistence and logging layers
Documentation
¶
Index ¶
- func Bool(v bool) *bool
- func BoolValue(v *bool) bool
- func Float64(v float64) *float64
- func Float64Value(v *float64) float64
- func Int(v int) *int
- func Int64(v int64) *int64
- func Int64Value(v *int64) int64
- func IntValue(v *int) int
- func String(v string) *string
- func StringValue(v *string) string
- func Uint16(v uint16) *uint16
- func Uint16Value(v *uint16) uint16
- func Uint64(v uint64) *uint64
- func Uint64Value(v *uint64) uint64
- type BoltDB
- type Config
- type DB
- type Email
- type FSLogger
- type Logger
- type Mailer
- type Resync
- type Stat
- type Sync
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BoolValue ¶
BoolValue returns the value of the string pointer passed in or false if the pointer is nil.
func Float64Value ¶
Float64Value returns the value of the string pointer passed in or 0 if the pointer is nil.
func Int64Value ¶
Int64Value returns the value of the string pointer passed in or 0 if the pointer is nil.
func IntValue ¶
IntValue returns the value of the string pointer passed in or 0 if the pointer is nil.
func StringValue ¶
StringValue returns the value of the string pointer passed in or "" if the pointer is nil.
func Uint16Value ¶
Uint16Value returns the value of the string pointer passed in or 0 if the pointer is nil.
func Uint64Value ¶
Uint64Value returns the value of the string pointer passed in or 0 if the pointer is nil.
Types ¶
type BoltDB ¶
type BoltDB struct {
// contains filtered or unexported fields
}
BoltDB is the default and only database for storing stats. In the future other databases could be added.
func NewBoltDB ¶
NewBoltDB creates the underlying boltdb database. If retention is less than 1 than the database isn't created and all calls are no-ops.
type Config ¶
type Config struct {
// RsyncPath is the path to the rsync binary. Defaults to rsync or rsync.exe on Windows.
RsyncPath *string `yaml:"rsync_path"`
// LogPath is the directory on disk where resync logs will be stored. Defaults to /var/log/resync.
LogPath *string `yaml:"log_path"`
// LogLevel sets the level of logging. Valid levels are: panic, fatal, trace, debug, warn, info, and error. Defaults to error
LogLevel *string `yaml:"log_level"`
// LibPath is the directory on disk where resync lib files are stored. Defaults to /var/lib/resync.
LibPath *string `yaml:"lib_path"`
// The time format used when displaying sync stats. See formatting options in the go time.Time package.
// Defaults to Mon Jan 02 03:04:05 PM MST
TimeFormat *string `yaml:"time_format"`
// Retention is the number of logs and stats that are stored for each sync. If set to less than 1 no
// logs or are stats are saved. Defaults to 7.
Retention *int `yaml:"retention"`
// Enable the cron seconds field. This makes the first field in the cron expression handle seconds
// changes the expression to 6 fields. Defaults to false.
SecondsField *bool `yaml:"seconds_field"`
// TimeLimit is the maximum amount of time that a sync job will run before being killed. TimeLimit
// must be a string that can be passed to the time.Duration.ParseDuration() function. Default is no
// time limit.
TimeLimit *string `yaml:"time_limit"`
Email *Email `yaml:"email"`
Syncs map[string]*Sync `yaml:"syncs"`
// contains filtered or unexported fields
}
Config is an object representation of the YAML configuration file. Config is read from in multiple goroutines and should not be written after the NewConfig is called or data races will happen. Config is not thread safe and should be considered read only after being passed to any other object. Writing any of it's fields will cause a data race.
func OpenConfig ¶
OpenConfig returns a new Config option by reading the YAML file at path. If the file doesn't exist, can't be read, is invalid YAML, or doesn't match the resync spec then an error is returned.
type DB ¶
type DB interface {
// Prune removes old stats based on the configured retention value.
Prune() error
// List returns all stats stored.
List() ([]Stat, error)
// Insert adds a stat to the database.
Insert(Stat) error
}
DB defines an interface for persisting Stats. Stats are simple metrics for each job.
type Email ¶
type Email struct {
// Host is the hostname or IP of the SMTP server.
Host *string `yaml:"host"`
// Port is the port of the SMTP server.
Port *int `yaml:"port"`
// User is the username used to authenticate.
User *string `yaml:"user"`
// Pass is the password used to authenticate.
Pass *string `yaml:"pass"`
// StartTLS enables TLS security. If both StartTLS and SSL are true then StartTLS will be used.
StartTLS *bool `yaml:"starttls"`
// SSL enables SSL security. If both StartTLS and SSL are true then StartTLS will be used.
SSL *bool `yaml:"ssl"`
// From is the email address the email will be sent from.
From *string `yaml:"from"`
// To is an array of email addresses for which emails will be sent.
To []string `yaml:"to"`
// Schedule is a cron expression. If set then an email with sync history will be sent based on the schedule.
Schedule *string `yaml:"schedule"`
// OnFailure will send an email for each sync failure if true.
OnFailure *bool `yaml:"on_failure"`
}
Email defines the SMTP configuration options needed when sending email notifications.
type FSLogger ¶
type FSLogger struct {
// contains filtered or unexported fields
}
FSLoggger persists log files to a file system based on the LogPath config options. Beware that errors can occur when using the returned io.ReadClosers from Stderr and Stdout and calling Rotate.
func NewFSLogger ¶
Creates an FSLogger with the given config.
func (*FSLogger) Rotate ¶
func (l *FSLogger) Rotate(name string) (io.WriteCloser, io.WriteCloser, error)
Rotates the log files and keeps old logs based on the configured retention value.
func (*FSLogger) Stderr ¶
func (l *FSLogger) Stderr(name string) (io.ReadCloser, error)
Stderr returns the last STDERR from the rsync command. Expect errors Rotate is called while stderr is still open.
type Logger ¶
type Logger interface {
// Rotates the log files and keeps old logs based on the configured retention value.
Rotate(string) (io.WriteCloser, io.WriteCloser, error)
// Stdout returns the last STDOUT from the rsync command.
Stdout(string) (io.ReadCloser, error)
// Stderr returns the last STDERR from the rsync command.
Stderr(string) (io.ReadCloser, error)
// Zip returns file that contains all of the stored logs for all syncs.
Zip() (*os.File, error)
}
Logger defines an interface for persisting log files. Each rsync command logs the processes output from both STDOUT and STDERR.
type Mailer ¶
type Mailer struct {
// contains filtered or unexported fields
}
Mailer sends emails based on stats saved in the DB.
type Resync ¶
type Resync struct {
// contains filtered or unexported fields
}
Resync is responsible for running rsync commands based on cron schedules.
type Stat ¶
type Stat struct {
Name string
Success bool
Start string
End string
Duration time.Duration
// contains filtered or unexported fields
}
Stat defines basic statistics for a single sync. Stats are stored so that historical data from past syncs can be viewed.
type Sync ¶
type Sync struct {
// Command is the rsync command that will be run. It should be idential to an rsync command on the command
// line with just the rsync command itself omitted.
Command *string `yaml:"command"`
// Schedule is the cron expresion for this sync.
Schedule *string `yaml:"schedule"`
// TimeLimit is the maximum amount of time that a sync job will run before being killed. TimeLimit
// must be a string that can be passed to the time.Duration.ParseDuration() function.
TimeLimit *string `yaml:"time_limit"`
// contains filtered or unexported fields
}
Sync defines a single rsync command, cron expression, and other related options.
Source Files
Directories