resync

README

Resync

Resync is a layer on top of rsync and cron that adds some helpful functionality.

Resync aims to be a very simple replacement for people that are already familiar with Rsync and Cron. All you need to do is create the YAML file and you're ready to go.

Why Resync?

Rsync with Cron is a popular pattern. If that's working for you then you might not have a use for Resync. However, I ran into a few pain points that I wrote resync to address.

• Skips the next cron invocation if the previous rsync command is still running
• An optional maximum time on how long each rsync command is allowed to run
• Stores and rotates logs from each rsync invocation
• Provides historical data on each rsync invocation
• Send email notifications for failures and sync history
• An optional HTTP server that provides health checks

How does it work?

Instead of using the system crontab resync runs cron jobs internally using the same syntax. A YAML file is used to configure resync and define how and when rsync is run.

1. Download the latest release.
2. Create a YAML configuration file
3. Run it resync -conf resync.yaml

Configuration file

The YAML file defines how and when each rsync command is run.

Minimal config example

Just define the syncs you'd like to perform and use the defaults.

syncs:
  backup:
    rsync_args: -a
    rsync_source:
      - /home/user/
    rsync_destination: /mnt/backup/
    schedule: 0 0 * * *

Full config example

rsync_path: rsync
log_path: /var/log/resync
log_level: error
lib_path: /var/lib/resync
time_format: Mon Jan 02 03:04:05 PM MST
retention: 7
seconds_field: false
time_limit: 5h
http:
  addr: 127.0.0.1
  port: 4050
email:
  host: smtp.myserver.com
  port: 587
  user: myuser
  pass: mypass
  starttls: true
  insecure_skip_verify: false
  ssl: false
  from: me@myserver.com
  to:
    - user1@myserver.com
    - user2@myserver.com
  history_subject: Resync History
  history_schedule: "* * * * *"
  history_template: "/etc/resync/resync.tmpl"
  on_failure: false
syncs:
  data:
    rsync_args: -a
    rsync_source:
      - /data/
    rsync_destination: /mnt/backup/data/
    schedule: "0 0 * * *"
  data2:
    rsync_args: -a --stats
    rsync_source:
      - /other data/
    rsync_destination: /mnt/backup/data2/
    schedule: "0 2 * * *"

Global Options

rsync_path - Path to the rsync binary. Defaults to rsync or rsync.exe on Windows.

log_path - Directory on disk where resync logs will be stored. Defaults to /var/log/resync.

log_level - Sets the log level. Valid levels are: panic, fatal, trace, debug, warn, info, and error. Defaults to error.

lib_path - Directory on disk where resync lib files will be stored. Defaults to /var/lib/resync.

time_format - The time format used when displaying sync stats. See formatting options in the go time.Time package. Defaults to Mon Jan 02 03:04:05 PM MST

retention - The number of logs and stats that are stored for each sync. Defaults to 7.

seconds_field - Enable the cron seconds field. This makes the first field in the cron expression handle seconds changes the expression to 6 fields. Defaults to false.

time_limit - The maximum amount of time that a sync job will run before being killed. TimeLimit must be a string that can be passed to the time.Duration.ParseDuration() function. Default is no time limit.

HTTP

addr - The listening address used for the optional internal healthcheck http server. Defaults to 127.0.0.1.

port - The listening port used for the optional internal healthcheck http server. Defaults to 4050.

Email

host - The hostname or IP of the SMTP server.

port - The port of the SMTP server.

user - The username used to authenticate.

pass - The password used to authenticate.

start_tls - StartTLS enables TLS security. If both StartTLS and SSL are true then StartTLS will be used.

insecure_skip_verify - When using TLS skip verifying the server's certificate chain and host name.

ssl - SSL enables SSL security. If both StartTLS and SSL are true then StartTLS will be used.

from - The email address the email will be sent from.

to - An array of email addresses for which emails will be sent.

history_email - Optional subject to use when sending sync history emails.

history_schedule - Defines a cron expression used to send scheduled reports. If set then an email with sync history will be sent based on the schedule.

history_template - Optional template to use when sending history emails. See go's html/template for details. Uses the default template if blank.

on_failure - Send an email for each sync failure if true.

Syncs

rsync_args - The arguments used when calling rsync.

rsync_source - An array of source paths used when calling rsync.

rsync_destination -The desintation used when calling rsync.

time_limit - The maximum amount of time that a sync job will run before being killed. TimeLimit must be a string that can be passed to the time.Duration.ParseDuration() function. Default is no time limit.

Flags

-conf - Path to the resync configuration file

-debug - Log to STDOUT

HTTP Health Checks

The optional HTTP server creates two endpoints.

/live - A liveness check that always returns 200.

/health - A health check that returns 200 if the latest run for each sync was successful and 503 otherwise.

Road Map

• Docker Image
• Systemd service file
• Create rpm
• Create deb
• More persistence layers
• More notifiers
• More logggers

Thanks

Jayr Elpedes

Documentation

Index

• func Bool(v bool) *bool
• func BoolValue(v *bool) bool
• func Int(v int) *int
• func IntValue(v *int) int
• func String(v string) *string
• func StringValue(v *string) string
• type BoltDB
  • func NewBoltDB(config *Config) (*BoltDB, error)
  • func (s *BoltDB) Close() error
  • func (s *BoltDB) Insert(stat Stat) error
  • func (s *BoltDB) List() (map[string][]Stat, error)
  • func (s *BoltDB) Prune() error
• type Config
  • func OpenConfig(path string) (*Config, error)
  • func (c *Config) GetSync(name string) (*Sync, error)
  • func (c *Config) GetTimeLimit(name string) (time.Duration, error)
• type DB
• type Email
• type EmailNotifier
  • func NewEmailNotifier(config *Config, db DB, logger Logger) *EmailNotifier
  • func (m *EmailNotifier) Notify(stat Stat) error
  • func (m *EmailNotifier) NotifyHistory() error
• type FSLogger
  • func NewFSLogger(config *Config) *FSLogger
  • func (l *FSLogger) Rotate(name string) (io.WriteCloser, io.WriteCloser, error)
  • func (l *FSLogger) Stderr(name string) (io.ReadCloser, error)
  • func (l *FSLogger) Stdout(name string) (io.ReadCloser, error)
  • func (l *FSLogger) Zip() (*os.File, error)
• type HTTP
• type Logger
• type Notifier
• type Resync
  • func New(config *Config, db DB, logger Logger, notifier Notifier) *Resync
  • func (re *Resync) Dump() error
  • func (re *Resync) Start() error
  • func (re *Resync) Stop()
• type Stat
  • func NewStat(name string, format string) Stat
  • func (s Stat) Finish(err error) Stat
• type Sync
  • func (s *Sync) Args() []string

Functions

func Bool

func Bool(v bool) *bool

Bool returns a pointer to the bool value passed in.

func BoolValue

func BoolValue(v *bool) bool

BoolValue returns the value of the string pointer passed in or false if the pointer is nil.

func Int

func Int(v int) *int

Int returns a pointer to the int value passed in.

func IntValue

func IntValue(v *int) int

IntValue returns the value of the string pointer passed in or 0 if the pointer is nil.

func String

func String(v string) *string

String returns a pointer to the string value passed in.

func StringValue

func StringValue(v *string) string

StringValue returns the value of the string pointer passed in or "" if the pointer is nil.

Types

type BoltDB

type BoltDB struct {
	// contains filtered or unexported fields
}

BoltDB is the default and only database for storing stats. In the future other databases could be added.

func NewBoltDB

func NewBoltDB(config *Config) (*BoltDB, error)

NewBoltDB creates the underlying boltdb database. If retention is less than 1 than the database isn't created and all calls are no-ops.

func (*BoltDB) Close

func (s *BoltDB) Close() error

Close closes the bolt database file.

func (*BoltDB) Insert

func (s *BoltDB) Insert(stat Stat) error

Insert adds one Stat to bolt.

func (*BoltDB) List

func (s *BoltDB) List() (map[string][]Stat, error)

List returns all stats stored as a map. The map keys are sync names and the values are a list of all stored stats for. that sysnc. Stats are returned storted by Start in descending order.

func (*BoltDB) Prune

func (s *BoltDB) Prune() error

Prune removes all entries for a Sync job that exceed the rentention config.

type Config

type Config struct {
	// RsyncPath is the path to the rsync binary. Defaults to rsync or rsync.exe on Windows.
	RsyncPath *string `yaml:"rsync_path"`

	// LogPath is the directory on disk where resync logs will be stored. Defaults to /var/log/resync.
	LogPath *string `yaml:"log_path"`

	// LogLevel sets the level of logging. Valid levels are: panic, fatal, trace, debug, warn, info, and error. Defaults to error
	LogLevel *string `yaml:"log_level"`

	// LibPath is the directory on disk where resync lib files are stored. Defaults to /var/lib/resync.
	LibPath *string `yaml:"lib_path"`

	// The time format used when displaying sync stats. See formatting options in the go time.Time package.
	// Defaults to Mon Jan 02 03:04:05 PM MST
	TimeFormat *string `yaml:"time_format"`

	// Retention is the number of logs and stats that are stored for each sync. If set to less than 1 no
	// logs or are stats are saved. Defaults to 7.
	Retention *int `yaml:"retention"`

	// Enable the cron seconds field. This makes the first field in the cron expression handle seconds
	// changes the expression to 6 fields. Defaults to false.
	SecondsField *bool `yaml:"seconds_field"`

	// TimeLimit is the maximum amount of time that a sync job will run before being killed. TimeLimit
	// must be a string that can be passed to the time.Duration.ParseDuration() function. Default is no
	// time limit.
	TimeLimit *string `yaml:"time_limit"`

	HTTP  *HTTP            `yaml:"http"`
	Email *Email           `yaml:"email"`
	Syncs map[string]*Sync `yaml:"syncs"`
	// contains filtered or unexported fields
}

Config is an object representation of the YAML configuration file. Config is read from in multiple goroutines and should not be written after the NewConfig is called or data races will happen. Config is not thread safe and should be considered read only after being passed to any other object. Writing any of it's fields will cause a data race.

func OpenConfig

func OpenConfig(path string) (*Config, error)

OpenConfig returns a new Config option by reading the YAML file at path. If the file doesn't exist, can't be read, is invalid YAML, or doesn't match the resync spec then an error is returned.

func (*Config) GetSync

func (c *Config) GetSync(name string) (*Sync, error)

GetSync returns the Sync object by name. Otherwise it returns an error.

func (*Config) GetTimeLimit

func (c *Config) GetTimeLimit(name string) (time.Duration, error)

GetTimeLimit returns the time limit for name if it exists. If it doesn't exist it then tries to return the global time limit. If that also doesn't exist then this method returns an error.

type DB

type DB interface {
	// Prune removes old stats based on the configured retention value.
	Prune() error

	// List returns all stats stored as a map. The map keys are sync names and the values are a list of all stored stats for.
	//that sysnc. Stats should be returned storted by Start in descending order.
	List() (map[string][]Stat, error)

	// Insert adds a stat to the database.
	Insert(Stat) error

	// Closes the connection the database
	Close() error
}

DB defines an interface for persisting Stats. Stats are simple metrics for each job.

type Email

type Email struct {
	// Host is the hostname or IP of the SMTP server.
	Host *string `yaml:"host"`

	// Port is the port of the SMTP server.
	Port *int `yaml:"port"`

	// User is the username used to authenticate.
	User *string `yaml:"user"`

	// Pass is the password used to authenticate.
	Pass *string `yaml:"pass"`

	// StartTLS enables TLS security. If both StartTLS and SSL are true then StartTLS will be used.
	StartTLS *bool `yaml:"starttls"`

	// Skip verifying the server's certificate chain and host name.
	InsecureSkipVerify *bool `yaml:"insecure_skip_verify"`

	// SSL enables SSL security. If both StartTLS and SSL are true then StartTLS will be used.
	SSL *bool `yaml:"ssl"`

	// From is the email address the email will be sent from.
	From *string `yaml:"from"`

	// To is an array of email addresses for which emails will be sent.
	To []string `yaml:"to"`

	// HistorySubject is an optional subject to use when sending sync history emails.
	HistorySubject *string `yaml:"history_subject"`

	// HistorySchedule is a cron expression. If set then an email with sync history will be sent based on the schedule.
	HistorySchedule *string `yaml:"history_schedule"`

	// HistoryTemplate is an optional path to an email template to use when sending history emails. If not set uses the default template.
	HistoryTemplate *string `yaml:"history_template"`

	// OnFailure will send an email for each sync failure if true.
	OnFailure *bool `yaml:"on_failure"`
}

Email defines the SMTP configuration options needed when sending email notifications.

type EmailNotifier

type EmailNotifier struct {
	// contains filtered or unexported fields
}

EmailNotifier sends emails based on stats saved in the DB.

func NewEmailNotifier

func NewEmailNotifier(config *Config, db DB, logger Logger) *EmailNotifier

NewEmailNotifier creates a Mailer using config, db, and logger.

func (*EmailNotifier) Notify

func (m *EmailNotifier) Notify(stat Stat) error

Notify sends a single email for the stat based on the configured email settings.

func (*EmailNotifier) NotifyHistory

func (m *EmailNotifier) NotifyHistory() error

NotifyHistory sends status email for the stats in the db based on the configured email settings.

type FSLogger

type FSLogger struct {
	// contains filtered or unexported fields
}

FSLoggger persists log files to a file system based on the LogPath config options. Beware that errors can occur when using the returned io.ReadClosers from Stderr and Stdout and calling Rotate.

func NewFSLogger

func NewFSLogger(config *Config) *FSLogger

Creates an FSLogger with the given config.

func (*FSLogger) Rotate

func (l *FSLogger) Rotate(name string) (io.WriteCloser, io.WriteCloser, error)

Rotates the log files and keeps old logs based on the configured retention value.

func (*FSLogger) Stderr

func (l *FSLogger) Stderr(name string) (io.ReadCloser, error)

Stderr returns the last STDERR from the rsync command. Expect errors Rotate is called while stderr is still open.

func (*FSLogger) Stdout

func (l *FSLogger) Stdout(name string) (io.ReadCloser, error)

Stdout returns the last STDOUT from the rsync command. Expect errors Rotate is called while stdout is still open.

func (*FSLogger) Zip

func (l *FSLogger) Zip() (*os.File, error)

Zip returns file that contains all of the stored logs for all syncs. Expect errors Rotate is called while the zip is being created.

type HTTP

type HTTP struct {
	// The address the http server will listen on.
	Addr *string `yaml:"addr"`

	// The port the http server will listen on.
	Port *int `yaml:"port"`
}

HTTP defines the configuration for http health checks.

type Logger

type Logger interface {
	// Rotates the log files and keeps old logs based on the configured retention value.
	Rotate(string) (io.WriteCloser, io.WriteCloser, error)

	// Stdout returns the last STDOUT from the rsync command.
	Stdout(string) (io.ReadCloser, error)

	// Stderr returns the last STDERR from the rsync command.
	Stderr(string) (io.ReadCloser, error)

	// Zip returns file that contains all of the stored logs for all syncs.
	Zip() (*os.File, error)
}

Logger defines an interface for persisting log files. Each rsync command logs the processes output from both STDOUT and STDERR.

type Notifier

type Notifier interface {
	// Notify sends a notification about a single stat.
	Notify(Stat) error

	// Notify History sends a notification with the sync history.
	NotifyHistory() error
}

Notifier defines an interface for sending notifications.

type Resync

type Resync struct {
	// contains filtered or unexported fields
}

Resync is responsible for running rsync commands based on cron schedules.

func New

func New(config *Config, db DB, logger Logger, notifier Notifier) *Resync

New creates a new Resync object.

func (*Resync) Dump

func (re *Resync) Dump() error

Dump prints all of the stats to STDOUT.

func (*Resync) Start

func (re *Resync) Start() error

Start sets up and runs the configured cron jobs.

func (*Resync) Stop

func (re *Resync) Stop()

Stop stops running cron jobs, closes the db, and kills all running sync jobs.

type Stat

type Stat struct {
	Name     string
	Success  bool
	Start    string
	End      string
	Duration time.Duration
	// contains filtered or unexported fields
}

Stat defines basic statistics for a single sync. Stats are stored so that historical data from past syncs can be viewed.

func NewStat

func NewStat(name string, format string) Stat

NewStat creates a new Stat with Name set to name, Success set to false, and Start set to the current time.

func (Stat) Finish

func (s Stat) Finish(err error) Stat

Finish sets the Success based on err, End based on the current time, and Duration based on Start and End.

type Sync

type Sync struct {
	// RsyncArgs are all of the arguments needed to run your rsync command
	RsyncArgs *string `yaml:"rsync_args"`

	// RsyncSource is the location of the rsync command's source
	RsyncSource []string `yaml:"rsync_source"`

	// RsyncDestination is the location of the rsync command's destination
	RsyncDestination *string `yaml:"rsync_destination"`

	// Schedule is the cron expresion for this sync.
	Schedule *string `yaml:"schedule"`

	// TimeLimit is the maximum amount of time that a sync job will run before being killed. TimeLimit
	// must be a string that can be passed to the time.Duration.ParseDuration() function.
	TimeLimit *string `yaml:"time_limit"`
	// contains filtered or unexported fields
}

Sync defines a single rsync command, cron expression, and other related options.

func (*Sync) Args

func (s *Sync) Args() []string

Args returns a list of args suitable for exec.Command.