snappr

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 16, 2023 License: MIT Imports: 7 Imported by: 0

README

snappr

Go Reference

CLI and library for pruning time-based snapshots with a flexible retention policy.

Features

  • Standalone.
    Works with any tool or script which can output a list with dates somewhere in it.

  • Approximate snapshot selection.
    Snapshots periods are not fixed to specific dates. The first matching snapshot for each period is kept (note that this means you'll usually want to keep at least the last snapshot in addition to whatever other rules you have).

  • Robust retention policies.
    Multiple intervals are supported for each period (last, secondly, daily, monthly, yearly). You can have one snapshot every month for 6 months, while also having one every two for 12.

  • Flexible command line interface.
    Can extract dates in arbitrary formats from arbitrary parts of a line, preserving the entire line, and ignoring or passing-through unmatched or invalid lines.

  • Zone-aware timestamp handling.
    Will work correctly with different timezones, sorting and determining secondly snapshots by the real time, but using the calendar day/month/year from the zoned time.

  • Verbose debugging information.
    You can view which intervals caused a specific snapshot to be retained, and whether a retention policy wants more snapshots than it found.

[!WARNING] This tool is still very new. While most functionality has been tested and I am using this as part of my own backup scripts, it may still have rough edges, and the command-line interface and API are subject to change.

CLI Example

# install latest version
$ go install github.com/pgaskin/snappr/cmd/snappr@latest

# install from source
$ go install ./cmd/snappr

# testing a range of dates
$ seq 946684800 $((13+55*60*6)) 1735689600 |
  snappr -sw 1@last 7@daily 6@monthly 4@monthly:6 6@yearly 4@yearly:12 >/dev/null

# simple rsync+btrfs snapshots
$ rsync ... /mnt/bkp/cur/
$ btrfs subvol snap -r /mnt/bkp/cur/ /mnt/bkp/snap.$(date --utc +%Y%m%d-%H%M%S)
$ btrfs subvol list -r /mnt/bkp/ |
  snappr -sw \
    -e 'path snap\.([0-9-]{15})$' -Eqo \
    -p '20060102-150405' \
    1@last 12@secondly:1h 7@daily 4@daily:7 6@monthly 5@yearly yearly:10 |
  cut -d ' ' -f2- |
  xargs --no-run-if-empty btrfs subvolume delete

CLI Usage

usage: /tmp/go-build2822248938/b001/exe/snappr [options] policy...

options:
  -E, --extended-regexp     use full regexp syntax rather than POSIX (see pkg.go.dev/regexp/syntax)
  -e, --extract string      extract the timestamp from each input line using the provided regexp, which must contain up to one capture group
  -h, --help                show this help text
  -v, --invert              output the snapshots to keep instead of the ones to prune
  -o, --only                only print the part of the line matching the regexp
  -p, --parse string        parse the timestamp using the specified Go time format (see pkg.go.dev/time#pkg-constants and the examples below) rather than a unix timestamp
  -Z, --parse-timezone tz   use a specific timezone rather than whatever is set for --timezone if no timezone is parsed from the timestamp itself
  -q, --quiet               do not show warnings about invalid or unmatched input lines
  -s, --summarize           summarize retention policy results to stderr
  -z, --timezone tz         convert all timestamps to this timezone while pruning snapshots (use "local" for the default system timezone) (default UTC)
  -w, --why                 explain why each snapshot is being kept to stderr

time format examples:
  - Mon Jan 02 15:04:05 2006
  - 02 Jan 06 15:04 MST
  - 2006-01-02T15:04:05Z07:00
  - 2006-01-02T15:04:05

policy: N@unit:X
  - keep the last N snapshots every X units
  - omit the N@ to keep an infinite number of snapshots
  - if :X is omitted, it defaults to :1
  - there may only be one N specified for each unit:X pair

unit:
  last       snapshot count (X must be 1)
  secondly   clock seconds (can also use the format #h#m#s, omitting any zeroed units)
  daily      calendar days
  monthly    calendar months
  yearly     calendar years

notes:
  - output lines consist of filtered input lines
  - input is read from stdin, and should consist of unix timestamps (or more if --extract and/or --parse are set)
  - invalid/unmatched input lines are ignored, or passed through if --invert is set (and a warning is printed unless --quiet is set)
  - everything will still work correctly even if timezones are different
  - snapshots are always ordered by their real (i.e., UTC) time
  - if using --parse-in, beware of duplicate timestamps at DST transitions (if the offset isn't included whatever you use as the
    snapshot name, and your timezone has DST, you may end up with two snapshots for different times with the same name.
  - timezones will only affect the exact point at which calendar days/months/years are split

Library Example

var times []time.Time
// ...

var policy Policy
policy.MustSet(snappr.Yearly, 5, -1)
policy.MustSet(snappr.Yearly, 2, 10)
policy.MustSet(snappr.Yearly, 1, 3)
policy.MustSet(snappr.Monthly, 6, 4)
policy.MustSet(snappr.Monthly, 2, 6)
policy.MustSet(snappr.Daily, 1, 7)
policy.MustSet(snappr.Secondly, int(time.Hour/time.Second), 6)
policy.MustSet(snappr.Last, 1, 3)

keep, need := snappr.Prune(times, policy)
for at, reason := range keep {
    if len(reason) == 0 {
        // delete the snapshot times[at]
    }
}

Documentation

Overview

Package snappr prunes snapshots according to a flexible retention policy.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Period 

type Period struct {
	Unit     Unit
	Interval int // ignored if Unit is Last (normalized to 1), must be > 0
}

Period is a specific time interval for snapshot retention.

func (Period) Compare 

func (p Period) Compare(other Period) int

Compare strictly compares the provided periods.

func (Period) Normalize 

func (p Period) Normalize() (Period, bool)

Normalize validates and canonicalizes a period.

func (Period) String 

func (p Period) String() string

String formats the period in a human-readable form. The exact output is subject to change.

type Policy 

type Policy struct {
	// contains filtered or unexported fields
}

Policy defines a retention policy for snapshots.

All periods are valid and normalized.

func ParsePolicy 

func ParsePolicy(rule ...string) (Policy, error)

ParsePolicy parses a policy from the provided rules.

Each rule is in the form N@unit:X, where N is the snapshot count, unit is a unit name, and X is the interval. If N is negative, an infinite number of snapshots is retained. N must not be zero. X must be greater than zero. If N@ is omitted, it defaults to -1. If :X is omitted, it defaults to 1. For the "last" unit, X must be 1. For the "secondly" unit, X can also be a duration in the format used by time.ParseDuration. Each rule must be unique by the unit:X.

func Prune 

func Prune(snapshots []time.Time, policy Policy, loc *time.Location) (keep [][]Period, need Policy)

Prune prunes the provided list of snapshots, returning a matching slice of periods requiring that snapshot, and the remaining number of snapshots required to fulfill the original policy.

All snapshots are placed in the provided timezone, and the monotonic time component is removed. The timezone affects the exact point at which calendar days/months/years are split. Beware of duplicate timestamps at DST transitions (if the offset isn't included whatever you use as the snapshot name, and your timezone has DST, you may end up with two snapshots for different times with the same name).

See pruneCorrectness in snappr_test.go for some additional notes about guarantees provided by Prune.

Example 
var times []time.Time
for i := 0; i < 5000*24*2; i++ {
	times = append(times, time.Date(2000, 1, 1, 0, 30*i, prand(30*60, i, 0xABCDEF0123456789), 0, time.UTC))
}

var policy Policy
policy.MustSet(Yearly, 5, -1)
policy.MustSet(Yearly, 2, 10)
policy.MustSet(Yearly, 1, 3)
policy.MustSet(Monthly, 6, 4)
policy.MustSet(Monthly, 2, 6)
policy.MustSet(Daily, 1, 7)
policy.MustSet(Secondly, int(time.Hour/time.Second), 6)
policy.MustSet(Last, 1, 3)
fmt.Println(policy)

keep, need := Prune(times, policy, time.UTC)
for at, reason := range keep {
	at := times[at]
	if len(reason) != 0 {
		var b strings.Builder
		for i, r := range reason {
			if i != 0 {
				b.WriteString(", ")
			}
			b.WriteString(r.String())
		}
		fmt.Println(at.Format(time.ANSIC), "|", b.String())
	}
}
fmt.Println(need)

Output:

last (3), 1h time (6), 1 day (7), 2 month (6), 6 month (4), 1 year (3), 2 year (10), 5 year (inf)
Fri Dec 31 23:55:29 1999 | 2 year, 5 year
Sat Jan  1 00:36:00 2000 | 2 year, 5 year
Tue Jan  1 00:45:28 2002 | 2 year
Thu Jan  1 00:04:24 2004 | 2 year
Sat Jan  1 00:04:16 2005 | 5 year
Sun Jan  1 00:43:52 2006 | 2 year
Tue Jan  1 00:02:48 2008 | 2 year
Fri Jan  1 00:42:16 2010 | 2 year, 5 year
Sat Jan  1 00:11:21 2011 | 1 year
Thu Dec  1 00:18:09 2011 | 6 month
Sun Jan  1 00:01:12 2012 | 1 year, 2 year
Fri Jun  1 00:43:36 2012 | 6 month
Mon Oct  1 00:13:28 2012 | 2 month
Sat Dec  1 00:38:47 2012 | 2 month, 6 month
Tue Jan  1 00:01:04 2013 | 1 year
Fri Feb  1 00:33:52 2013 | 2 month
Mon Apr  1 00:27:37 2013 | 2 month
Sat Jun  1 00:12:41 2013 | 2 month, 6 month
Thu Aug  1 00:38:00 2013 | 2 month
Mon Sep  2 00:01:04 2013 | 1 day
Tue Sep  3 00:31:51 2013 | 1 day
Wed Sep  4 00:01:37 2013 | 1 day
Thu Sep  5 00:32:24 2013 | 1 day
Fri Sep  6 00:12:25 2013 | 1 day
Sat Sep  7 00:43:12 2013 | 1 day
Sun Sep  8 00:03:28 2013 | 1 day
Sun Sep  8 18:18:52 2013 | 1h time
Sun Sep  8 19:09:38 2013 | 1h time
Sun Sep  8 20:20:09 2013 | 1h time
Sun Sep  8 21:51:26 2013 | 1h time
Sun Sep  8 22:01:57 2013 | 1h time
Sun Sep  8 22:12:12 2013 | last
Sun Sep  8 23:22:43 2013 | last, 1h time
Sun Sep  8 23:33:14 2013 | last
last (0), 1h time (0), 1 day (0), 2 month (0), 6 month (0), 1 year (0), 2 year (2), 5 year (inf)

func (Policy) Clone 

func (p Policy) Clone() Policy

Clone returns a copy of the policy.

func (Policy) Each 

func (p Policy) Each(fn func(period Period, count int))

Each loops over all periods in order.

func (Policy) Get 

func (p Policy) Get(period Period) (count int)

Get gets the count for a period if it is set.

func (Policy) MarshalText 

func (p Policy) MarshalText() ([]byte, error)

MarshalText encodes the policy into a form usable by UnmarshalText. The output is the canonical form of the rules (i.e., all equivalent policies will result in the same output).

func (*Policy) MustSet 

func (p *Policy) MustSet(unit Unit, interval, count int)

MustSet is like Set, but panics if the period is invalid or has already been used.

func (*Policy) Set 

func (p *Policy) Set(period Period, count int) (ok bool)

Set sets the count for a period if it is valid, replacing any existing count. A count of zero removes the period.

func (Policy) String 

func (p Policy) String() string

String formats the policy in a human-readable form. The exact output is subject to change.

func (*Policy) UnmarshalText 

func (p *Policy) UnmarshalText(b []byte) error

UnmarshalText parses the provided text into p, replacing the existing policy. It splits the text by whitespace and calls ParsePolicy.

type Unit 

type Unit int

Unit represents a precision and unit of measurement. 

const (
	Last     Unit = iota // snapshot count
	Secondly             // wallclock seconds
	Daily                // calendar days
	Monthly              // calendar months
	Yearly               // calendar years

)

func (Unit) Compare 

func (u Unit) Compare(other Unit) int

Compare strictly compares two units.

func (Unit) IsValid 

func (u Unit) IsValid() bool

IsValid checks if the unit is known.

func (Unit) String 

func (u Unit) String() string

String returns the name of the unit, which is identical to the constant name, but in lowercase.

Source Files

View all Source files

Directories

Path Synopsis
cmd
snappr
Command snappr prunes time-based snapshots from stdin.
 Command snappr prunes time-based snapshots from stdin.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.