polidor

README

polidor

Polidor is a program and library for cleaning up a disk based storage hierachy.

Sonia Hamilton, sonia@snowfrog.net, http://www.snowfrog.net.

Overview

Polidor is Portuguese for "polisher".

The files in Polidor are:

• cleaner/cleaner.go and cleaner/retentions.yml - the main program and it's configuration file. Directories being scanned are printed to the screen, to give an indication of progress. Run using:

% cd cleaner
% go run cleaner.go

• generator/generator.go and generator/generator.yml - for generating test data for the cleaner program, and it's configuration file. The names of files being generated are printed to the screen, to give an indication of progress. Run using:

% cd generator
% go run generator.go

• polidor.go and polidor_test.go - library and tests

Program Logic

• the configuration file is read in
• the storage directory is walked using filepath.Walk()
• to prevent the program from taking too long, a channel named timeout is used
• the polidor library provides various directory parsing functions

Tests

Tests are run using Travis https://travis-ci.org/soniah/polidor. They can also be run by:

% go test
% go test -run <test name>

Documentation

Index

• func Check(err error)
• func FindRetentionPeriod(retentionPeriods RetentionT, companyID int) int
• type Directory
  • func BuildDirectory(storage string, path string) (Directory, error)
  • func (d Directory) Retain(date time.Time, path string, period int) (bool, error)
  • func (d Directory) StripStart(path string) string
  • func (d Directory) ToEpoch(path string) (time.Time, error)
  • func (d Directory) ToPath(epoch time.Time) string
• type RetentionT
• type SubRetentionT

Functions

func Check

func Check(err error)

Check makes checking errors easy, so they actually get a minimal check

func FindRetentionPeriod

func FindRetentionPeriod(retentionPeriods RetentionT, companyID int) int

FindRetentionPeriod finds the retention period for a companyID eg 30 days, or returns the default retention period

Types

type Directory

type Directory struct {
	// Base directory for storage eg /data/fb
	Base string

	// CompanyID
	CompanyID int

	// Device name eg j1_readnews_com
	DeviceName string

	// Device Number eg 2466
	DeviceNumber int
}

Directory represents a directory for storage eg "/data/fb/1289/j1_readnews_com" TODO rename Directory to Database

func BuildDirectory

func BuildDirectory(storage string, path string) (Directory, error)

BuildDirectory is a constructor for the Directory struct. It takes a 'storage' like "/data/fb" and a 'path' like "/data/fb/1289/j1_readnews_com/2466/2015/08/11/09/18" and returns a 'Directory' struct

func (Directory) Retain

func (d Directory) Retain(date time.Time, path string, period int) (bool, error)

Retain returns true if, on a given 'date', the 'path' should be kept if the retention period is 'period'

func (Directory) StripStart

func (d Directory) StripStart(path string) string

StripStart strips directories, for example "/data/fb/1289/j1_readnews_com/2466/" from the start of a path eg "/data/fb/1289/j1_readnews_com/2466/2015/08/11/09/18" giving the date part of the path eg "2015/08/11/09/18"

func (Directory) ToEpoch

func (d Directory) ToEpoch(path string) (time.Time, error)

ToEpoch taks a path eg "/data/fb/1289/j1_readnews_com/2466/2015/08/11" and returns an epoch time eg 1470907113

func (Directory) ToPath

func (d Directory) ToPath(epoch time.Time) string

ToPath takes an epoch time eg 1470907113 and produces a path eg "/data/fb/1289/j1_readnews_com/2466/2015/08/11" ie discarding hours and minutes

type RetentionT

type RetentionT map[int]SubRetentionT

RetentionT is used for storing retention settings

type SubRetentionT

type SubRetentionT map[string]int

SubRetentionT is used for storing retention settings