smugmug

README

SmugMug backup

Makes a full backup of a SmugMug account (images and videos are supported).

The program loops over the images and videos of the user's albums and saves them in the destination folder, replicating the SmugMug paths.

You can run the app multiple times, all exising files will be skipped if their sizes match.

With a good internet connection, a full backup of ~200GB can be completed in around 90 minutes using 10 analyzers and 10 downloaders (see #configuration for these options tuning) and few minutes for a daily incremental backup.

• SmugMug backup
  • Releases
  • Automated backups
  • Configuration
  • Run
  • Credentials
    • Obtain API keys
    • Obtain Tokens
      • Alternative ways to obtain the tokens
  • Build and install
  • Debug for errors
  • Visualize software stats
  • Credits
  • Code of conduct
  • Bugs and contributing

Releases

Releases for multiple systems (including ARM) can be found in the project releases page

Automated backups

As mentioned above, the app can be executed multiple times with the same configuration to perform incremental backups.
My suggestion is to use a linux home server (like a RaspberryPI with an attached NAS or external disk) and use cron to execute the backup daily.

A simple setup includes the configuration file into $HOME/.smgmg/config.toml, the binary file into $HOME/bin/smugmug-backup and this simple line into user's crontab (run crontab -e to edit the crontab configuration):

3 4 * * * $HOME/bin/smugmug-backup

If your system is configured to send emails then you'll also receive a daily recap of the backup job in your inbox 🎉

Configuration

The app expects to find the TOML configuration file in ./config.toml or $HOME/.smgmg/config.toml.

For the configuration structure, check the config.example.toml file.

Some values can be overridden by environment variables, that have the following names:

SMGMG_BK_API_KEY = "<API Key>"
SMGMG_BK_API_SECRET = "<API Secret>"
SMGMG_BK_USER_TOKEN = "<User Token>"
SMGMG_BK_USER_SECRET = "<User Secret>"
SMGMG_BK_DESTINATION = "<Backup destination folder>"
SMGMG_BK_FILE_NAMES = "<Filename with template replacements>"

Configuration identifier	Required	Default Value	Description
authentication.*	Yes		See credentials below for details about how to obtain them.
store.destination	Yes		Local path to save SmugMug pictures and videos into. If the folder is not empty, then only new or changed files will be downloaded. Windows users The value of destination must use slash / or double backslash \\ Examples: toml destination = "C:/folder/subfolder" destination = "C:\\folder\\subfolder" destination = "/folder/subfolder" # This writes to the primary partition C:
store.file_names	No	{{.FileName}}	Is a string including template replacements that will be used to build the file names for the files on disk. Accepted keys are FileName, ImageKey, ArchivedMD5 and UploadKey and their values comes from the AlbumImage API response. If an invalid replacement is used, an error is returned
store.use_metadata_times	No	false	When true, the last modification timestamp of the objects will be set based on SmugMug metadata for newly downloaded files. If also force_metadata_times is true, then the timestamp is applied to all existing files. This configuration can be required if you notice that the images creation datetime is wrong by ~7h. This is a bug in the SmugMug Uploader: "Our uploader process currently isn't time zone aware and takes the DateTimeOriginal field without time zone information". The solution is to use the Metadata API endpoint to retrieve the EXIF informations, but it requires an additional API call for each image/video. In my case, a full backup that requires ~10 minutes, increases to 2+ hours with this option.
store.force_metadata_times	No	false
store.write_csv	No	false	When true, a metadata.csv file is created (or overwritten) storing some information about the user files (both downloaded or skipped).
store.force_video_download	No	false	When true, videos are downloaded also if marked as "under processing"
store.concurrent_albums	No	1	Number of concurrently analyzed albums. It multiplies API calls, don't stress this number or you'll be rate limited. 5 is a good start.
store.concurrent_downloads	No	1	Number of concurrently downloaded images and videos.

Run

Once the configuration file and/or the environment variables are set, you can perform the account backup with:

./smugmug-backup

Running the backup can take a lot of time, depending on the size of your account and the connection speed. Check the command line logs to see what's going on.

Credentials

SmugMug requires OAuth1 authentication. OAuth1 requires 4 values: an API key and secret that identify the app with SmugMug and that you can get from SmugMug registering the app and user credentials (token and secret) that each user must obtain (those credentials authorize the app to access the user's data).

Obtain API keys

Apply for an API key from https://api.smugmug.com/api/developer/apply and wait the app to be authorized.

Once you have obtained the API key and secret, save them in the authentication section of the configuration file:

[authentication]
api_key = "<API Key>"
api_secret = "<API Secret>"

Obtain Tokens

Once your app has been accepted by SmugMug and you got the API key and secret, then go to your Account Settings > Privacy page and scroll down to "Authorized Services", where you'll find the app and a link to see the tokens.

Add them to the authentication section of the configuration file:

[authentication]
user_token = "<User Token>"
user_secret = "<User Secret>"

Alternative ways to obtain the tokens

Based on the examples from SmugMug (that you can find in the get_tokens folder, I've written a small web app that can help everyone to obtain their user token and secret.

The app has its own GitHub repo and a live version is deployed to heroku at https://smugmug-api-authenticator.herokuapp.com/.
You can use that app, it doesn't store any personal data in the server, but (as you should) you don't trust me, you can easily clone the GitHub repo, check the code, run the app locally and get the tokens.

If you prefer to use the console, the get_tokens folder contains a script from SmugMug to obtain the OAuth1 tokens. You need to create a config.json file with your API key/secret using example.json as example. Then, using a python3 environment, run run-console.sh. The script will show you a link you must open with your browser. SmugMug will give you a 6-digit code you must then paste to the console prompt. That's the last step, the console will show the user token and secret

Build and install

To build and install the program from source:

git clone https://github.com/tommyblue/smugmug-backup.git
cd smugmug-backup
make build

This will produce the ./smugmug-backup binary.
More make commands are available, run make help to get help

Debug for errors

To increase the logging, export a DEBUG=1 environment variable:

DEBUG=1 ./smugmug-backup

Visualize software stats

In case you want to monitor the software performances you can use the -stats flag to enable statsviz debug page (available at http://localhost:6060/debug/statsviz/)

Credits

OAuth1 signature has been heavily inspired by https://github.com/gomodule/oauth1

The code in the get_tokens folder is a copy of https://gist.github.com/smugkarl/10046914

Code of conduct

Before interacting with the community around this project, be sure to read the code of conduct.

Bugs and contributing

See CONTRIBUTING.md

Documentation

Overview

Package smugmug implements the logic to perform a full backup of a SmugMug account (images and videos).

The program loops over the images and videos of the user's albums and saves them in the destination folder, replicating the SmugMug paths.

You can run the app multiple times, all existing files will be skipped if their sizes match.

Creating a binary with this package is as simple as:

package main

import (
	log "github.com/sirupsen/logrus"
	"github.com/tommyblue/smugmug-backup"
)

func main() {
	cfg, err := smugmug.ReadConf()
	if err != nil {
		log.WithError(err).Fatal("Configuration error")
	}

	wrk, err := smugmug.New(cfg)
	if err != nil {
		log.WithError(err).Fatal("Can't initialize the package")
	}

	if err := wrk.Run(); err != nil {
		log.Fatal(err)
	}
}

The app reads its configuration from ./config.toml or $HOME/.smgmg/config.toml.

The supported configuration keys/values are the following:

[authentication]
username = "<SmugMug username>"
api_key = "<API Key>"
api_secret = "<API Secret>"
user_token = "<User Token>"
user_secret = "<User Secret>"

[store]
destination = "<Backup destination folder>"
file_names = "{{.FileName}}"

All values can be overridden by environment variables, that have the following names:

SMGMG_BK_USERNAME = "<SmugMug username>"
SMGMG_BK_API_KEY = "<API Key>"
SMGMG_BK_API_SECRET = "<API Secret>"
SMGMG_BK_USER_TOKEN = "<User Token>"
SMGMG_BK_USER_SECRET = "<User Secret>"
SMGMG_BK_DESTINATION = "<Backup destination folder>"
SMGMG_BK_FILE_NAMES = "<Backup destination folder>"

All configuration values are required. They can be omitted in the configuration file as long as they are overridden by environment values.

Index

• Constants
• type Conf
  • func ReadConf(cfgPath string) (*Conf, error)
• type FileMetadata
• type Worker
  • func New(cfg *Conf) (*Worker, error)
  • func (w *Worker) Run() error
  • func (w *Worker) Stop()
  • func (w *Worker) Wait()

Constants

const METADATA_FNAME = "metadata.csv"

METADATA_FNAME is the name of the CSV file used to store files metadata

Types

type Conf

type Conf struct {
	ApiKey              string // API key
	ApiSecret           string // API secret
	UserToken           string // User token
	UserSecret          string // User secret
	Destination         string // Backup destination folder
	Filenames           string // Template for files naming
	UseMetadataTimes    bool   // When true, the last update timestamp will be retrieved from metadata
	ForceMetadataTimes  bool   // When true, then the last update timestamp is always retrieved and overwritten, also for existing files
	WriteCSV            bool   // When true, a CSV file including downloaded files metadata is written
	ForceVideoDownload  bool   // When true, download videos also if marked as under processing
	ConcurrentDownloads int    // number of concurrent downloads of images and videos, default is 1
	ConcurrentAlbums    int    // number of concurrent albums analyzed via API calls
	HTTPBaseUrl         string // Smugmug API URL, defaults to https://api.smugmug.com
	HTTPMaxRetries      int    // Max number of retries for HTTP calls, defaults to 3
	// contains filtered or unexported fields
}

Conf is the configuration of the smugmug worker

func ReadConf

func ReadConf(cfgPath string) (*Conf, error)

ReadConf produces a configuration object for the Smugmug worker.

It reads the configuration from ./config.toml or "$HOME/.smgmg/config.toml"

type FileMetadata

type FileMetadata struct {
	FileName    string
	ArchivedUri string
	Caption     string
	Keywords    string
}

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker actually implements the backup logic

func New

func New(cfg *Conf) (*Worker, error)

New return a SmugMug backup configuration. It returns an error if it fails parsing the command line arguments

func (*Worker) Run

func (w *Worker) Run() error

Run performs the backup of the provided SmugMug account.

The workflow is the following:

• Get user albums
• Iterate over all albums and:
• create folder
• iterate over all images and videos
• if existing and with the same size, then skip
• if not, download

func (*Worker) Stop

func (w *Worker) Stop()

func (*Worker) Wait

func (w *Worker) Wait()