periscope

README

Periscope

Periscope gives you "duplicate vision" to help you organize and de-duplicate your files without losing data.

Periscope (psc) works differently from most other duplicate file finders. It is designed to be used interactively: Periscope will help you explore the filesystem, understand which files are duplicated, determine where duplicate copies live, and safely delete duplicates without losing data.

Following a psc scan, Periscope lets you navigate and explore your filesystem with the workflow you're already used to — using your shell and commands like cd, ls, tree, and so on — while providing additional duplicate-aware commands that mirror core filesystem utilities. For example, psc ls gives a directory listing that highlights duplicates, and psc rm deletes files only if a duplicate exists elsewhere. This makes it easy to understand how data is organized (and duplicated), reorganize files, and delete duplicates without worrying about accidentally losing data.

Workflow · Commands · Installation · Contributing

Workflow

Find duplicates

Start with psc scan to scan folders for duplicates. Once you run this, you shouldn't need to run it again while looking at and deleting duplicates, unless you move files around. If you delete files manually (rather than with psc rm), you can make Periscope detect deletions with psc refresh, which runs much faster than a full scan. psc scan is incremental, so if you want to scan a new directory or re-analyze one that was already scanned, you can always run the command again.

Understand duplicates

You can get a high-level understanding of how many duplicates you have and where they are located:

• psc summary gives statistics on duplicate files
• psc report shows a full list of duplicates, sorted by file size

After identifying areas to explore with psc report, you can navigate to those directories in your shell with cd, and then you can use Periscope commands to understand duplicates:

• psc ls gives a duplicate-aware directory listing (optionally recursively, with the -R flag)
• psc info shows information on a specific file (and its duplicates)

Delete duplicates

You can use the psc rm command to delete duplicates. You can think of it like a safe version of rm: it will not let you delete files unless there are duplicate copies elsewhere. A psc rm -r will recursively delete duplicates but not unique files. A psc rm --contained <path> will delete duplicates only if a copy is contained in the given folder.

Remove duplicate database

When you're done with a Periscope session, you can delete the duplicate database with psc finish.

Commands

Run psc help to see the full list of commands and psc help [command] to see help on a specific command.

psc scan scans for duplicates

Scans paths for duplicates and populates the database with information about duplicates. Scans the current directory if given no argument. Scanning is incremental; if you want to start from scratch, run psc finish first.

psc refresh removes deleted files from the database

Removes deleted files from the duplicate database. psc rm does this automatically, so this command only needs to be used if you use some other program (e.g. coreutils rm) and want to remove missing files from the database. This command does not re-analyze files, so if you've made substantial changes to the filesystem, like moving files around or adding new files, it's best to do a psc scan of the relevant directories.

psc finish deletes the duplicate database

Deletes the duplicate database. Once you're done using Periscope, it's good to use this command to delete the duplicate database, so it doesn't waste space on disk.

psc summary reports statistics

Prints statistics about the duplicate database, such as number of duplicate files and the amount of space duplicates consume.

psc report reports scan results

Lists all duplicates in the duplicate database, sorted by file size. Because this list is usually large, it's helpful to pipe the output to a pager, e.g. psc report | less.

psc export exports scan results

Exports information about duplicates in a machine-readable format (default JSON). This is the only output from Periscope that other programs should consume. Future versions of Periscope may add to the information that's included in the dump, but the layout of existing data will not change.

psc ls lists a directory

Lists files and folders in the given directory (or the current directory, if none is given). This command shows the number of duplicates that each file has: 1 means that there is a single duplicate elsewhere in the filesystem; if a file has no duplicates, the number is omitted. Directories are tagged with a 'd', and special files are tagged with a character describing their type, e.g. 'p' for named pipes. -a shows hidden files. -d lists only duplicates, while -u lists only unique files. -v lists all duplicates of every file, and -r shows the path to the duplicate as a relative path instead of an absolute path. -R lists files recursively; this flag combines well with the -d flag, to list only duplicate files recursively contained in a given directory.

psc tree lists all duplicates in a given directory

Lists all files recursively contained in the given directory (or the current directory, if none is given) that have a duplicate file elsewhere. This command hides hidden files and folders by default; the -a flag includes hidden files.

This command shows a "flattened" representation; in most cases, a psc ls -Rd is more useful.

psc info inspects a file

Shows information about a single file's duplicates. Like with psc ls, the -r flag shows the path to the duplicate as a path relative to the given file.

psc rm deletes duplicates

Deletes duplicates but not unique files; no way of invoking this command will delete unique files. This command makes use of the database, but it double-checks files and their copies before it deletes anything, so a stale duplicate database will not result in data loss. The -n flag will perform a dry run, listing files that would be deleted but not actually deleting anything. -r deletes duplicates recursively. The --contained <path> argument gives more fine-grained control over deletion: files are only deleted if they have a duplicate in the given location. This is useful, for example, for deleting files from a "to organize" directory only if they are also contained in the "organized" directory, as in the demo video above. By default, psc rm does not delete any files when it's given a set where there are no duplicates outside the set: for example, if files "/a/x1" and "/a/x2" are duplicates, recursively removing "/a" will leave both files untouched. Passing the --arbitrary flag will result in such duplicates being handled by arbitrarily choosing one file to save and deleting the rest.

Installation

Install with Homebrew (on macOS):

brew install periscope

Download a binary release: Periscope releases.

Periscope has binary releases for macOS and Linux. It has not been tested on Windows.

Install from source with go install:

go install -v github.com/anishathalye/periscope/cmd/psc@latest

Periscope depends on go-sqlite3, which uses cgo, so you need a C compiler present in your path. You might also need to set CGO_ENABLED=1 if you have it disabled otherwise.

Contributing

Bug reports, feature requests, feedback on the tool or documentation, and pull requests are all appreciated. If you are planning on making substantial changes that you hope to have merged, it is highly recommended that you first open an issue to discuss your proposed change.

License

Copyright (c) Anish Athalye (me@anishathalye.com). Released under GPLv3. See LICENSE.txt for details.