Hard dependencies:¶

• glib \(\geq 2.32\) (general C Utility Library)

Soft dependencies:¶

• libblkid (detecting mountpoints)
• libelf (nonstripped binary detection)
• libjson-glib (parsing rmlint's own json as caching layer)

Build dependencies:¶

• git (version control)
• scons (build system)
• sphinx (manpage/documentation generation)
• gettext (support for localization)

Here's a list of readily prepared commands for known distributions:

• Fedora \(\geq 21\):

  $ yum -y install git scons python3-sphinx gettext json-glib-devel
  $ yum -y install glib2-devel libblkid-devel elfutils-libelf-devel
  # Optional dependencies for the GUI:
  $ yum -y install pygobject3 gtk3 librsvg2
  

  There are also pre-built packages on Fedora Copr:

  $ dnf copr enable sahib/rmlint
  $ dnf install rmlint
  

  Those packages are built from master snapshots and might be outdated.

• ArchLinux:

  There is an official package in [community] here:

  $ pacman -S rmlint
  

  Alternatively you can use rmlint-git in the AUR:

  $ pacman -S git scons python-sphinx
  $ pacman -S glib2 libutil-linux elfutils json-glib
  # Optional dependencies for the GUI:
  $ pacman -S gtk3 python-gobject librsvg
  

  There is also a PKGBUILD on the ArchLinux AUR:

  $ # Use your favourite AUR Helper.
  $ yaourt -S rmlint-git
  

  It is built from git master, not from the develop branch.

• Ubuntu \(\geq 12.04\):

  This most likely applies to most distributions that are derived from Ubuntu.

  $ apt-get install git scons python3-sphinx python3-nose gettext build-essential
  # Optional dependencies for more features:
  $ apt-get install libelf-dev libglib2.0-dev libblkid-dev libjson-glib-1.0 libjson-glib-dev
  # Optional dependencies for the GUI:
  $ apt-get install python3-gi gir1.2-rsvg gir1.2-gtk-3.0
  

• FreeBSD \(\geq 10.1\):

  $ pkg install git scons py27-sphinx
  $ pkg install glib gettext libelf json-glib
  

Send us a note if you want to see your distribution here. The commands above install the full dependencies, therefore some packages might be stripped if you do not need the feature they enable. Only hard requirement is glib.

Also be aware that the GUI needs at least \(gtk \geq 3.14\) to work!

Limit files by size using --size¶

# only check files between 20 MB and 1 Gigabyte:
$ rmlint --size 20M-1G
# short form (-s) works just as well:
$ rmlint -s 20M-1G
# only check files bigger than 4 kB:
$ rmlint -s 4K
# only check files smaller than 1234 bytes:
$ rmlint -s 0-1234

Valid units include:

If no units are given, rmlint assumes bytes.

Limit duplicate matching according to basename¶

By default, rmlint compares file contents, regardless of file name. So if afile.jpg has the same content as bfile.txt (which is unlikely!), then rmlint will find and report this as a duplicate. You can speed things up a little bit by telling rmlint not to try to match files unless they have the same or similar file names. The three options here are:

Examples:

# Find all duplicate files with the same basename:
$ rmlint -b some_dir/
ls some_dir/one/hello.c
rm some_dir/two/hello.c
# Find all duplicate files that have the same extension:
$ rmlint -e some_dir/
ls some_dir/hello.c
rm some_dir/hello_copy.c
# Find all duplicate files that have the same basename:
# minus the extension
$ rmlint -e some_dir/
ls some_dir/hello.c
rm some_dir/hello.bak

Limit files by their modification time¶

This is an useful feature if you want to investigate only files newer than a certain date or if you want to progessively update the results, i.e. when you run rmlint in a script that watches a directory for duplicates.

The manual way is using -N (--newer-than=<timestamp>):

# Use a Unix-UTC Timestamp (seconds since epoch)
$ rmlint -N 1414755960

# Find all files newer than file.png
$ rmlint -N $(stat --print %Y file.png)

# Alternatively use a ISO8601 formatted Timestamp
$ rmlint -N 2014-09-08T00:12:32+0200

If you are periodically checking the same directory tree for duplicates, you can get a substantial speedup by creating an automatic timestamp file each time you run rmlint. To do this, use command line options: -n (--newer-than-stamp) and -O stamp:stamp.file (we'll come to outputs in a minute): Here's an example for incrementally scanning your home folder:

# First run of rmlint:
$ rmlint /home/foobar -O stamp:/home/foobar/.rmlint.stamp
ls /home/foobar/a.file
rm /home/foobar/b.file

# Second run, no changes:
$ rmlint /home/foobar -n /home/foobar/.rmlint.stamp
<nothing>

# Second run, new file copied:
$ cp /home/foobar/a.file /home/foobar/c.file
$ rmlint /home/foobar -n /home/foobar/.rmlint.stamp
ls /home/foobar/a.file
rm /home/foobar/b.file
rm /home/foobar/c.file

Note that -n updates the timestamp file each time it is run.

Flagging original directories¶

Sometimes you have a specific path that only contains originals, or only contains backups. In this case you can flag directories on the commandline by using a special separator (//) between the duplicate and original paths. Every path after the // separator is considered to be "tagged" and will be treated as an original where possible. Tagging always takes precedence over the -S options above.

$ rmlint a // b
ls b/file
rm a/file

If there are more than one tagged files in a duplicate group then the highest ranked (per -S options) will be kept. In order to never delete any tagged files, there is the -k (--keep-all-tagged) option. A slightly more esoteric option is -m (--must-match-tagged), which only looks for duplicates where there is an original in a tagged path.

Here's a real world example using these features: You have an portable backup drive with some old backups on it. You have just backed up your home folder to a new backup drive. You want to reformat the old backup drive and use it for something else. But first you want to check that there is nothing on the old drive that you don't have somewhere else. The old drive is mounted at /media/portable.

# Find all files on /media/portable that can be safely deleted:
$ rmlint --keep-all-tagged --keep-all-tagged /media/portable // ~
# check the shellscript looks ok:
$ less ./rmlint.sh # or use gedit or any other viewer/editor
# run the shellscript to delete the redundant backups
$ ./rmlint.sh
# run again (to delete empty dirs)
$ rmlint -km /media/portable // ~
$ ./rmlint.sh
# see what files are left:
$ tree /media/portable
# recover any files that you want to save, then you can safely reformat the drive

In the case of nested mountpoints, it may sometimes makes sense to use the opposite variations, -K (--keep-all-untagged) and -M (--must-match-untagged).

Location view¶

Shows a list of locations the user might want to scan. A number of locations is guessed from the list of mounted volumes, recently used files and a static set of paths. The user can of course add a new location via a filebrowser.

The user can select one or multiple paths and hit Scan. In prior he might choose to prefer certain paths, so only files in non-preferred paths are deleted if they have a twin in a preferred path.

Runner view¶

After hitting scan in the locations view, the application will start rmlint in the background. The output will be shown live in the treeview on the left.

Once finished, a chart will be shown on the right that shows how the duplicates are distributed over the scanned directories. The treeview will show the detailed list of found files. A red cross will indicate that Shredder wants to delete this file, a green checkmark will make it keep it. The user can edit those to his liking.

Additionally, the view can be filtered after a search query. In the simplest case this filters by a path element, in more complex usecases you can also filter by size, mtime and twincount. The latter can be done by adding size:10K or size:1M-2M,3M-4M to the query (similar with mtime: and count:)

Once happy with the results, the user can generate a script out of the results (all or just those that are visible).

Editor view¶

A source editor will show the generated script. It can be edited and searched through. Apart from that, the file can be saved not only a .sh file, but also as .csv and .json file.

The user can now choose to save the script and execute it himself, or to click the Run Script button. If this button is blue, it indicates a dry run, where nothing will be deleted. A red button however will kill your files. In any way, a running counter of deleted bytes will be shown.

Settings view¶

The settings view is the leftmost view and will not be shown by default in the process. It can always be accessed by:

• Going to the leftmost view.
• Hitting the Settings menu entry.
• Hit the gear button in the runner view.

Normal user probably do not need to adjust anything. The options shown here, resemble the option that may be given to the commandline of rmlint.

Design¶

The design loosely follows the Gnome Human Interface Guidelines. [*] Beside the appeareance, this means that the program should be easy and intuitive to use. Suggested actions should be clear to recognize and the application should always be responsive and never just do work in the background.

Internal¶

Shredder works by forking off rmlint and reading it's json output in parallel. The script generation works by calling rmlint --replay on the generated json file, since this is the only sane way to filter the results of all formats properly.

Backup your data¶

There is a wise adage, "if it's not backed up, it's not important". It's just good practice to keep your important data backed up. In particular, any time you are contemplating doing major file reorganisations or deletions, that's a good time to make sure that your backups are up to date.

What about when you want to clean up your backups by deleting duplicate files from your backup drive? Well as long as your duplicate file finder is reliable, you shouldn't have any problems. Consider replacing the duplicate with a link (hardlink, symlink or reflink) to the original data. This still frees up the space, but makes it easier to find the file if and when it comes time to restore files from backup.

Measure twice, cut once¶

This is a popular saying amongst builders; the same goes for your files. Do at least some sort of sanity check on which files are going to be deleted. All duplicate file finders (including rmlint) are capable of identifying false positives or more serious bugs.

Beware of unusual filename characters¶

Even a space in a filename is capable of causing grief. Make sure your file deletion command (or the one used by your duplicate finder) has the filename properly escaped.

Consider safe removal options¶

Rather than deleting duplicates, consider moving them to a holding area or trash folder. The trash-cli utility is one option for this. Alternatively if using the rmlint shell script you can replace the remove_cmd section as follows to move the files to /tmp:

remove_cmd() {
    echo 'Deleting:' "$1"
    if original_check "$1" "$2"; then
        if [ -z "$DO_DRY_RUN" ]; then
            # was: rm -rf "$1"
            mv -p "$1" "/tmp$1"
        fi
    fi
}

Another safe alternative, if your files are on a btrfs filesystem and you have linux kernel 4.2 or higher, is to reflink the duplicate to the original. You can do this via cp --reflink or using rmlint --btrfs-clone:

$ cp --reflink=always original duplicate   # deletes dupicate and replaces it with reflink copy of original
$ rmlint --btrfs-clone original duplicate  # does and in-place clone

If you pass -c sh:link to rmlint, it will even check for you if your filesystem is capable of reflinks and emit the correct command conveniently.

The second option is actually safer because it verifies (via the kernel) that the files are identical before creating the reflink. Also it does not change the mtime or other metadata of the duplicate.

You might think hardlinking as a safe alternative to deletion, but in fact hardlinking first deletes the duplicate and then creates a hardlink to the original in its place. If your duplicate finder has found a false positive, it is possible that you may lose your data.

Traversal Robustness¶

Path Doubles

One simple trap for a dupe finder is to not realise that it has reached the same file via two different paths. This can happen due to user inputting overlapping paths to traverse, or due to symlinks or other filesystem loops such as bind mounts. Here's a simple "path double" example trying to trick a duplicate file finder by "accidentally" feeding it the same path twice. We'll use fdupes for this example:

$ mkdir dir
$ echo "important" > dir/file
$ fdupes -r --delete --noprompt dir dir
[no output]
$ ls dir
file

So far so good, fdupes didn't fall for the trick. It has a check built-in which looks at the files' device and inode numbers, which automatically filters out path doubles.

Let's try again using the -H option to find hardlinked duplicates:

$ fdupes -r -H --delete --noprompt dir dir
   [+] dir/file
   [-] dir/file
$ ls -l dir/
total 0

Oh dear, our file is gone! The problem is that hardlinks share the same device and inode numbers, so the inode check is turned off for this option.

Dupe finders rdfind and dupd can also be tricked with the right combination of settings:

$ rdfind -removeidentinode false -deleteduplicates true a a
[snip]
Now deleting duplicates:
Deleted 1 files.
$ ls -l dir/
total 0

$ dupd scan --path /home/foo/a --path /home/foo/a
Files scanned: 2
Total duplicates: 2
Run 'dupd report' to list duplicates.
$ dupd report
Duplicate report from database /home/foo/.dupd_sqlite:
20 total bytes used by duplicates:
  /home/foo/a/data
  /home/foo/a/data

Solution:

For a duplicate finder to be able to find hardlinked duplicates, without also inadvertently identifying a file as a duplicate or itself, a more sophisticated test is required. Path doubles will always have:

• matching device and inode.
• matching basename.
• parent directories also have matching device and inode.

That seems pretty fool-proof (see rmlint example below) but please file an issue on our Issue Tracker if you find an exception.

$ echo "data" > dir/file
$ # rmlint with default settings:
$  rmlint dir dir
==> In total 2 files, whereof 0 are duplicates in 0 groups.
==> This equals 0 B of duplicates which could be removed.
$
$ # rmlint with hardlink duplicate detection enabled:
$  rmlint --hardlinked dir dir
==> In total 2 files, whereof 0 are duplicates in 0 groups.
==> This equals 0 B of duplicates which could be removed.
$ ls dir
file

Symlinks:

"Ah but I'm not silly enough to enter the same path twice" you say. Well maybe so, but there are other ways that folder traversal can reach the same path twice, for example via symbolic links:

$ mkdir dir
$ echo "important" > dir/file
$ ln -s dir link
$ fdupes -r --delete --noprompt .
$ ls -l dir/
total 0

Symlinks can make a real mess out of filesystem traversal:

$ mkdir dir
$ cd dir
$ ln -s . link
$ cd ..
$ echo "data" > dir/file
$ fdupes -rHs dir
dir/file
dir/link/file
dir/link/link/file
[snip]
dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file

Set 1 of 1, preserve files [1 - 41, all]:

Solution:

During traversal, the duplicate finder should keep track of all folders visited (by device and inode number). Don't re-traverse folders that were already traversed.

Hardlinks:

Also as noted above, replacing duplicates with hardlinks can still end badly if there are false positives. For example, using rdfind's the -makehardlinks option:

$ echo "data" > dir/file
$ rdfind -removeidentinode false -makehardlinks true dir dir
[snip]
It seems like you have 2 files that are not unique
Totally, 5 b can be reduced.
Now making results file results.txt
Now making hard links.
failed to make hardlink dir/file to dir/file
$ ls -l dir
total 0

Solution:

Don't find false positives. Check files are on same filesystem before trying to create hardlink. Temporarily rename the duplicate before creating the hardlink and then deleting the renamed file.

Collision Robustness¶

Duplicate detection by file hash

If a duplicate finder uses file hashes to identify duplicates, there is a very small risk that two different files have the same hash value. This is called a hash collision and can result in the two files being falsely flagged as duplicates.

Several duplicate finders use the popular MD5 Hash, which is 128 bits long. With a 128-bit hash, if you have a million sets of same-size files, each set containing a million different files, the chance of a hash collision is about 0.000 000 000 000 000 000 147%. To get a 0.1% chance of a hash collision you would need nine hundred thousand million (\(9\times10^{11}\)) groups of (\(9\times10^{11}\)) files each, or one group of eight hundred thousand million million (\(8\times10^{17}\)) files.

If someone had access to your files, and wanted to create a malicious duplicate, they could potentially do something like this (based on http://web.archive.org/web/20071226014140/http://www.cits.rub.de/MD5Collisions/):

$ mkdir test && cd test
$ # get two different files with same md5 hash:
$ wget http://web.archive.org/web/20071226014140/http://www.cits.rub.de/imperia/md/content/magnus/order.ps
$ wget http://web.archive.org/web/20071226014140/http://www.cits.rub.de/imperia/md/content/magnus/letter_of_rec.ps
$ md5sum *  # verify that they have the same md5sum
a25f7f0b29ee0b3968c860738533a4b9  letter_of_rec.ps
a25f7f0b29ee0b3968c860738533a4b9  order.ps
$ sha1sum * # verify that they are not actually the same
07835fdd04c9afd283046bd30a362a6516b7e216  letter_of_rec.ps
3548db4d0af8fd2f1dbe02288575e8f9f539bfa6  order.ps
$ rmlint -a md5 . -o pretty  # run rmlint using md5 hash for duplicate file detection
# Duplicate(s):
    ls '/home/foo/test/order.ps'
    rm '/home/foo/test/letter_of_rec.ps'
$ rmlint test   -o summary   # run using default sha1 hash
==> In total 2 files, whereof 0 are duplicates in 0 groups.

If your intention was to free up space by hardlinking the duplicate to the original, you would end up with two hardlinked files, one called order.ps and the other called letter_of_rec.ps, both containing the contents of order.ps.

Solution:

fdupes detects duplicates using MD5 Hashes, but eliminates the collision risk by doing a byte-wise comparison of the duplicates detected. This means each file is read twice, which can tend to slow things down.

dupd uses direct file comparison, unless there are more than 3 files in a set of duplicates, in which case it uses MD5 only.

rmlint's default option uses a 160-bit SHA1 hash which means you need at least \(5.4\times10^{22}\) files before you get a \(0.1\%\) probability of collision. rmlint's -p option uses SHA512 (\(5.2\times10^{75}\) files for \(0.1\%\) risk), while rmlint's -pp option uses direct file comparison to eliminate the risk altogether. Refer to the Benchmarks chapter for speed and memory overhead implications.

Unusual Characters Robustness¶

Spaces, commas, nonprinting characters etc can all potentially trip up a duplicate finder or the subsequent file deletion command. For example:

$ mkdir test
$ echo "data" > 'test/\t\r\"\b\f\\,.'
$ cp test/\\t\\r\\\"\\b\\f\\\\\,. test/copy  # even just copying filenames like this is ugly!
$ ls -1 test/
copy
\t\r\"\b\f\\,.
$ md5sum test/*  # md5's output gets a little bit mangled by the odd characters
6137cde4893c59f76f005a8123d8e8e6  test/copy
\6137cde4893c59f76f005a8123d8e8e6  test/\\t\\r\\"\\b\\f\\\\,.
$ dupd scan --path /home/foo/test
SKIP (comma) [/home/foo/test/\t\r\"\b\f\\,.]
Files scanned: 1
Total duplicates: 0

Solution: Be careful!

"Seek Thrash" Robustness¶

Duplicate finders use a range of strategies to find duplicates. It is common to reading and compare small increments of potential duplicates. This avoids the need to read the whole file if the files differ in the first few megabytes, so this can give a major speedup in some cases. However, in the case of hard disk drives, constantly reading small increments from several files at the same time causes the hard drive head to have to jump around ("seek thrash").

Here are some speed test results showing relative speed for scanning my /usr folder (on SSD) and a HDD copy of same. The speed ratio gives an indication of how effectively the search algorithm manages disk seek overheads:

$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Solution:

Achieving good speeds on HDD's requires a balance between small file increments early on, then switching to bigger file increments. Fiemap information (physical location of files on the disk) can be used to sort the files into an order that reduces disk seek times.

Memory Usage Robustness¶

When scanning very large filesystems, duplicate finders may have to hold a large amount of information in memory at the same time. Once this information exceeds the computers' RAM, performance will suffer signficantly. dupd handles this quite nicely by storing a lot of the data in a sqlite database file, although this may have a slight performance penalty due to disk read/write time to the database file. rmlint uses a path tree structure to reduce the memory required to store all traversed paths.

Control Variables¶

The behaviour of the testsuite can be controlled by certain environment variables which are:

• RM_TS_DIR: Testdir to create files in. Can be very large with some tests, sometimes tmpfs might therefore slow down your computer. By default /tmp will be used.
• RM_TS_USE_VALGRIND: Run each test inside of valgrind's memcheck. (slow)
• RM_TS_USE_GDB: Run tests inside of gdb. Fatal signals will trigger an backtrace.
• RM_TS_PEDANTIC: Run each test several times with different optimization options and check for errors between the runs. (slow).
• RM_TS_SLEEP: Waits a long time before executing a command. Useful for starting the testcase and manually running rmlint on the priorly generated testdir.
• RM_TS_PRINT_CMD: Print the command that is currently run.
• RM_TS_KEEP_TESTDIR: If a test failed, keep the test files.

Additionally slow tests can be omitted with by appending -a '!slow' to the commandline. More information on this syntax can be found on the nosetest documentation.

Before each release we call the testsuite (at least) like this:

$ sudo RM_TS_USE_VALGRIND=1 RM_TS_PRINT_CMD=1 RM_TS_PEDANTIC=1 nosetests-3.4 -s -a '!slow'

The sudo here is there for executing some tests that need root access (like the creating of bad user and group ids). Most tests will work without.

Coverage¶

To see which functions need more testcases we use gcov to detect which lines were executed (and how often) by the testsuite. Here's a short quickstart using lcov:

$ CFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS="-fprofile-arcs -ftest-coverage" scons -j4 DEBUG=1
$ sudo RM_TS_USE_VALGRIND=1 RM_TS_PRINT_CMD=1 RM_TS_PEDANTIC=1 nosetests-3.4 -s -a '!slow'
$ lcov --capture --directory . --output-file coverage.info
$ genhtml coverage.info --output-directory out

The coverage results are updated from time to time here:

http://sahib.github.io/rmlint/gcov/index.html

Structure¶

tests
├── test_formatters   # Tests for output formatters (like sh or json)
├── test_options      # Tests for normal options like --merge-directories etc.
├── test_types        # Tests for all lint types rmlint can find
└── utils.py          # Common utilities shared amon tests.

Templates¶

A template for a testcase looks like this:

from nose import with_setup
from tests.utils import *

@with_setup(usual_setup_func, usual_teardown_func)
def test_basic():
    create_file('xxx', 'a')
    create_file('xxx', 'b')

    head, *data, footer = run_rmlint('-a city -S a')

    assert footer['duplicate_sets'] == 1
    assert footer['total_lint_size'] == 3
    assert footer['total_files'] == 2
    assert footer['duplicates'] == 1

Rules¶

• Test should be able to run as normal user.

• If that's not possible, check at the beginning of the testcase with this:

  if not runs_as_root():
      return
  

• Regressions in rmlint should get their own testcase so they do not appear again.

• Slow tests can be marked with a slow attribute:

  from nose.plugins.attrib import attr
  
  @attr('slow')
  @with_setup(usual_setup_func, usual_teardown_func)
  def test_debian_support():
      assert random.choice([True, False]):
  

Environment Variables¶

# Use clang and enable profiling, verbose build and enable debugging
CC=clang CFLAGS='-pg' LDFLAGS='-pg' scons VERBOSE=1 DEBUG=1

Variables¶

Arguments¶

All --without-* options come with a --with-* option that inverses its effect. By default rmlint is built with all features available on the system, so you do not need to specify any --with-* option normally.

Notable targets¶

$ USE_VALGRIND=1 nosetests  # or nosetests-3.3, python3 needed.

Obvious ones¶

• Do not compare each file with each other by content, use a hashfunction to reduce comparison overhead drastically (introduces possibility of collisions though).
• Only compare files of same size with each other.
• Use incremental hashing, i.e. hash block-wise each size group and stop as soon a difference occurs or the file is read fully.
• Create one reading thread for each physical disk. This gives a big speedup if files are roughly evenly spread over multiple physical disks [note: currently using 2 reading threads per disk as a workaround for a speed regression but hoping to fix this for rmlint 2.5].
• Disk traversal is similarly multi-threaded, one thread per disk.
• Create separate hashing threads (one for each file) so that the reader threads don't have to wait for hashing to catch up.

Subtle ones¶

• Check only executable files to be non-stripped binaries.
• Use preadv(2) based reading for small speeedups.
• Every thread in rmlint is shared, so only few calls to pthread_create are made.

Insane ones¶

• Use fiemap ioctl(2) to analyze the harddisk layout of each file, so each block can read it in perfect order on a rotational device.
• Check the device ID of each file to see if it on a rotational (normal hard disks) or on a non-rotational device (like a SSD). On the latter the fiemap optimisation is bypassed.
• Use a common buffer pool for IO buffers and recycle used buffers to reduce memory allocation overheads.
• Use only one hashsum per group of same-sized files.
• Implement paranoia check using the same algorithm as the incremental hash. The difference is that large chunks of the file are read and kept in memory instead of just keeping the hash in memory. This avoids the need for a two-pass algorithm (find matches using hashes then confirm via bytewise comparison). Each file is read once only. This achieves bytewise comparison in O(N) time, even if there are large clusters of same-size files. The downside is that it is somewhat memory-intensive (can be configured by --limit-mem option).

SYNOPSIS¶

rmlint [TARGET_DIR_OR_FILES ...] [//] [TAGGED_TARGET_DIR_OR_FILES ...] [-] [OPTIONS]

DESCRIPTION¶

rmlint finds space waste and other broken things on your filesystem.

Types of waste include: * Duplicate files and directories. * Nonstripped Binaries (Binaries with debug symbols). * Broken links. * Empty files and directories. * Files with broken user or group id.

rmlint will not delete any files. It does however produce executable output (for example a shell script) to help you delete the files if you want to.

In order to find the lint, rmlint is given one or more directories to traverse. If no directories or files were given, the current working directory is assumed. By default, rmlint will ignore hidden files and will not follow symlinks (see traversal options below). rmlint will first find "other lint" and then search the remaining files for duplicates.

Duplicate sets will be displayed as an original and one or more duplicates. You can set criteria for how rmlint chooses using the -S option (by default it chooses the first-named path on the command line, or if that is equal then the oldest file based on mtime). You can also specify that certain paths only contain originals by naming the path after the special path separator //.

Examples are given at the end of this manual.

OPTIONS¶

$ rmlint large_file_cluster/ -U --xattr-write   # first run.
$ rmlint large_file_cluster/ --xattr-read       # second run.

FORMATTERS¶

• csv: Output all found lint as comma-separated-value list.

  Available options:

  • no_header: Do not write a first line describing the column headers.

• sh: Output all found lint as shell script This formatter is activated

  as default.

  Available options:

  • cmd: Specify a user defined command to run on duplicates. The command can be any valid /bin/sh-expression. The duplicate path and original path can be accessed via "$1" and "$2". The command will be written to the user_command function in the sh-file produced by rmlint.

  • handler Define a comma separated list of handlers to try on duplicate files in that given order until one handler succeeds. Handlers are just the name of a way of getting rid of the file and can be any of the following:

    • clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will physically delete duplicate extents. Needs at least kernel 4.2.
    • reflink: Try to reflink the duplicate file to the original. See also --reflink in man 1 cp. Fails if the filesystem does not support it.
    • hardlink: Replace the duplicate file with a hardlink to the original file. Fails if both files are not on the same partition.
    • symlink: Tries to replace the duplicate file with a symbolic link to the original. Never fails.
    • remove: Remove the file using rm -rf. (-r for duplicate dirs). Never fails.
    • usercmd: Use the provided user defined command (-c sh:cmd=something). Never fails.

    Default is remove.

  • link: Shortcut for -c sh:clone,reflink,hardlink,symlink.

  • hardlink: Shortcut for -c sh:hardlink,symlink.

  • symlink: Shortcut for -c sh:symlink.

• json: Print a JSON-formatted dump of all found reports. Outputs all finds as a json document. The document is a list of dictionaries, where the first and last element is the header and the footer respectively, everything between are data-dictionaries.

  Available options:

  • no_header=[true|false]: Print the header with metadata.
  • no_footer=[true|false]: Print the footer with statistics.
  • oneline=[true|false]: Print one json document per line.

• py: Outputs a python script and a JSON document, just like the json formatter. The JSON document is written to .rmlint.json, executing the script will make it read from there. This formatter is mostly intented for complex use-cases where the lint needs special handling. Therefore the python script can be modified to do things standard rmlint is not able to do easily.

• stamp:

  Outputs a timestamp of the time rmlint was run.

  Available options:

  • iso8601=[true|false]: Write an ISO8601 formatted timestamps or seconds since epoch?

• progressbar: Shows a progressbar. This is meant for use with stdout or stderr [default].

  See also: -g (--progress) for a convenience shortcut option.

  Available options:

  • update_interval=number: Number of milliseconds to wait between updates. Higher values use less resources (default 50).
  • ascii: Do not attempt to use unicode characters, which might not be supported by some terminals.
  • fancy: Use a more fancy style for the progressbar.

• pretty: Shows all found items in realtime nicely colored. This formatter is activated as default.

• summary: Shows counts of files and their respective size after the run. Also list all written output files.

• fdupes: Prints an output similar to the popular duplicate finder fdupes(1). At first a progressbar is printed on stderr. Afterwards the found files are printed on stdout; each set of duplicates gets printed as a block separated by newlines. Originals are highlighted in green. At the bottom a summary is printed on stderr. This is mostly useful for scripts that were set up for parsing fdupes output. We recommend the json formatter for every other scripting purpose.

  Available options:

  • omitfirst: Same as the -f / --omitfirst option in fdupes(1). Omits the first line of each set of duplicates (i.e. the original file.
  • sameline: Same as the -1 / --sameline option in fdupes(1). Does not print newlines between files, only a space. Newlines are printed only between sets of duplicates.

EXAMPLES¶

This is a collection of common usecases and other tricks:

• Check the current working directory for duplicates.

  $ rmlint

• Show a progressbar:

  $ rmlint -g

• Quick re-run on large datasets using different ranking criteria on second run:

  $ rmlint large_dir/ # First run; writes rmlint.json

  $ rmlint --replay rmlint.json large_dir -S MaD

• Search only for duplicates and duplicate directories

  $ rmlint -T "df,dd" .

• Compare files byte-by-byte in current directory:

  $ rmlint -pp .

• Find duplicates with same basename (excluding extension):

  $ rmlint -e

• Do more complex traversal using find(1).

  $ find /usr/lib -iname '*.so' -type f | rmlint - # find all duplicate .so files

  $ find ~/pics -iname '*.png' | ./rmlint - # compare png files only

• Limit file size range to investigate:

  $ rmlint -s 2GB    # Find everything >= 2GB

  $ rmlint -s 0-2GB  # Find everything <  2GB

• Only find writable and executable files:

  $ rmlint --perms wx

• Reflink on btrfs, else try to hardlink duplicates to original. If that does not work, replace duplicate with a symbolic link:

  $ rmlint -c sh:link

• Inject user-defined command into shell script output:

  $ ./rmlint -o sh -c sh:cmd='echo "original:" "$2" "is the same as" "$1"'

• Use data as master directory. Find only duplicates in backup that are also in data. Do not delete any files in data:

  $ rmlint backup // data --keep-all-tagged --must-match-tagged

PROBLEMS¶

1. False Positives: Depending on the options you use, there is a very slight risk of false positives (files that are erroneously detected as duplicate). The default hash function (SHA1) is pretty safe but in theory it is possible for two files to have then same hash. This happens about once in 2 ** 80 files, so it is very very unlikely. If you're concerned just use the --paranoid (-pp) option. This will compare all the files byte-by-byte and is not much slower than SHA1.
2. File modification during or after rmlint run: It is possible that a file that rmlint recognized as duplicate is modified afterwards, resulting in a different file. If you use the rmlint-generated shell script to delete the duplicates, you can run it with the -p option to do a full re-check of the duplicate against the original before it deletes the file.

SEE ALSO¶

• find(1)
• rm(1)
• cp(1)

Extended documentation and an in-depth tutorial can be found at:

• http://rmlint.rtfd.org

BUGS¶

If you found a bug, have a feature requests or want to say something nice, please visit https://github.com/sahib/rmlint/issues.

Please make sure to describe your problem in detail. Always include the version of rmlint (--version). If you experienced a crash, please include at least one of the following information with a debug build of rmlint:

• gdb --ex run -ex bt --args rmlint -vvv [your_options]
• valgrind --leak-check=no rmlint -vvv [your_options]

You can build a debug build of rmlint like this:

• git clone git@github.com:sahib/rmlint.git
• cd rmlint
• scons DEBUG=1
• sudo scons install  # Optional

LICENSE¶

rmlint is licensed under the terms of the GPLv3.

See the COPYRIGHT file that came with the source for more information.

PROGRAM AUTHORS¶

rmlint was written by:

• Christopher <sahib> Pahl 2010-2015 (https://github.com/sahib)
• Daniel <SeeSpotRun> T. 2014-2015 (https://github.com/SeeSpotRun)

Also see the http://rmlint.rtfd.org for other people that helped us.

If you consider a donation you can use Flattr or buy us a beer if we meet:

https://flattr.com/thing/302682/libglyr