Dependencies¶

Compilation¶

Compilation consists of getting the source and translating it into a usable binary:

$ # Omit -b develop if you want to build from the stable master
$ git clone -b develop https://github.com/sahib/rmlint.git
$ cd rmlint/
$ scons config       # Look what features scons would compile
$ scons DEBUG=1 -j4  # For releases you can omit DEBUG=1
$ sudo scons DEBUG=1 --prefix=/usr install

Done!

You should be now able to see the manpage with rmlint --help or man 1 rmlint.

Beginner Examples¶

Let's just dive in into some examples:

$ rmlint

This simply scans your current working directory for lint and reports them in your terminal. Note that nothing will be removed (even if it prints rm).

Despite it's name, rmlint just finds suspicious files, but never modifies the filesystem itself [*]. Instead it gives you detailed reports in different formats to get rid of them yourself. These reports are called outputs. By default a shellscript will be written to rmlint.sh that contains readily prepared shell commands to remove duplicates and other finds,

So for the above example, the full process would be:

$ rmlint some/path
# (wait for rmlint to finish running)
$ gedit rmlint.sh
# (or any editor you prefer... review the content of rmlint.sh to
#  check what it plans to delete; make any edits as necessary)
$ ./rmlint.sh
# (the rmlint.sh script will ask for confirmation, then delete the
#  appropriate lint, then delete itself)

On larger runs, it might be more preferable to show a progressbar instead of a long list of files. You can do this easily with the -g switch:

$ rmlint -g

Filtering input files¶

What if we do not want to check all files as dupes? rmlint has a good reportoire of options to select only certain files. We won't cover all options, but the useful ones. If those options do not suffice, you can always use external tools to feed rmlint's stdin:

$ find pics/ -iname '*.png' | rmlint -

# only check files between 20 MB and 1 Gigabyte:
$ rmlint --size 20M-1G
# short form (-s) works just as well:
$ rmlint -s 20M-1G
# only check files bigger than 4 kB:
$ rmlint -s 4K
# only check files smaller than 1234 bytes:
$ rmlint -s 0-1234

# Find all duplicate files with the same basename:
$ rmlint -b some_dir/
ls some_dir/one/hello.c
rm some_dir/two/hello.c
# Find all duplicate files that have the same extension:
$ rmlint -e some_dir/
ls some_dir/hello.c
rm some_dir/hello_copy.c
# Find all duplicate files that have the same basename:
# minus the extension
$ rmlint -e some_dir/
ls some_dir/hello.c
rm some_dir/hello.bak

# Use a Unix-UTC Timestamp (seconds since epoch)
$ rmlint -N 1414755960

# Find all files newer than file.png
$ rmlint -N $(stat --print %Y file.png)

# Alternatively use a ISO8601 formatted Timestamp
$ rmlint -N 2014-09-08T00:12:32+0200

# First run of rmlint:
$ rmlint /home/foobar -O stamp:/home/foobar/.rmlint.stamp
ls /home/foobar/a.file
rm /home/foobar/b.file

# Second run, no changes:
$ rmlint /home/foobar -n /home/foobar/.rmlint.stamp
<nothing>

# Second run, new file copied:
$ cp /home/foobar/a.file /home/foobar/c.file
$ rmlint /home/foobar -n /home/foobar/.rmlint.stamp
ls /home/foobar/a.file
rm /home/foobar/b.file
rm /home/foobar/c.file

Outputs & Formatters¶

rmlint is capable to create it's reports in several output-formats. Actually if you run it with the default options you already see two of those formatters: Namely pretty and summary.

Formatters can be added via the -O (--add-output) switch. The -o (--output) instead clears all defaults first and does the same as -O afterwards.

$ rmlint -g -VVV /usr  # -VVV is just to prevent pointless warnings

Here's an example:

$ rmlint -o json:stderr

Here you would get this output printed on stderr:

[{
  "description": "rmlint json-dump of lint files",
  "cwd": "/home/user/",
  "args": "rmlint -o json:stderr"
},
{
  "type": "duplicate_file",
  "path": "/home/user/test/b/one",
  "size": 2,
  "inode": 2492950,
  "disk_id": 64771,
  "progress": 100,
  "is_original": true,
  "mtime": 1414587002
},
... snip ...
{
  "aborted": false,
  "total_files": 145,
  "ignored_files": 9,
  "ignored_folders": 4,
  "duplicates": 11,
  "duplicate_sets": 2,
  "total_lint_size": 38
}]

You probably noticed the colon in the commandline above. Everything before it is the name of the output-format, everything behind is the path where the output should land. Instead of an path you can also use stdout and stderr, as we did above or just omit the colon which will print everything to stdout.

Some formatters might be configured to generate subtly different output using the -c (--config) command. Here's the list of currently available formatters and their config options:

$ rmlint -o sh:stdout
#!/bin/sh
# This file was autowritten by rmlint
# rmlint was executed from: /home/user/
# You command line was: ./rmlint -o sh:rmlint.sh

# ... snip ...

echo  '/home/user/test/b/one' # original
remove_cmd '/home/user/test/b/file' # duplicate
remove_cmd '/home/user/test/a/two' # duplicate
remove_cmd '/home/user/test/a/file' # duplicate

if [ -z $DO_REMOVE ]
then
  rm -f 'rmlint.sh';
fi

$ rmlint -o sh:stdout -o sh:rmlint.sh -c sh:link
...
echo  '/home/user/test/b/one' # original
cp_symlink '/home/user/test/b/file' '/home/user/test/b/one' # duplicate
$ ./rmlint.sh -d
Keeping: /home/user/test/b/one
Symlinking to original: /home/user/test/b/file

$ rmlint -o sh -c sh:cmd='echo "Stuff with" "$1" "->" "$2"'

$ rmlint -o py:remover.py
$ ./remover.py --dry-run    # Needs Python3
Deleting twins of /home/user/sub2/a
Handling (duplicate_file): /home/user/sub1/a
Handling (duplicate_file): /home/user/a

Deleting twins of /home/user/sub2/b
Handling (duplicate_file): /home/user/sub1/b

$ rmlint -o csv -D
type,path,size,checksum
emptydir,"/home/user/tree2/b",0,00000000000000000000000000000000
duplicate_dir,"/home/user/test/b",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/test/a",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/tree/b",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree/a",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree2/a",4,311095bc5669453990cd205b647a1a00

Paranoia mode¶

Let's face it, why should you trust rmlint?

Technically it only computes a hash of your file which might, by it's nature, collide with the hash of a totally different file. If we assume a perfect hash function (i.e. one that distributes it's hash values perfectly even over all possible values), the probablilty of having a hash-collision is \(\frac{1}{2^{160}}\) for the default 160-bit hash. Of course hash functions are not totally random, so the collision probability is slightly higher. Due to the "birthday paradox", this starts to become a real risk if you have more than about \(2^{80}\) files.

If you're wary, you might want to make a bit more paranoid than the default. By default the sha1 hash algorithm is used, which we consider a good trade-off of speed and accuracy. rmlint's paranoia level can be easily inc/decreased using the -p (--paranoid)/ -P (--less-paranoid) option (which might be given twice each).

Here's what they do in detail:

• -p is equivalent to --algorithm=sha512
• -pp is equivalent to --algorithm=paranoid

As you see, it just enables a certain hash algorithm. --algorithm changes the hash algorithm to someting more secure. One level up the well-known sha512 (with 512bits obviously) is used. Another level up, no hash function is used. Instead, files are compared byte-by-byte (which guarantees collision free output). While this might sound slow it's often only a few seconds slower than the default behaviour.

There is a bunch of other hash functions you can lookup in the manpage. We recommend never to use the -P option.

Original detection / selection¶

As mentioned before, rmlint divides a group of dupes in one original and clones of that one. While the chosen original might not be the one that was there first, it is a good thing to keep one file of a group to prevent dataloss.

By default, if you specify multiple paths in the rmlint command, the files in the first-named paths are treated as the originals. If there are two files in the same path, then the older one will be treated as the original. If they have the same modification time then it's just a matter of chance which one is selected as the original.

The way rmlint chooses the original can be driven by the -S (--sortcriteria) option.

Here's an example:

# Normal run:
$ rmlint
ls c
rm a
rm b

# Use alphabetically first one as original
$ rmlint -S a
ls a
rm b
rm c

Alphabetically first makes sense in the case of backup files, ie a.txt.bak comes after a.txt.

Here's a table of letters you can supply to the -S option:

The default setting is -S pm. Multiple sort criteria can be specified, eg -S mpa will sort first by mtime, then (if tied), based on which path you specified first in the rmlint command, then finally based on alphabetical order of file name. Note that "original directory" criteria (see below) take precedence over any -S options.

$ rmlint a // b
ls b/file
rm a/file

# Find all files on /media/portable that can be safely deleted:
$ rmlint -km /media/portable // ~
# check the shellscript looks ok:
$ less ./rmlint.sh
# run the shellscript to delete the redundant backups
$ ./rmlint.sh
# run again (to delete empty dirs)
$ rmlint -km /media/portable // ~
$ ./rmlint.sh
# see what files are left:
$ tree /media/portable
# recover any files that you want to save, then you can safely reformat the drive

Finding duplicate directories¶

As far as we know, rmlint is the only duplicate finder that can do this. Basically, all you have to do is to specify the -D (--merge-directories) option and rmlint will cache all duplicates until everything is found and then merge them into full duplicate directories (if any). All other files are printed normally.

This may sound simple after all, but there are some caveats you should know of.

Let's create a tricky folder structure to demonstrate the feature:

$ mkdir -p fake/one/two/ fake/one/two_copy fake/one_copy/two fake/one_copy/two_copy
$ echo xxx > fake/one/two/file
$ echo xxx > fake/one/two_copy/file
$ echo xxx > fake/one_copy/two/file
$ echo xxx > fake/one_copy/two_copy/file
$ echo xxx > fake/file
$ echo xxx > fake/another_file

Now go run rmlint on it like that:

$ rmlint fake -D -S a
# Duplicate Directorie(s):
    ls -la /home/sahib/rmlint/fake/one
    rm -rf /home/sahib/rmlint/fake/one_copy
    ls -la /home/sahib/rmlint/fake/one/two
    rm -rf /home/sahib/rmlint/fake/one/two_copy

# Duplicate(s):
    ls /home/sahib/rmlint/fake/another_file
    rm /home/sahib/rmlint/fake/one/two/file
    rm /home/sahib/rmlint/fake/file

==> In total 6 files, whereof 5 are duplicates in 1 groups.
==> This equals 20 B of duplicates which could be removed.

As you can see it correctly recognized the copies as duplicate directories. Also, it did not stop at fake/one but also looked at what parts of this original directory could be possibly removed too.

Files that could not be merged into directories are printed separately. Note here, that the original is taken from a directory that was preserved. So exactly one copy of the xxx-content file stays on the filesystem in the end.

rmlint finds duplicate directories by counting all files in the directory tree and looking up if there's an equal amount of duplicate and empty files. If so, it tries the same with the parent directory.

Some file like hidden files will not be recognized as duplicates, but still added to the count. This will of course lead to unmerged directories. That's why the -D option implies the -r (--hidden) and -l (--hardlinked) option in order to make this convenient.

A note to symbolic links: The default behaviour is to not follow symbolic links, but to compare the link targets. If the target is the same, the link will be the same. This is a sane default for duplicate directories, since twin copies often are created by doing a backup of some files. In this case any symlinks in the backupped data will still point to the same target. If you have symlinks that reference a file in each respective directory tree, consider using -f.

Sometimes it might be nice to only search for duplicate directories, banning all the sole files from littering the screen. While this will not delete all files, it will give you a nice overview of what you copied where.

Since duplicate directories are just a lint type as every other, you can just pass it to -T: -T "none +dd" (or -T "none +duplicatedirs"). There's also a preset of it to save you some typing: -T minimaldirs.

Miscellaneous options¶

If you read so far, you know rmlint pretty well by now. Here's just a list of options that are nice to know, but not essential:

• Consecutive runs of rmlint can be speed up by using --cache.

  $ rmlint large_dataset/ -O json:cache.json --write-unfinished
  $ rmlint large_dataset/ -C cache.json
  

  Here, the second run should (or might) run a lot faster. But be sure to read the caveats stated in the manpage!

• -r (--hidden): Include hidden files and directories - this is to save you from destroying git repositories (or similar programs) that save their information in a .git directory where rmlint often finds duplicates.

  If you want to be safe you can do something like this:

  $ # find all files except everything under .git or .svn folders
  $ find . -type d | grep -v '\(.git\|.svn\)' | rmlint - --hidden
  

  But you would have checked the output anyways, wouldn't you?

• If something ever goes wrong, it might help to increase the verbosity with -v (up to -vvv).

• Usually the commandline output is colored, but you can disable it explicitly with -w (--with-color). If stdout or stderr is not an terminal anyways, rmlint will disable colors itself.

• You can limit the traversal depth with -d (--max-depth):

  $ rmlint -d 0
  <finds everything in the same working directory>
  

• If you want to prevent rmlint from crossing mountpoints (e.g. scan a home directory, but no the HD mounted in there), you can use the -X (--no-crossdev) option.

• It is possible to tell rmlint that it should not scan the whole file. With -q (--clamp-low) / -Q (--clamp-top) it is possible to limit the range to a starting point (-q) and end point (-Q). The point where to start might be either given as percent value, factor (percent / 100) or as an absolute offset.

  If the file size is lower than the absolute offset, the file is simply ignored.

  This feature might prove useful if you want to examine files with a constant header. The constant header might be different, i.e. by a different ID, but the content might be still the same. In any case it is advisable to use this option with care.

  Example:

  # Start hashing at byte 100, but not more than 90% of the filesize.
  $ rmlint -q 100 -Q .9
  

rmlint finds more/less dupes than tool X!¶

Make sure that none of the following applies to you:

Both tools might investigate a different number of files. rmlint e.g. does not look through hidden files by default, while other tools might follow symlinks by default. Suspicious options you should look into are:

• --hidden: Disabled by default, since it might screw up .git/ and similar directories.
• --hardlinked: Might find larger amount files, but not more lint itself.
• --followlinks: Might lead rmlint to different places on the filesystem.
• --merge-directories: pulls in both --hidden and --hardlinked.

If there's still a difference, check with another algorithm. In particular use -pp to enable paranoid mode. Also make sure to have -D (--merge-directories) disabled to see the raw number of duplicate files.

Still here? Maybe talk to us on the issue tracker.

Can you implement feature X?¶

Depends. Go to to the issue tracker and open a feature request.

Here is a list of features where you probably have no chance:

• Port it to Windows.
• Find similar files like ssdeep does.

I forgot to add some options before running on a large dataset. Do I need to re-run it?¶

Probably. It's not as bad as it sounds though. Your filesystem is probably very good at caching.

Still there are some cases where re-running might take a long time, like running on network mounts. By default, rmlint writes a rmlint.json file along the rmlint.sh. This can be used to speed up the next run by passing it to --cache. It should be noted that using the cache file for later runs is discouraged since false positives will get likely if the data is changed in between. Therefore there will never be an "auto-pickup" of the cache file.

I have a very large number of files and I run out of memory and patience.¶

As a rule of thumb, rmlint will allocate ~150 bytes for every file it will investigate. Additionally paths are stored in a patricia trie, which will compress paths and save memory therefore.

The memory peak is usually short after it finished traversing all files. For example, 5 million files will result in a memory footprint of roughly 1.0GB of memory in average.

If that's still not enough read on.

Some things to consider:

• Use --with-metadata-cache to swap paths to disk. When needed the path is selected from disk instead of keeping them all in memory. This lowers the memory footprint per file by a few bytes. Sometimes the difference may be very subtle since all paths in rmlint are stored by common prefix, i.e. for long but mostly identically paths the point after the difference is stored.

  This option will most likely only make sense if you files with long basenames. You might expect 10%-20% less memory as a rule of thumb.

• Use --without-fiemap on rotational disk to disable this optimization. With it enabled a table of the file's extents is stored to optimize disk access patterns. This lowers the memory footprint per file by around 50 bytes.

• Enable the progress bar with -g to keep track of how much data is left to scan.

Caveats:

• Some options like -D will not work well with --with-metadata-cache and use a fair bit of memory themselves. This is by the way they're working. Avoid them in this case. Also --cache might be not very memory efficient.
• The CPU usage might go up quite a bit, resulting longer runs.

Also:

rmlint have been successfully used on datasets of 5 million files. See this bug report for more information: #109.

Bugs¶

Please use the issue tracker to post and discuss bugs and features:

https://github.com/sahib/rmlint/issues

Philosophy¶

We try to adhere to some principles when adding features:

• Try to stay compatible to standard unix' tools and ideas.
• Try to stay out of the users way and never be interactive.
• Try to make scripting as easy as possible.
• Never make rmlint modify the filesystem itself, only produce output to let the user easily do it.

Also keep this in mind, if you want to make a feature request.

Making contributions¶

The code is hosted on GitHub, therefore our preferred way of receiving patches is using GitHub's pull requests (normal git pull requests are okay too of course).

Here's a short step-by-step:

1. Fork it.
2. Create a branch from develop. (git checkout develop && git checkout -b my_feature)
3. Commit your changes. (git commit -am "Fixed it all.")
4. Check if your commit message is good. (If not: git commit --amend)
5. Push to the branch (git push origin my_feature)
6. Open a Pull Request.
7. Enjoy a refreshing Tea and wait until we get back to you.

Here are some other things to check before submitting your contribution:

• Does your code look alien to the other code? Is the style the same? You can run this command to make sure it is the same:

  $ clang-format -style=file -i $(find lib src -iname '*.[ch]')
  

• Do all tests run? Go to the test documentation for more info. Also after opening the pull request, your code will be checked via TravisCI.

• Is your commit message descriptive? whatthecommit.com has some good examples how they should not look like.

• Is rmlint running okay inside of valgrind (i.e. no leaks and no memory violations)?

For language-translations/updates it is also okay to send the .po files via mail at sahib@online.de, since not every translator is necessarily a software developer.

Testsuite¶

rmlint has a not yet complete but quite powerful testsuite. It is not complete yet (and probably never will), but it's already an valueable boost of confidence in rmlint's correctness.

The tests are based on nosetest and are written in python>=3.0. Every testcase just runs the (previously built) rmlint binary a and parses it's json output. So they are technically blackbox-tests.

On every commit, those tests are additionally run on TravisCI.

$ sudo USE_VALGRIND=1 PRINT_CMD=1 PEDANTIC=1 nosetests-3.4 -s -a '!slow'

$ CFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS="-fprofile-arcs -ftest-coverage" scons -j4 DEBUG=1
$ sudo USE_VALGRIND=1 PRINT_CMD=1 PEDANTIC=1 nosetests-3.4 -s -a '!slow'
$ lcov --capture --directory . --output-file coverage.info
$ genhtml coverage.info --output-directory out

tests
├── test_formatters   # Tests for output formatters (like sh or json)
├── test_options      # Tests for normal options like --merge-directories etc.
├── test_types        # Tests for all lint types rmlint can find
└── utils.py          # Common utilities shared amon tests.

from nose import with_setup
from tests.utils import *

@with_setup(usual_setup_func, usual_teardown_func)
def test_basic():
    create_file('xxx', 'a')
    create_file('xxx', 'b')

    head, *data, footer = run_rmlint('-a city -S a')

    assert footer['duplicate_sets'] == 1
    assert footer['total_lint_size'] == 3
    assert footer['total_files'] == 2
    assert footer['duplicates'] == 1

Buildsystem Helpers¶

# Use clang and enable profiling, verbose build and enable debugging
CC=clang CFLAGS='-pg' LDFLAGS='-pg' scons VERBOSE=1 DEBUG=1

$ USE_VALGRIND=1 nosetests  # or nosetests-3.3, python3 needed.

Sourcecode layout¶

• All C-source lives in lib, the file names should be self explanatory.
• As an exception, the main lives in src/rmlint.c.
• All documentation is inside docs.
• All translation stuff should go to po.
• All packaging should be done in pkg/<distribution>.
• Tests are written in Python and live in tests.

Hashfunctions¶

Here is a short comparasion of the existing hashfunctions in rmlint (linear scale). For reference: Those plots were rendered with these sources - which are very ugly, sorry.

If you want to add new hashfunctions, you should have some arguments why it is valueable and possiblye even benchmark it with the above scripts to see if it's really that much faster.

Also keep in mind that most of the time the hashfunction is not the bottleneck.

Optimizations¶

For sake of overview, here is a short list of optimizations implemented in rmlint:

Adding new languages¶

# Fork a new .po file from the po-template (here swedish):
$ msginit -i po/rmlint.pot -o po/se.po --locale se --no-translator

# Edit the po/se.po file, the format is self describing
$ vim po/se.po

# .po files need to be compiled, but that's handled by scons already.
$ scons
$ scons install

# You should see your changes now:
$ LANG=se ./rmlint

If you'd like to contribute your new translation you want to do a pull request (if you really dislike that, you may also send the translation to us via mail). Here is a small introduction on Pull Requests.

Updating existing languages¶

# Edit the file to your needs:
$ vim po/xy.po

# Install:
$ scons install

# Done
$ LANG=xy ./rmlint

Marking new strings for translations¶

If you want to mark strings in the C-code to be translated, you gonna need to mark them so the xgettext can find it. The latter tool goes through the source and creates a template file with all translations left out.

/* Mark the string with the _() macro */
fprintf(out, _("Stuff is alright: %s\n"), (alright) ? "yes" : "no");

It gets a little harder when static strings need to be marked, since they cannot be translated during compile time. You have to mark them first and translate them at a later point:

static const char * stuff = _N("Hello World");

void print_world(void) {
    printf("World is %s\n", _(stuff));
}

After you're done with marking the new strings, you have to update the template:

# scons can do this for you already:
$ scons xgettext

You need to add the new strings to the existing translations now:

$ msgmerge po/de.po po/rmlint.pot > po/de_new.po
$ EDITOR po/de_new.po   # check if everything was merged alright.
$ mv po/de_new.po po/de.po

After that you can translate the new strings and proceed like in the upper steps.

find duplicate files and other space waste efficiently¶

# -ed is recognized as -e and -d here, -d takes "-s 10M" as parameter.
# This will fail to do the supposed, finding also files smaller than 10M.
$ rmlint -T all -ef -ed -s10M /media/music/
# Actual user wanted to do this:
$ rmlint -T "all -ef -ed" -s10M /media/music

$ rmlint large_cluster/ -O json:cache.json -U   # first run.
$ rmlint large_cluster/ -C cache.json           # second run.