Dependencies¶

Compilation¶

Compilation consists of getting the source and translating it into a usable binary:

$ git clone -b develop https://github.com/sahib/rmlint.git
$ cd rmlint/
$ scons config       # Look what features scons would compile
$ scons DEBUG=1 -j4  # For releases you can omit DEBUG=1
$ sudo scons DEBUG=1 --prefix=/usr install

Done!

You should be now able to see the manpage with rmlint -h or man 1 rmlint.

Beginner Examples¶

Let's just dive in into some examples:

$ rmlint

This simply scans your current working directory for lint and reports them in your terminal. Note that nothing will be removed (even if it prints rm).

Despite it's name, rmlint just finds suspicious files, but never modifies the filesystem itself [*]. Instead it gives you detailed reports in different formats to get rid of them yourself. These reports are called outputs. By default a shellscript will be written to rmlint.sh that contains readily prepared shell commands to remove duplicates and other finds,

So for the above example, the full process would be:

`-FF`` together with this option. rmlint
# (wait for rmlint to finish running)
$ gedit rmlint.sh
# (or any editor you prefer... review the content of rmlint.sh to
#  check what it plans to delete; make any edits as necessary)
$ ./rmlint.sh
# (the rmlint.sh script will ask for confirmation, then delete the
#  appropriate lint, then delete itself)

Filtering¶

What if we do not want to check all files as dupes? rmlint has a good reportoire of options to select only certain files. We won't cover all options, but the useful ones. If those options do not suffice, you can always use external tools to feed rmlint's stdin:

$ find pics/ -iname '*.png' | rmlint -

# only check files between 20 MB and 1 Gigabyte:
$ rmlint --size 20M-1G
# short form (-s) works just as well:
$ rmlint -s 20M-1G
# only check files bigger than 4 kB:
$ rmlint -s 4K
# only check files smaller than 1234 bytes:
$ rmlint -s 0-1234

# Find all duplicate files with the same basename:
$ rmlint -b some_dir/
ls some_dir/one/hello.c
rm some_dir/two/hello.c
# Find all duplicate files that have the same extension:
$ rmlint -e some_dir/
ls some_dir/hello.c
rm some_dir/hello_copy.c
# Find all duplicate files that have the same basename:
# minus the extension
$ rmlint -e some_dir/
ls some_dir/hello.c
rm some_dir/hello.bak

# Use a Unix-UTC Timestamp (seconds since epoch)
$ rmlint -N 1414755960%

# Find all files newer than file.png
$ rmlint -N $(stat --print %Y file.png)

# Alternatively use a ISO8601 formatted Timestamp
$ rmlint -N 2014-09-08T00:12:32+0200

# First run of rmlint:
$ rmlint /home/foobar -O stamp:/home/foobar/.rmlint.stamp
ls /home/foobar/a.file
rm /home/foobar/b.file

# Second run, no changes:
$ rmlint /home/foobar -n /home/foobar/.rmlint.stamp
<nothing>

# Second run, new file copied:
$ cp /home/foobar/a.file /home/foobar/c.file
$ rmlint /home/foobar -n /home/foobar/.rmlint.stamp
ls /home/foobar/a.file
rm /home/foobar/b.file
rm /home/foobar/c.file

Outputs¶

rmlint is capable to create it's reports in several output-formats. Actually if you run it with the default options you already see two of those formatters: Namely pretty and summary.

Formatters can be added via the -O (--add-output) switch. The -o (--output) instead clears all defaults first and does the same as -O afterwards.

Here's an example:

$ rmlint -o json:stderr

Here you would get this output printed on stderr:

[{
  "description": "rmlint json-dump of lint files",
  "cwd": "/home/user/",
  "args": "rmlint -o json:stderr"
},
{
  "type": "duplicate_file",
  "path": "/home/user/test/b/one",
  "size": 2,
  "inode": 2492950,
  "disk_id": 64771,
  "is_original": true,
  "mtime": 1414587002
},
... snip ...
{
  "aborted": false,
  "total_files": 145,
  "ignored_files": 9,
  "ignored_folders": 4,
  "duplicates": 11,
  "duplicate_sets": 2,
  "total_lint_size": 38
}]

You probably noticed the colon in the commandline above. Everything before it is the name of the output-format, everything behind is the path where the output should land. Instead of an path you can also use stdout and stderr, as we did above.

Some formatters might be configured to generate subtly different output using the -c (--config) command. Here's the list of currently available formatters and their config options:

$ rmlint -o sh:stdout
#!/bin/sh
# This file was autowritten by rmlint
# rmlint was executed from: /home/user/
# You command line was: ./rmlint -o sh:rmlint.sh

# ... snip ...

echo  '/home/user/test/b/one' # original
rm -f '/home/user/test/b/file' # duplicate
rm -f '/home/user/test/a/two' # duplicate
rm -f '/home/user/test/a/file' # duplicate

if [ -z $DO_REMOVE ]
then
  rm -f 'rmlint.sh';
fi

$ rmlint -o sh:stdout -o sh:rmlint.sh -c sh:use_ln=true -c sh:symlinks_only=true
...
echo  '/home/user/test/b/one' # original
ln -s '/home/user/test/b/file' # duplicate
$ ./rmlint.sh -d
/home/user/test/b/one

$ rmlint -o py:remover.py
$ ./remover.py --dry-run    # Needs Python3
Deleting twins of /home/user/sub2/a
Handling (duplicate_file): /home/user/sub1/a
Handling (duplicate_file): /home/user/a

Deleting twins of /home/user/sub2/b
Handling (duplicate_file): /home/user/sub1/b

$ rmlint -o csv -D
type,path,size,checksum
emptydir,"/home/user/tree2/b",0,00000000000000000000000000000000
duplicate_dir,"/home/user/test/b",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/test/a",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/tree/b",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree/a",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree2/a",4,311095bc5669453990cd205b647a1a00

Paranoia¶

Let's face it, why should you trust rmlint?

Technically it only computes a hash of your file which might, by it's nature, collide with the hash of a totally different file. If we assume a perfect hash function (i.e. one that distributes it's hash values perfectly even over all possible values), the probablilty of having a hash-collision is \(\frac{1}{2^{128}}\) for the default 128-bit hash. Of course hash functions are not totally random, so the collision probability is slightly higher.

If you're wary you might want to make a bit more paranoid than it's default. By default the spooky hash algorithm is used, which we consider a good tradeoff of speed and accuracy. rmlint's paranoia level can be easily inc/decreased using the -p (--paranoid)/ -P (--less-paranoid) option (which might be given up to three times each).

Here's what they do in detail:

• -p is equivalent to --algorithm=bastard
• -pp is equivalent to --algorithm=sha512
• -ppp is equivalent to --algorithm=paranoid

As you see, it just enables a certain hash algorithm. --algorithm changes the hash algorithm to someting more secure. bastard is a 256bit hash that consists of two 128bit subhashes (murmur3 and city if you're curious). One level up the well-known sha512 (with 512bits obviously) is used. Another level up, no hash function is used. Instead, files are compared byte-by-byte (which guarantees collision free output).

There is a bunch of other hash functions you can lookup in the manpage. We recommend never to use the -P option.

Original detection¶

As mentioned before, rmlint divides a group of dupes in one original and clones of that one. While the chosen original might not be the one that was there first, it is a good thing to keep one file of a group to prevent dataloss.

The way rmlint chooses the original can be driven by the -S (--sortcriteria) option.

Here's an example:

# Normal run:
$ rmlint
ls c
rm a
rm b

# Use alphabetically first one as original
$ rmlint -S a
ls a
rm b
rm c

Alphabetically first makes sense in the case of backup files, ie a.txt.bak comes after a.txt.

Here's a table of letters you can supply to the -S option:

The default setting is -S m -- which takes the oldest file, determined by it's modification time. Multiple sort criteria can be specified, eg -S mpa will sort first by mtime, then (if tied), based on which path you specified first in the rmlint command, then finally based on alphabetical order of file name. Note that "original directory" criteria (see below) take precedence over the -S options.

$ rmlint a // b
ls b/file
rm a/file

# Find all files on /media/portable that can be safely deleted:
$ rmlint -km /media/portable // ~
# check the shellscript looks ok:
$ less ./rmlint.sh
# run the shellscript to delete the redundant backups
$ ./rmlint.sh
# run again (to delete empty dirs)
$ rmlint -km /media/portable // ~
$ ./rmlint.sh
# see what files are left:
$ tree /media/portable
# recover any files that you want to save, then you can safely reformat the drive

Finding duplicate directories¶

As far as we know, rmlint is the only duplicate finder that can do this. Basically, all you have to do is to specify the -D (--merge-directories) option and rmlint will cache all duplicate until everything is found and then merge them into full duplicate directories (if any). All other files are printed normally.

This may sound simple after all, but there are some caveats you should know of.

Let's create a tricky folder structure to demonstrate the feature:

$ mkdir -p fake/one/two/ fake/one/two_copy fake/one_copy/two fake/one_copy/two_copy
$ echo xxx > fake/one/two/file
$ echo xxx > fake/one/two_copy/file
$ echo xxx > fake/one_copy/two/file
$ echo xxx > fake/one_copy/two_copy/file
$ echo xxx > fake/file
$ echo xxx > fake/another_file

Now go run rmlint on it like that:

$ rmlint fake -D -S a
# Duplicate Directorie(s):
    ls -la /home/sahib/rmlint/fake/one
    rm -rf /home/sahib/rmlint/fake/one_copy
    ls -la /home/sahib/rmlint/fake/one/two
    rm -rf /home/sahib/rmlint/fake/one/two_copy

# Duplicate(s):
    ls /home/sahib/rmlint/fake/another_file
    rm /home/sahib/rmlint/fake/one/two/file
    rm /home/sahib/rmlint/fake/file

==> In total 6 files, whereof 5 are duplicates in 1 groups.
==> This equals 20 B of duplicates which could be removed.

As you can see it correctly recognized the copies as duplicate directories. Also, it did not stop at fake/one but also looked at what parts of this original directory could be possibly removed too.

Files that could not be merged into directories are printed separately. Note here, that the original is taken from a directory that was preserved. So exactly one copy of the xxx-content file stays on the filesystem in the end.

rmlint finds duplicate directories by counting all files in the directory tree and looking up if there's an equal amount of duplicate and empty files. If so, it tries the same with the parent directory.

Some file like hidden files will not be recognized as duplicates, but still added to the count. This will of course lead to unmerged directories. That's why the -D option implies the -r (--hidden) and -l (--hardlinked) option in order to make this convenient.

A note to symbolic links: The default behaviour is to not follow symbolic links, but to compare the link targets. If the target is the same, the link will be the same. This is a sane default for duplicate directories, since twin copies often are created by doing a backup of some files. In this case any symlinks in the backupped data will still point to the same target. If you have symlinks that reference a file in each respective directory tree, consider using -f.

Sometimes it might be nice to only search for duplicate directories, banning all the sole files from littering the screen. While this will not delete all files, it will give you a nice overview of what you copied where.

Since duplicate directories are just a lint type as every other, you can just pass it to -T: -T "none +dd" (or -T "none +duplicatedirs"). There's also a preset of it to save you some typing: -T minimaldirs.

Misc options¶

If you read so far, you know rmlint pretty well by now. Here's just a list of options that are nice to know, but not essential:

• -r (--hidden): Include hidden files and directories - this is to save you from destroying git repositories (or similar programs) that save their information in a .git directory where rmlint often finds duplicates.

  If you want to be safe you can do something like this:

  $ find . | grep -v '\(.git\|.svn\)' | rmlint -
  

  But you would have checked the output anyways?

• If something ever goes wrong, it might help to increase the verbosity with -v (up to -vvv).

• Usually the commandline output is colored, but you can disable it explicitly with -w (--with-color). If stdout or stderr is not an terminal anyways, rmlint will disable colors itself.

• You can limit the traversal depth with -d (--max-depth):

  $ rmlint -d 0
  <finds everything in the same working directory>
  

• If you want to prevent rmlint from crossing mountpoints (e.g. scan a home directory, but no the HD mounted in there), you can use the -X (--no-crossdev) option.

• It is possible to tell rmlint that it should not scan the whole file. With -q (--clamp-low) / -Q (--clamp-top) it is possible to limit the range to a starting point (-q) and end point (-Q). The point where to start might be either given as percent value, factor (percent / 100) or as an absolute offset.

  If the file size is lower than the absolute offset, the file is simply ignored.

  This feature might prove useful if you want to examine files with a constant header. The constant header might be different, i.e. by a different ID, but the content might be still the same. In any case it is advisable to use this option with care.

  Example:

  # Start hashing at byte 100, but not more than 90% of the filesize.
  $ rmlint -q 100 -Q.9
  

rmlint finds more/less dupes than tool X!¶

Make sure that none of the following applies to you:

Both tools might investigate a different number of files. rmlint e.g. does not look through hidden files by default, while other tools might follow symlinks by default. Suspicious options you should look into are:

• --hidden: Disabled by default, since it might screw up .git/ and similar directories.
• --hardlinked: Might find larger amount files, but not more lint itself.
• --followlinks: Might lead rmlint to different places on the filesystem.
• --merge-directories: pulls in both --hidden and --hardlinked.

If there's still a difference, check with another algorithm. In particular use -ppp to enable paranoid mode. Also make sure to have -D (--merge-directories) disabled to see the raw number of duplicate files.

Still here? Maybe talk to us on the issue tracker.

Can you implement feature X?¶

Depends. Go to to the issue tracker and open a feature request.

Here is a list of features where you probably have no chance:

• Port it to Windows.
• Find similar files like ssdeep does.
• Make a graphical user interface (totally okay as separate project, though).

Bugs¶

Please use the issue tracker to post and discuss bugs and features:

https://github.com/sahib/rmlint/issues

Philosophy¶

We try to adhere to some principles when adding features:

• Try to stay compatible to standard unix' tools and ideas.
• Try to stay out of the users way and never be interactive.
• Try to make scripting as easy as possible.
• Never make rmlint modify the filesystem itself, only produce output to let the user easily do it.

Also keep this in mind, if you want to make a feature request.

Making contributions¶

The code is hosted on GitHub, therefore our preferred way of receiving patches is using GitHub's pull requests (normal git pull requests are okay too of course).

Here's a short step-by-step:

1. Fork it.
2. Create a branch from develop. (git checkout develop && git checkout -b my_feature)
3. Commit your changes. (git commit -am "Fixed it all.")
4. Check if your commit message is good. (If not: git commit --amend)
5. Push to the branch (git push origin my_feature)
6. Open a Pull Request.
7. Enjoy a refreshing Tea and wait until we get back to you.

Here are some other things to check before submitting your contribution:

• Does your code look alien to the other code? Is the style the same?
• Do all tests run? (Simply run nosetests to find out) Also after opening the pull request, your code will be checked via TravisCI.
• Is your commit message descriptive?
• Is rmlint running okay inside of valgrind (i.e. no leaks and no memory violations)?

For language-translations/updates it is also okay to send the .po files via mail at sahib@online.de, since not every translator is necessarily a software developer.

Buildsystem Helpers¶

# Use clang and enable profiling, verbose build and enable debugging
CC=clang CFLAGS='-pg' LDFLAGS='-pg' scons VERBOSE=1 DEBUG=1

$ USE_VALGRIND=1 nosetests  # or nosetests-3.3, python3 needed.

Sourcecode layout¶

• All C-source lives in src, the file names should be self explanatory.
• All documentation is inside docs.
• All translation stuff should go to po.
• All packaging should be done in pkg/<distribution>.
• Tests are written in Python and live in tests.

Hashfunctions¶

Here is a short comparasion of the existing hashfunctions in rmlint (linear scale). For reference: Those plots were rendered with these sources - which are very ugly, sorry.

If you want to add new hashfunctions, you should have some arguments why it is valueable and possiblye even benchmark it with the above scripts to see if it's really that much faster.

Also keep in mind that most of the time the hashfunction is not the bottleneck.

Optimizations¶

For sake of overview, here is a short list of optimizations implemented in rmlint:

Adding new languages¶

# Fork a new .po file from the po-template (here swedish):
$ msginit -i po/rmlint.pot -o po/se.po --locale se --no-translator

# Edit the po/se.po file, the format is self describing
$ vim po/se.po

# .po files need to be compiled, but that's handled by scons already.
$ scons
$ scons install

# You should see your changes now:
$ LANG=se ./rmlint

If you'd like to contribute your new translation you want to do a pull request (if you really dislike that, you may also send the translation to us via mail). Here is a small introduction on Pull Requests.

Updating existing languages¶

# Edit the file to your needs:
$ vim po/xy.po

# Install:
$ scons install

# Done
$ LANG=xy ./rmlint

Marking new strings for translations¶

If you want to mark strings in the C-code to be translated, you gonna need to mark them so the xgettext can find it. The latter tool goes through the source and creates a template file with all translations left out.

/* Mark the string with the _() macro */
fprintf(out, _("Stuff is alright: %s\n"), (alright) ? "yes" : "no");

It gets a little harder when static strings need to be marked, since they cannot be translated during compile time. You have to mark them first and translate them at a later point:

static const char * stuff = _N("Hello World");

void print_world(void) {
    printf("World is %s\n", _(stuff));
}

After you're done with marking the new strings, you have to update the template:

# scons can do this for you already:
$ scons xgettext

You need to add the new strings to the existing translations now:

$ msgmerge po/de.po po/rmlint.pot > po/de_new.po
$ EDITOR po/de_new.po   # check if everything was merged alright.
$ mv po/de_new.po po/de.po

After that you can translate the new strings and proceed like in the upper steps.

find duplicate files and other space waste efficiently¶

# -ed is recognized as -e and -d here, -d takes "-s 10M" as parameter.
# This will fail to do the supposed, finding also files smaller than 10M.
$ rmlint -T all -ef -ed -s10M /media/music/