The 5 Best Tools to Find and Remove Duplicate Files in Linux

File management is a complicated task in itself. Add to that large number of duplicate files that usually take up storage space, and the process just gets harder and harder.

While the standard way to deal with duplicate files is to locate and delete them manually. However, using a special program to find duplicate files can speed up the process considerably.

So if you’re planning to remove duplicate files and clean up your computer, here’s a list of some of the best tools for finding and removing duplicate files in Linux.

Fslint is a GUI and CLI based utility for cleaning up various types of junk from your system. It calls this junk “fluff” and provides multiple tools to help you perform a variety of tasks, including finding duplicate files, empty folders, and problematic file names.

By providing both graphical and command-line modes, fslint makes it easier for new Linux users to clear their computer storage of all kinds of system fluff.

To access fslint through the GUI, all you need to do is open the terminal and the fslint-gui order.

As for advanced functionality, the program offers 10 different functionalities in CLI mode such as findup, findu8, findnl, findtf and found. This allows you to narrow down the search results to increase your chances of finding specific types of duplicate files on your system.

How to install fslint?

On Debian based distributions such as Ubuntu:

sudo apt install fslint

RHEL-based distributions such as CentOS and Fedora:

sudo yum install fslint
sudo dnf install fslint

On Arch Linux and Manjaro:

sudo pacman -S fslint

Fdupes is one of the easiest programs to identify and remove duplicate files in folders. Released under the MIT license on GitHub, it is free and open-source.

The program works using md5sum signature and byte-by-byte comparison verification to determine duplicate files in a directory. If necessary, you can also perform recursive searches, filter out search results and get an overview of the discovered duplicate files.

Once you have identified duplicate files in a folder, you can use fdupes to delete the files or replace them with links to the original file.

Fdupes installation

On Debian-based distributions:

sudo apt install fdupes

RHEL-based distributions:

sudo yum install fdupes
sudo dnf install fdupes

Install on Arch Linux and Manjaro:

sudo pacman -S fdupes

Related: How to Find and Remove Duplicate Files on Linux with fdupes

Rdfind is another Linux utility that helps you find redundant files on your computer in different folders. It relies on comparing files based on their content – not their name – to identify duplicates, making it more effective at its job.

To achieve this, the program works by arranging similar files in a folder and determining the original and duplicates: the highest ranked is selected as the original, while the rest are duplicates.

By the way, rdfind can also calculate: checksums to compare files if necessary. And the best part is that it saves the scanned results in a results.txt file in the home directory so you can reference it when you go to remove duplicates to make sure you don’t remove the wrong one.

Of course, like most other duplicate file finders, rdfind also offers some preprocessors to sort files, ignore empty files, or set symlinks. Last but not least, there is an option to delete duplicate files as well.

Related: What is a Symbolic Link (Symlink)? How to make one in Linux

How do I install rdfind

On Debian/Ubuntu:

sudo apt install rdfind

On Fedora/CentOS:

sudo dnf install rdfind

DupeGuru is a cross-platform tool for finding and removing duplicate files on your computer. One of its best features is the option to customize the matching engine to increase your chances of finding the right kind of duplicate files in a folder. And similar to a few other duplicate search programs, it also offers a GUI to facilitate easier operations.

Speaking of functionality, dupeGuru takes advantage of its vague matching algorithm to scan file names or file contents and find duplicates quickly and efficiently.

In addition, it is also good at handling music and photo-specific information, giving it an edge over other duplicate file finders. Plus, if needed, you have the option to customize the associated engine to locate exactly the kind of duplicate files you want to remove.

DupeGuru also lets you remove duplicate files. And for this, it has a reference directory system, which prevents you from accidentally deleting the wrong files. In addition to deletion, there is the option to move or copy them elsewhere as well.

DupeGuru Installation

On Debian-based distributions:

sudo add-apt-repository ppa:dupeguru/ppa
sudo apt-get update
sudo apt-get install dupeguru

On Arch Linux:

sudo pacman -S dupeguru

Rmlint is yet another ribbon – and not just duplicate files – finder and remover for Linux. It is free to use and extremely fast in identifying duplicate files and folders on your system. You also get support for the Btrfs storage format, which sets it apart from other tools on this list.

Speaking of which, some of the other aspects where rmlint trumps the other competing duplicate file removal tools are its ability to search for files based on a certain time frame, find files with broken user/ group IDs and unstripped binaries that take up a lot of space. Besides, like a few other programs, it also saves the scanned results in rmlint.json and files, which are useful while deleting.

Keep in mind, though, that unlike other tools, rmlint isn’t the easiest to use: it generates a script to remove duplicates, which requires a certain level of understanding to be used effectively.

How do I install rmlint

On Debian-based distributions:

sudo apt install rmlint

On Fedora and CentOS:

sudo yum install rmlint
sudo dnf install rmlint

Arch-based distros like Manjaro:

sudo pacman -S rmlint

Keep duplicate files at bay on Linux

Using the duplicate file finder programs above, you can easily identify the duplicate files that may be taking up space on your computer and remove them altogether. However, one piece of advice when working with such tools is to be extra careful with your actions to avoid eventually deleting important files and documents on your system.

If you’re a little skeptical about which files to delete and which to keep, back up all data on your system just to be safe.

Leave a Reply

Your email address will not be published. Required fields are marked *