File management is a complicated task in and of itself. When you add that large volume of duplicate files that normally take up the disk space, the process becomes more difficult.
While the standard way to deal with duplicate files is to manually find and delete them. However, using a dedicated program to find duplicate files can speed up the process significantly.
So, if your plan is to get rid of duplicate files and clean up your computer, here is a list of some of the best tools to find and remove duplicate files in Linux.
Fslint is a GUI and CLI based utility for cleaning up various types of clutter on your system. It calls this clutter "lint," and offers several tools to help you complete a variety of tasks, including finding duplicate files, empty directories, and problematic file names.
By offering both graphical and command line modes of operation, fslint makes it easy for new Linux users to rid their computer memory of all kinds of system glitches.
To access fslint through the GUI, all you have to do is open the terminal and the fslint-gui Command.
As for the extended functionality, the program offers 10 different functionalities in CLI mode such as findup, findu8, findnl, findtf and found. These can be used to refine the search results to increase your chances of finding certain types of duplicate files on your system.
How to install fslint
On Debian-based distributions like Ubuntu:
sudo apt install fslint
On RHEL-based distributions like CentOS and Fedora:
sudo yum install fslint
sudo dnf install fslint
On Arch Linux and Manjaro:
sudo pacman -S fslint
Fdupes is one of the easiest programs to identify and delete duplicate files in directories. It was published on GitHub under the MIT license and is free and open source.
The program uses the md5sum signature and byte-by-byte comparison checking to identify duplicate files in a directory. If necessary, you can also perform recursive searches, filter out search results and get a summarized view of the duplicate files found.
Once you've identified duplicate files in a directory, you can use fdupes to either delete the files or replace them with links to the original file.
On Debian-based distributions:
sudo apt install fdupes
On RHEL-based distributions:
sudo yum install fdupes
sudo dnf install fdupes
To install on Arch Linux and Manjaro:
sudo pacman -S fdupes
Rdfind is another Linux utility that can be used to find redundant files on your computer in different directories. It relies on comparing files by their content – rather than their name – to identify duplicates, which makes the job more effective.
To achieve this, the program works by arranging identical files in a directory and determining the original and duplicates: the highest-ranking is selected as the original, while the rest are duplicates.
If necessary, rdfind can also calculate checksums in order to compare files. And the best part is that the scanned results are rolled into one Results.txt File in the home directory so that when you delete duplicates you can refer to it to make sure you don't remove the wrong ones.
Of course, like most other file finders for duplicates, rdfind also offers some preprocessors to sort files, ignore empty files or set symbolic links. Last but not least, there is an option to delete duplicate files.
How to install rdfind
Under Debian / Ubuntu:
sudo apt install rdfind
On Fedora / CentOS:
sudo dnf install rdfind
DupeGuru is a cross-platform tool for finding and deleting duplicate files on your computer. One of its best features is the ability to customize the matching engine to suit your preferences to increase your chances of finding the right kind of duplicate files in a directory. And similar to some other duplicate search programs, it also has a GUI to make it easier to use.
Speaking of functionality: dupeGuru uses its fuzzy matching algorithm to scan either file names or file contents and to find duplicates quickly and efficiently.
It's also good at handling music and image-specific information, which gives it an edge over other duplicate file finders. In addition, if necessary, you have the option to tweak the matching engine to find the exact type of duplicate files you want to remove.
You can also delete duplicate files with DupeGuru. And for that, it has a reference directory system that prevents you from accidentally deleting the wrong files. In addition to deleting them, there is also the option of moving them to another location or copying them.
On Debian-based distributions:
sudo add-apt-repository ppa: dupeguru / ppa
sudo apt-get update
sudo apt-get install dupeguru
Under Arch Linux:
sudo pacman -S dupeguru
Rmlint is another lint – and not just duplicate files – finder and remover for Linux. It's free to use and extremely fast in identifying duplicate files and directories on your system. You also get support for the Btrfs storage format, which sets it apart from other tools on this list.
Speaking of which, some of the other aspects where rmlint trumps the other competing duplicate file removal tools include the ability to search for files based on a specific timeframe, find files with corrupted user / group IDs, and unstripped binaries find that take up a lot of space. In addition, similar to some other programs, it also saves the scanned results rmlint.json and rmlint.sh Files that come in handy during the deletion process.
Note, however, that unlike other tools, rmlint is not the easiest to use: it generates a duplicate deletion script that requires some understanding to be used effectively.
How to install rmlint
On Debian-based distributions:
sudo apt install rmlint
On Fedora and CentOS:
sudo yum install rmlint
sudo dnf install rmlint
On Arch-based distributions like Manjaro:
sudo pacman -S rmlint
Keep duplicate files at bay on Linux
Using the duplicate files search programs listed above, you can easily identify and completely remove the duplicate files that may be taking up space on your computer. However, one piece of advice when working with such tools is to be extra careful with your actions to avoid deleting important files and documents on your system.
If you are a little skeptical about which files to delete and which to keep, you should make a backup copy of all the data on your system to be on the safe side.
The top 8 file backup apps for Linux
Regular data backups are crucial, whether it's a server or a local machine. Check out these eight file backup applications for Linux.
About the author
(23 articles published)
Yash is Staff Writer at MUO for DIY, Linux, programming and security. Before he discovered his passion for writing, he developed for the web and iOS. You can also find his writing on TechPP where he covers other industries. Aside from technology, he likes to talk about astronomy, Formula 1 and clocks.
By Yash Wate
Subscribe to our newsletter
Subscribe to our newsletter for tech tips, reviews, free e-books, and exclusive offers!
Click here to subscribe