Removing duplicate music files with different names from a large directory free

During a recent Mac migration, I hosed my iTunes library and unintentionally generated hundreds of duplicate songs (the actual media files). The problem was exacerbated by the fact that a numeric was generally appended to the duplicate filenames so that I had directories full of files like the following:
/artist/album/song.mp3/artist/album/song 1.mp3/artist/album/song 2.mp3
Apple wasn't doing me any favors by injecting a space in the new names either....what a mess!

I stole a few ideas from this excellent forum thread and cobbled together a script to remove files based on exact byte counts and md5sum signatures. Here's the good stuff:

#!/bin/bashfind . ! -empty -type f -printf "%s " -exec ls -dQ {} ; | sort -n | cut -d" " -f2- | xargs md5sum | sort | uniq -w32 -d | cut -c35- | while read filenamedo echo "Removing $filenamen" rm "$filename"done
Now here's the play-by-play:
  1. Prints out the size and filename of each file found on the path and sorts using the filesize as the key
  2. Trims off the file size in preparation for next stage
  3. Creates the checksum for the files of the same size and then sorts by result
  4. Strip out any checksums that are unique, leaving only the duplicates
  5. Strips out the checksum part, just leaving the duplicate filenames
  6. Loops through the results removing the first duplicate file of each set.
You'll need to run the script multiple times if you've managed to generate more than one copy of any given file. Good luck!
Share Comments