Removing duplicate music files with different names from a large directory free

2008-10-28 238 words 2 minutes

During a recent Mac migration, I hosed my iTunes library and unintentionally generated hundreds of duplicate songs (the actual media files). The problem was exacerbated by the fact that a numeric was generally appended to the duplicate filenames so that I had directories full of files like the following:

/artist/album/song.mp3/artist/album/song 1.mp3/artist/album/song 2.mp3

Apple wasn't doing me any favors by injecting a space in the new names either....what a mess!

I stole a few ideas from this excellent forum thread and cobbled together a script to remove files based on exact byte counts and md5sum signatures. Here's the good stuff:

#!/bin/bashfind . ! -empty -type f -printf "%s " -exec ls -dQ {} ; | sort -n | cut -d" " -f2- | xargs md5sum | sort | uniq -w32 -d | cut -c35- | while read filenamedo echo "Removing $filenamen" rm "$filename"done

Now here's the play-by-play:

Prints out the size and filename of each file found on the path and sorts using the filesize as the key
Trims off the file size in preparation for next stage
Creates the checksum for the files of the same size and then sorts by result
Strip out any checksums that are unique, leaving only the duplicates
Strips out the checksum part, just leaving the duplicate filenames
Loops through the results removing the first duplicate file of each set.

You'll need to run the script multiple times if you've managed to generate more than one copy of any given file. Good luck!