1. take a sha1 hash of the last 4KB of each file.

2. for any equal hashes, compare the whole file.

With this method you should be able to skip reading many large files in their entirety.