I had to rewrite some large back-ups recently in order for restic to properly de-duplicate them. I checked that the re-written back-ups are identical to the original ones by comparing the entire directory trees of the mounted back-ups.

I had some doubts on whether diff -qr or rsync -acni --delete would be the best approach and did some benchmarks and tests using /usr (total: 3.2GB) which has a large mix of files and permissions.

I first tried a bind mount, but diff skips comparing anything entirely since it determines that both directories have the same device/inode. I used a copy of /usr instead.

Timing results are:

A quirk of using diff is that is follows symlinks, which is problematic for symlinks which point out of the tree being compared. This fails for broken symlinks, where diff exits with an error. Busybox diff ignores all permissions errors, which doesn’t sound right at all.

Finally, a key factor is that rsync can will compare permissions.

The key part of the comparison script is:

changes=$(rsync -acni --delete "$OLD_MOUNT/ids/$old_id/" "$NEW_MOUNT/ids/$new_id/" 2>&1)
rc=$?
if [ "$rc" -eq 0 ] && [ -z "$changes" ]; then
    printf 'OK   %s -> %s\n' "$old_id" "$new_id"
else
    printf 'FAIL %s -> %s\n' "$old_id" "$new_id"
    [ -z "$changes" ] || printf '%s\n' "$changes" | sed 's/^/    /'
    failed=$((failed + 1))
fi

annex: diffoscope

[permalink]

diffoscope was not usable: it tries to parse each file, and fails in case of a JPEG with invalid metadata or if objdump fails to open some random file.

I understand that parsing file is required for its intended purpose, but I don’t see why it needs to parse files before comparing if they’re identical. When comparing hundreds of gigabytes of data, preemptively parsing every single file is a huge cost (and many times outright fails), an unnecessary work for every file that’s identical on both sides.