vDedup

vDedup <--purge_secondary> <--merge_into_primary> primary_starting_path secondary_starting_path

Features Roadmap

• list likely duplicate files between two path roots
• purge duplicates from secondary
• merge remaining secondary files into primary paths

   Prim: oval "primary path" ht 50%; move
   Sec: oval "secondary path" ht 50%; move
   Db: box "SQLite"; move
   arrow right down from Prim to Db;
   arrow right up from Sec to Db

• roughly 2.6 million filenames loaded into sqlite in 30 seconds

• and 47 million in 4 minutes with 18 threads reading and 1 writing (that's sorta the limit of my machine... ate up 32GB RAM + 4GB swap + 3GB cache.... I was waiting for the machine to fail... just finished then) results in 7.5GB SQLite file on disk

DO

• x list local
• x list primary
• x list secondary
• x load SQLite with primary
• x load SQLite with secondary
• find length of reversed path string matches