vDedup <--purge_secondary> <--merge_into_primary> primary_starting_path secondary_starting_path
Features Roadmap
- list likely duplicate files between two path roots
- purge duplicates from secondary
- merge remaining secondary files into primary paths
Prim: oval "primary path" ht 50%; move Sec: oval "secondary path" ht 50%; move Db: box "SQLite"; move arrow right down from Prim to Db; arrow right up from Sec to Db
roughly 2.6 million filenames loaded into sqlite in 30 seconds
and 47 million in 4 minutes with 18 threads reading and 1 writing (that's sorta the limit of my machine... ate up 32GB RAM + 4GB swap + 3GB cache.... I was waiting for the machine to fail... just finished then) results in 7.5GB SQLite file on disk
DO
- x list local
- x list primary
- x list secondary
- x load SQLite with primary
- x load SQLite with secondary
- find length of reversed path string matches