Duplicut - Remove Duplicates From MASSIVE Wordlist, Without Sorting It (For Dictionary-Based Password Cracking)

Duplicut - Remove Duplicates From MASSIVE Wordlist, Without Sorting It (For Dictionary-Based Password Cracking)


Quickly dedupe massive wordlists, without changing the order 

Modern password wordlist creation usually implies concatenating multiple data sources.


Ideally, most probable passwords should stand at start of the wordlist, so most common passwords are cracked instantly.


With existing dedupe tools you are forced to choose if you prefer to preserve the order OR handle massive wordlists.


Unfortunately, wordlist creation requires both:


So i wrote duplicut in highly optimized C to address this very specific need



Quick start git clone https://github.com/nil0x42/duplicutcd duplicut/ && make./duplicut wordlist.txt -o clean-wordlist.txt
Features:
Handle massive wordlists, even those whose size exceeds available RAM
Filter lines by max length (-l option)
Can remove lines containing non-printable ASCII chars (-p option)
Press any key to show program status at runtime. Implementation:
Written in pure C code, designed to be fast
Compressed hashmap items on 64 bit platforms
Multithreading support
[TODO]: Use huge memory pages to increase performance L ..

Support the originator by clicking the read the rest link below.