DeDup is a Java command-line utilty for removing duplicate
lines in text files.
The dedup process compares adjacent lines only. It does not
sort first. The comparison is case sensitive. It removes
adjacent indentical lines. It replaces the original file, so
do a backup first. You can dedup 1 to N files in one
execution, just put their names on the command line.
DeDup deletes blank lines from both the beginning and end of
the file. It also deletes trailing blanks from each line. It
also cleans up line terminators r n rn and converts them
all to platform standard. If the file is already deduped,
it does not change the date.
dedup util FREE dedup util is a lightweight open-source files packing tool. It is based on block-level data de-duplication technology, and can reduce the data capacity to save storage space. The key feature is no block data collsion with md5 hashkey. Download
pcap-dedup FREE This is a very simple utility to fix PCAP files that have duplicated traffic from span ports and the like. It uses the IP ID field to identify any duplicates based on source/destination that have appeared in the last 10 packets for that flow. Download