Abstract

We present a tool, called sif, for finding all similar files in a large file system. Files are considered similar if they have significant number of common pieces, even if they are very different otherwise. For example, one file may be contained, possibly with some changes, in another file, or a file may be a reorganization of another file. The running time for finding all groups of similar files, even for as little as 25% similarity, is on the order of 500MB to 1GB an hour. The amount of similarity and several other customized parameters can be determined by the user at a post-processing stage, which is very fast. Sif can also be used to very quickly identify all similar files to a query file using a preprocessed index. Application of sif can be found in file management, information collecting (to remove duplicates), program reuse, file synchronization, data compression, and maybe even plagiarism detection. 1. Introduction Our goal is to identify files that came from the same source ...

Citations

175	Fingerprinting by random polynomials - Rabin - 1981
125	A good file system - Alex - 1992
121	Scalable Internet Resource Discovery: Research Problems and Approaches - Bowman, Danzig, et al. - 1994
115	Agrep-a fast approximate patternmatching tool - Wu, Manber - 1992
34	Essence: A resource discovery system based on semantic file indexing - Hardy, Schwartz - 1993
1	A theory of parameterized patern matching: Algorithms and applications - Baker - 1993
1	Detecting duplicates: a searcher’s dream come true - Miller - 1990