Why is Guava Bloom Filter Performing so Poorly?
<p>I am trying to determine whether <a href="https://github.com/google/guava/wiki/HashingExplained#bloomfilter" rel="nofollow noreferrer">Google Guava's Bloom Filter</a> will work for a project of mine, however in my tests, I'm getting an extremely high false rate of false positives (presumably due to a high level of hash collisions?).</p>

<p>I'm running the experiment using 2 data files. The first contains 22 million unique numbers (Integers) which I put into the bloom filter. The second contains another set of entirely different numbers, also unique, which I use to test the Bloom Filter for false positives.</p>

<p>This is an example of what some of these numbers look like:</p>

<pre><code>1010061
904436
859990
854448
839175
754186
904491
233955
904491
876342
919575
603051
1012863
989713
323424
</code></pre>

<p>My code is the following:</p>

<pre><code>private static void experiment() {

    // Load 22m unique IDs from file
    ArrayList&lt;String&gt; skus = loadSkus("sku_1.txt");
    int numInsertions = skus.size();

    // Google Guava Bloom Filter
    Funnel&lt;String&gt; strFunnel = (Funnel&lt;String&gt;) (from, into) -&gt; into.putString(from, Charset.forName("US-ASCII"));
    BloomFilter&lt;String&gt; bf = BloomFilter.create(strFunnel, numInsertions, 0.001);

    for (String sku : skus) {
        bf.put(sku);
    }

    int falsePositiveCount = 0;
    double falsePositiveRate;

    // Load another set of unique IDs that are NOT in the first set
    ArrayList&lt;String&gt; skus2 = loadSkus("sku_2.txt");

    for (String sku : skus2) {
        if (bf.mightContain(sku)) {
            falsePositiveCount++;
        }
    }

    falsePositiveRate = (double)falsePositiveCount / (double)skus2.size();

    System.out.println("Expected FPP:     " + Double.toString(bf.expectedFpp()));
    System.out.println("Measured FP rate: " + Double.toString(falsePositiveRate));
}
</code></pre>

<p>the result:</p>

<pre><code>Expected FPP:     7.276343403395039E-27
Measured FP rate: 0.9979594547309587
</code></pre>

<p>The measured rate of false positives seems unbelievably high! This is not how this data structure should behave. Am I misusing the library in some way? I would really like to achieve proper performance with the Bloom Filter.</p>