This question was previously posted at https://math.stackexchange.com/questions/1220292/example-of-pairwise-independent-random-process-with-expected-max-load-sqrtn where it has no answers. I now realise this is relevant to hashing so I am reposting it here.

Throw $n$ balls into $n$ bins. Each bin is selected uniformly at random but the process is only pairwise independent. Call the maximum number of balls in any bin the max load.

Lemma 2 in these notes tells us that the max load is at most $\sqrt{2n}$ with probability at least $1/2$.

Can we give an explicit uniform but only pairwise independent random process for selecting the next bin to put a ball into which gives the expected max load to be $\Theta(\sqrt{n})$ asymptotically?

  • 2
    $\begingroup$ mathoverflow.net/a/202583 $\endgroup$ – Emil Jeřábek Apr 10 '15 at 19:07
  • 1
    $\begingroup$ As @EmilJeřábek implies, it seems that you can do it more generally for $k$-independence too, using theorem 1 of fr.arxiv.org/abs/1502.05729. The paper points out that for pairwise independence you can use theorem 8 of brics.dk/RS/97/16/BRICS-RS-97-16.pdf, but it seems false (i think that the universal hash family used there is not 2-independent). $\endgroup$ – didest Apr 11 '15 at 17:27
  • $\begingroup$ @DiegodeEstrada Thank you. It is also certainly very interesting if there indeed such an error in that paper. $\endgroup$ – felix Apr 11 '15 at 20:03

Here's how to do it.

First, choose a random $k$ between 1 and $n$ to be the "crowded bin". Next, choose a random permutation $\pi$ of $1,2,\ldots, n-1$.

Now, for $1 \leq i \leq n-1$,
$$ \mbox{put ball } i \mbox{ into bin } \begin{cases} k \ \ \ \ \ \ \ \ \ \ \ \ \mbox{with probability }\ \frac{1}{\sqrt{n}}, \\ k + \pi(i) \mbox{ with probability }1 - \frac{1}{\sqrt{n}}, \end{cases}$$ where the sum is taken mod $n$.

Put ball $n$ into a random bin.

Consider balls $i,j$ where $1 \leq i < j \leq n-1$. The bins these two balls fall into are independent. They both go into the crowded bin with probability $\frac{1}{n}$, and the crowded bin is equally likely to be any of the $n$ bins. And if they don't both go into the crowded bin, by symmetry they are equally likely to be in any two different bins. Thus, the bins these two balls are placed into are independent.

Since ball $n$ is placed into a random bin, it is independent of the other balls.

The expected max load here is essentially the expected number of balls in the crowded bin, which is $$\frac{n-1}{\sqrt{n}}+\frac{1}{n} = \sqrt{n} + o(1).$$

  • 1
    $\begingroup$ Thank you very much. This is a really nice and clear answer. $\endgroup$ – felix Apr 11 '15 at 8:51
  • $\begingroup$ I suppose you don't literally mean "the sum is taken mod $n$" as that would mean you sometimes place in bin $0$ which doesn't exist. $\endgroup$ – felix Apr 11 '15 at 15:55
  • 1
    $\begingroup$ @felix: not quite literally: $0 \equiv n$ (mod $n$). $\endgroup$ – Peter Shor Apr 11 '15 at 16:50
  • 2
    $\begingroup$ In my case I would like to throw $2n$ balls into $n$ bins. I could just run your process twice with the second run completely independent of the first but that doesn't seem right. $\endgroup$ – felix Apr 11 '15 at 19:03
  • 3
    $\begingroup$ Instead of a random permutation $\pi$, use a random reordering of the sequence $1,1,2,2, 3,3,\ldots,n-1,n-1$. And instead of probability $1/\sqrt{n}$, figure out which probability makes the first $2n-2$ balls pairwise independent. And just toss the last two balls into random bins. (There's undoubtedly a better way of dealing with the last two balls, but it won't make a substantial difference in the expected max load and it will make everything more complicated.) $\endgroup$ – Peter Shor Apr 11 '15 at 19:58

Your Answer

