5. $\frac{4 \sqrt{2}-1}{4} R$.

Consider the centers of the four spheres lying in the same plane.
Let the centers of the spheres with radii $R$, $2r$, and the two spheres with radius $r$ be $O$, $O_{1}$, $O_{2}$, and $O_{3}$, respectively, where,
$$
\begin{array}{l}
O O_{1}=R-2 r, O_{1} O_{2}=O_{1} O_{3}=3 r, \\
O O_{2}=O O_{3}=R-r .
\end{array}
$$

It is sufficient to satisfy $\mathrm{O}_{2} \mathrm{O}_{3} \geqslant 2 r$.
Extend $O O_{1}$ to intersect $O_{2} O_{3}$ at point $M$. Then
$$
\begin{array}{l}
\sin \angle O_{2} O_{1} M=\frac{O_{2} M}{O_{1} O_{2}} \geqslant \frac{1}{3} \\
\Rightarrow \frac{2 \sqrt{2}}{3} \geqslant \cos \angle O_{2} O_{1} M \\
\quad=\frac{9 r^{2}+(R-2 r)^{2}-(R-r)^{2}}{6 r(R-2 r)}=\frac{6 r-R}{3 R-6 r} \\
\Rightarrow r \leqslant \frac{4 \sqrt{2}-1}{4} R .
\end{array}
$$
5. If a hollow sphere with radius $R$ contains two solid spheres with radius $r$ and one solid sphere with radius $2r$, then the maximum value of $r$ is