5. C.

As shown in Figure 7, when $\angle A D E>90^{\circ}$, connect $A A^{\prime}$, extend $E D$, and let it intersect $A A^{\prime}$ at point $M$.
By the Pythagorean theorem, we have
$$
\begin{array}{l}
A B=\sqrt{6}, \\
A D=D B=\frac{\sqrt{6}}{2} .
\end{array}
$$

From the problem, we know that $A D=A^{\prime} D, A E=A^{\prime} E$, so $E D$ is the perpendicular bisector of $A A^{\prime}$.
Since $A D=D B=A^{\prime} D$
$\Rightarrow \triangle A B A^{\prime}$ is a right triangle
$\Rightarrow A A^{\prime} \perp A^{\prime} B \Rightarrow M E / / A^{\prime} B$
$$
\Rightarrow \angle M E F=\angle F A^{\prime} B \text {. }
$$

Given that $S_{\triangle D E F}=\frac{1}{4} S_{\triangle A E B}$, we have
$$
\begin{array}{l}
D F=\frac{1}{4} A B=\frac{1}{2} D B \\
\Rightarrow D F=F B .
\end{array}
$$

Thus, $\triangle A^{\prime} F B \cong \triangle E F D$
$$
\Rightarrow E F=A^{\prime} F \text {. }
$$

Therefore, $\triangle B F E \cong \triangle D F A^{\prime}$
$$
\Rightarrow A D=B E=\frac{\sqrt{6}}{2} \text {. }
$$

If $\angle A D E<90^{\circ}$, as shown in Figure 8.
Similarly, $A E=A E^{\prime}=B D=\frac{\sqrt{6}}{2}$.
Thus, $C E=\frac{\sqrt{2}}{2}$.
Hence, $B E=B C-C E=\frac{\sqrt{5}-\sqrt{2}}{2}$.
In summary, $B E=\frac{\sqrt{6}}{2}$ or $\frac{\sqrt{5}-\sqrt{2}}{2}$.
5. In Rt $\triangle A B C$, $\angle C=90^{\circ}, A C=1$, $B C=\sqrt{5}$, $D$ is the midpoint of side $A B$, and $E$ is a point on side $B C$. When $\triangle A D E$ is folded along $D E$ to get $\triangle A^{\prime} D E$, such that the overlapping area of $\triangle A^{\prime} D E$ and $\triangle B D E$ is $\frac{1}{4}$ of the area of $\triangle A B E$. Then the length of $B E$ is ( ).
(A) $\frac{\sqrt{6}}{2}$
(B) $\frac{\sqrt{5}-\sqrt{2}}{2}$
(C) $\frac{\sqrt{6}}{2}$ or $\frac{\sqrt{5}-\sqrt{2}}{2}$
(D) $\frac{3}{2}$ or $\frac{\sqrt{5}-1}{2}$