5.6.

Substitution $\left(x_{1}, x_{2}\right)=(2 \cos \alpha, 2 \cos \beta)$.
Then the given inequality becomes
$$
2 x_{1}^{2}+x_{1} x_{2}+2 x_{2}^{2}-3 x_{1}-3 x_{2}-2 k<0 \text {. }
$$

Since $x_{1}, x_{2} \in(-1,2)$, we have
$$
\begin{array}{l}
\left(x_{1}+1\right)\left(x_{1}-2\right)+\left(x_{2}+1\right)\left(x_{2}-2\right)+ \\
\frac{1}{2}\left(x_{1}+1\right)\left(x_{2}-2\right)+\frac{1}{2}\left(x_{1}-2\right)\left(x_{2}+1\right)<0 \\
\Rightarrow 2 x_{1}^{2}+x_{1} x_{2}+2 x_{2}^{2}-3 x_{1}-3 x_{2}-12<0 .
\end{array}
$$

When $x_{1}=x_{2}=x \rightarrow 2$ (or -1), the left side of the above expression
$\rightarrow 0$.
Thus, $k_{\min }=6$.
5. For any $\alpha, \beta \in\left(0, \frac{2 \pi}{3}\right)$, we have
$$
\begin{array}{l}
4 \cos ^{2} \alpha+2 \cos \alpha \cdot \cos \beta+4 \cos ^{2} \beta- \\
3 \cos \alpha-3 \cos \beta-k<0 .
\end{array}
$$

Then the minimum value of $k$ is $\qquad$