3. 50 .
$$
\text { Given } \begin{array}{l} 
f(x) \\
=|a \sin x+b \cos x-1|+|b \sin x-a \cos x| \\
=\left|\sqrt{a^{2}+b^{2}} \sin (x+\alpha)-1\right|+ \\
\left|\sqrt{a^{2}+b^{2}} \cos (x+\alpha)\right|\left(\tan \alpha=\frac{b}{a}\right) \\
\leqslant\left|\sqrt{a^{2}+b^{2}} \sin (x+\alpha)\right|+ \\
\left|\sqrt{a^{2}+b^{2}} \cos (x+\alpha)\right|+1 \\
= \sqrt{a^{2}+b^{2}}(|\sin (x+\alpha)|+|\cos (x+\alpha)|)+1 \\
= \sqrt{2\left(a^{2}+b^{2}\right)}\left|\sin \left(x+\alpha \pm \frac{\pi}{4}\right)\right|+1,
\end{array}
$$

and the maximum value of the function $f(x)$ is 11, so
$$
\begin{array}{l}
\sqrt{2\left(a^{2}+b^{2}\right)}+1=11 \\
\Rightarrow a^{2}+b^{2}=50
\end{array}
$$
$$
\begin{aligned}
f(x)= & |a \sin x+b \cos x-1|+ \\
& |b \sin x-a \cos x| \quad(a, b \in \mathbf{R})
\end{aligned}
$$

If the maximum value of the function is 11, then $a^{2}+b^{2}=$ $\qquad$ .