11. (1) Let $l_{A B}: y=k x+1$. Then
$$
l_{A C}: y=-\frac{1}{k} x+1 \text {. }
$$

Substituting it into the ellipse equation, we get
$$
\begin{array}{l}
\left(1+a^{2} k^{2}\right) x^{2}+2 a^{2} k x=0 \\
\Rightarrow x=0 \text { or } x=\frac{-2 a^{2} k}{1+a^{2} k^{2}} \\
\Rightarrow A(0,1), B\left(\frac{-2 a^{2} k}{1+a^{2} k^{2}}, \frac{1-a^{2} k^{2}}{1+a^{2} k^{2}}\right) \\
\Rightarrow A B=\sqrt{1+k^{2}} \frac{\left|2 a^{2} k\right|}{1+a^{2} k^{2}} .
\end{array}
$$

Similarly, $A C=\sqrt{1+\frac{1}{k^{2}}} \frac{\left|\frac{2 a^{2}}{k}\right|}{1+\frac{a^{2}}{k^{2}}}$.
Thus, $S=\frac{1}{2} \sqrt{1+k^{2}} \frac{\left|2 a^{2} k\right|}{1+a^{2} k^{2}} \sqrt{1+\frac{1}{k^{2}}} \frac{\left|\frac{2 a^{2}}{k}\right|}{1+\frac{a^{2}}{k^{2}}}$
$$
\begin{array}{l}
=\frac{2 a^{4}\left|k+\frac{1}{k}\right|}{1+a^{4}+a^{2}\left(k^{2}+\frac{1}{k^{2}}\right)} \leqslant \frac{a^{4}}{a\left(a^{2}-1\right)}=\frac{27}{8} \\
\Rightarrow a=3 \text { or } \frac{3+\sqrt{297}}{16} \text { (does not satisfy the equality }
\end{array}
$$

condition, discard).
Therefore, $a=3$.
(2) Let $B\left(x_{1}, y_{1}\right), C\left(x_{2}, y_{2}\right)$, the slopes of lines $A B$ and $A C$ be $k_{1}$ and $k_{2}$, and $l_{B C}: y=k x+t$.
Combining with the ellipse equation, we get
$$
\begin{array}{l}
\left(b^{2}+a^{2} k^{2}\right) x^{2}+2 k t a^{2} x+a^{2} t^{2}-a^{2} b^{2}=0 \\
\Rightarrow x_{1}+x_{2}=\frac{-2 k t a^{2}}{b^{2}+a^{2} k^{2}}, x_{1} x_{2}=\frac{a^{2} t^{2}-a^{2} b^{2}}{b^{2}+a^{2} k^{2}}, \\
k_{1} k_{2}=\frac{\left(k x_{1}+t-b\right)\left(k x_{2}+t-b\right)}{x_{1} x_{2}} \\
=\frac{b^{2}}{a^{2}} \cdot \frac{t-b}{t+b} .
\end{array}
$$

If $k_{1} k_{2}=-1$, then
$\frac{b^{2}}{a^{2}} \cdot \frac{t-b}{t+b}=-1$
$\Rightarrow t=-b \cdot \frac{a^{2}-b^{2}}{a^{2}+b^{2}}$ is a constant.
And $S=\frac{1}{2}|t-b|\left|x_{1}-x_{2}\right|$
$=\frac{2 a^{3} b^{2}}{a^{2}+b^{2}} \sqrt{\frac{b^{2}+a^{2} k^{2}-t^{2}}{\left(b^{2}+a^{2} k^{2}\right)^{2}}}$.
Let $u=b^{2}+a^{2} k^{2}, u \geqslant b^{2}>t^{2}$, then we need to find the maximum value of $\sqrt{\frac{u-t^{2}}{u^{2}}}$, which is equivalent to finding the minimum value of $\frac{u^{2}}{u-t^{2}}$.
Notice that,
$$
\frac{u^{2}}{u-t^{2}}=2 t^{2}+u-t^{2}+\frac{t^{4}}{u-t^{2}} \geqslant 4 t^{2},
$$

with equality if and only if $u=2 t^{2}$.
We need $b^{2} \leqslant 2 t^{2}$, i.e., $a \geqslant(\sqrt{2}+1) b$, in which case,
$$
\begin{array}{l}
S=\frac{2 a^{3} b^{2}}{a^{2}+b^{2}} \sqrt{\frac{u-t^{2}}{u^{2}}} \\
\leqslant \frac{2 a^{3} b^{2}}{a^{2}+b^{2}} \cdot \frac{1}{-2 t}=\frac{a^{3} b}{a^{2}-b^{2}}=\frac{a b}{e^{2}} .
\end{array}
$$

Thus, the maximum area of the triangle is $\frac{a b}{e^{2}}$.
If $at^{2}$, then when $k=0$, $u-t^{2}=b^{2}-t^{2}$, $\frac{u^{2}}{u-t^{2}}=2 t^{2}+u-t^{2}+\frac{t^{4}}{u-t^{2}}$ achieves its minimum value. At this time,
$$
\begin{array}{l}
S=\frac{2 a^{3} b^{2}}{a^{2}+b^{2}} \sqrt{\frac{u-t^{2}}{u^{2}}} \\
\leqslant \frac{2 a^{3} b^{2}}{a^{2}+b^{2}} \cdot \frac{\sqrt{b^{2}-t^{2}}}{b^{2}} \\
=\frac{4 a^{4} b^{2}}{\left(a^{2}+b^{2}\right)^{2}} \text {. } \\
\text { In summary, } S_{\text {max }}=\left\{\begin{array}{ll}
\frac{a b}{e^{2}}, & a \geqslant(\sqrt{2}+1) b ; \\
\frac{4 a^{4} b^{2}}{\left(a^{2}+b^{2}\right)^{2}}, & b<a<(\sqrt{2}+1) b .
\end{array}\right. \\
\end{array}
$$
11. (20 points) In the Cartesian coordinate system $x O y$, the three vertices of the right triangle $\triangle A B C$ are all on the ellipse $\frac{x^{2}}{a^{2}}+\frac{y^{2}}{b^{2}}=1$ $(a>b>0)$, where $A(0, b)$ is the right-angle vertex.
(1) If $b=1$, the maximum area of the triangle is $\frac{27}{8}$, find the value of the real number $a$;
(2) Find the maximum area of the triangle.