Inside a right circular cone with base radius $5$ and height $12$ are three congruent spheres with radius $r$. Each sphere is tangent to the other two spheres and also tangent to the base and side of the cone. What is $r$?
We can take half of a cross section of the sphere, as such: [asy] unitsize(0.5cm); real r = (90-40*sqrt(3))/11; pair C = (0,0); pair A = (-5,0); pair B = (0,12); pair O = (-((2*sqrt(3))/3) * r, r); draw(A--B--C--cycle); draw(circle(O,r)); pair D = (-(2*sqrt(3))/3 * r - (12/13)*r, (18/13)*r); pair E = (-2.2, 0); draw(O--E); draw(D--O); label("$A$", A, SW); label("$B$", B, N); label("$C$", C, SE); label("$O$", O, (1,0)); label("$D$", D, NW); label("$E$", E, S); dot(D); dot(E); dot(O); [/asy] Notice that we chose a cross section where one of the spheres was tangent to the lateral surface of the cone at $D$. To evaluate $r$, we will find $AE$ and $EC$ in terms of $r$; we also know that $AE+EC = 5$, so with this, we can solve $r$. Firstly, to find $EC$, we can take a bird's eye view of the cone: [asy] unitsize(0.8cm); pair C = (0,0); draw(circle(C,5)); label("$C$", C, N); dot(C);  real r = (90-40*sqrt(3))/11; real raise = r*(2/3*sqrt(3)); pair E = (-r,raise/-2); pair X = (0,raise); pair Y = (r,raise/-2); label("$E$", E, SW); dot(E); label("$X$", X, N); dot(X); label("$Y$", Y, SE); dot(Y);  draw(circle(X,r),dashed); draw(circle(E,r),dashed); draw(circle(Y,r),dashed); draw(E--X,dashed); draw(X--Y,dashed); draw(E--Y,dashed); [/asy] Note that $C$ is the centroid of equilateral triangle $EXY$. Also, since all of the medians of an equilateral triangle are also altitudes, we want to find two-thirds of the altitude from $E$ to $XY$; this is because medians cut each other into a $2$ to $1$ ratio. This equilateral triangle has a side length of $2r$, therefore it has an altitude of length $r \sqrt{3}$; two thirds of this is $\frac{2r \sqrt{3}}{3}$, so $EC = \frac{2r \sqrt{3}}{3}.$ [asy] unitsize(0.5cm); real r = (90-40*sqrt(3))/11; pair C = (0,0); pair A = (-5,0); pair B = (0,12); pair O = (-((2*sqrt(3))/3) * r, r); draw(A--B--C--cycle); draw(circle(O,r)); pair D = (-(2*sqrt(3))/3 * r - (12/13)*r, (18/13)*r); pair E = (-2.2, 0); pair F = (-2.2, 6.72); draw(E--F); draw(D--O); draw(A--O, dashed); label("$A$", A, SW); label("$B$", B, N); label("$C$", C, SE); label("$O$", O, (1,0)); label("$D$", D, NW); label("$E$", E, S); label("$F$", F, NW); dot(D); dot(E); dot(O); [/asy] To evaluate $AE$ in terms of $r$, we will extend $\overline{OE}$ past point $O$ to $\overline{AB}$ at point $F$.$\triangle AEF$ is similar to $\triangle ACB$. Also, $AO$ is the angle bisector of $\angle EAB$. Therefore, by the angle bisector theorem, $\frac{OE}{OF} = \frac{AE}{AF} = \frac{5}{13}$. Also, $OE = r$, so $\frac{r}{OF} = \frac{5}{13}$, so $OF = \frac{13r}{5}$. This means that\[AE = \frac{5 \cdot EF}{12} = \frac{5 \cdot (OE + OF)}{12} = \frac{5 \cdot (r + \frac{13r}{5})}{12} = \frac{18r}{12} = \frac{3r}{2}.\] We have that $EC = \frac{2r \sqrt{3}}{3}$ and that $AE = \frac{3r}{2}$, so $AC = EC + AE = \frac{2r \sqrt{3}}{3} + \frac{3r}{2} = \frac{4r \sqrt{3} + 9r}{6}$. We also were given that $AC = 5$. Therefore, we have \[\frac{4r \sqrt{3} + 9r}{6} = 5.\] This is a simple linear equation in terms of $r$. We can solve for $r$ to get $r = \fbox{\frac{90-40\sqrt{3}}{11}}.$