Suppose that the angles of $\triangle ABC$ satisfy $\cos(3A)+\cos(3B)+\cos(3C)=1.$ Two sides of the triangle have lengths 10 and 13. There is a positive integer $m$ so that the maximum possible length for the remaining side of $\triangle ABC$ is $\sqrt{m}.$ Find $m.$
WLOG, let C be the largest angle in the triangle. As above, we can see that $\cos3A+\cos3B-\cos(3A+3B)=1$ Expanding, we get $\cos3A+\cos3B-\cos3A\cos3B+\sin3A\sin3B=1$ $\cos3A\cos3B-\cos3A-\cos3B+1=\sin3A\sin3B$ $(\cos3A-1)(\cos3B-1)=\sin3A\sin3B$ CASE 1: If $\sin 3A = 0$ or $\sin 3B = 0$, This implies one or both of A or B are 60 or 120. If one of A or B is 120, we have a contradiction, since C must be the largest angle. Otherwise, if one of A or B is 60, WLOG, assume A = 60, we would have $\cos(3B) + \cos(3C) = 2$, and thus, cos(3B) and cos(3C) both equal 1, implying $B = C = 120$, a contradiction to the fact that the sum of the angles of a triangle must be 180 degrees. CASE 2: If $\sin 3A \neq 0$ and $\sin 3B \neq 0$ $\frac{\cos3A-1}{\sin3A}\cdot\frac{\cos3B-1}{\sin3B}=1$ $\tan{\frac{3A}{2}}\tan{\frac{3B}{2}}=1$ Note that $\tan{x}=\frac{1}{\tan(90-x)}$, or $\tan{x}\tan(90-x)=1$ Thus $\frac{3A}{2}+\frac{3B}{2}=90$, or $A+B=60$. Now we know that $C=120$, so we can just use the Law of Cosines to get $\fbox{399}$ -Alexlikemath