Define a new number $B$ such that $\cos(B)=\frac{1}{2}$ and $\sin(B)=-\frac{\sqrt{3}}{2}$. Notice that $B=\frac{5\pi}{3}$. Therefore, we have $\sin\frac{A}{2}-\sqrt{3}\cos\frac{A}{2}= 2\cos(B)\sin\left(\frac{A}{2}\right)+2\sin(B)\cos\left(\frac{A}{2}\right)$. 
Recognizing the angle sum formula, we can rewrite this as $\sin\frac{A}{2}-\sqrt{3}\cos\frac{A}{2}=2\sin\left(\frac{A}{2}+B\right)$. We can now clearly see that the minimum is $-2$, and this is achieved when $\frac{A}{2}+B=\frac{3\pi}{2}$. Plugging in $B=\frac{5\pi}{3}$, we get $\frac{A}{2}=-\frac{\pi}{6}\implies A=-\frac{\pi}{3}=-60^\circ$. $\boxed{\text{E}}$
The minimum of $\sin\frac{A}{2}-\sqrt{3}\cos\frac{A}{2}$ is attained when $A$ is 
$\mathrm{(A)\ } -180^\circ \qquad \mathrm{(B) \ }60^\circ \qquad \mathrm{(C) \  } 120^\circ \qquad \mathrm{(D) \  } 0^\circ \qquad \mathrm{(E) \  }\text{none of these}$