If you make only a single side trip from Madrid, the legendary city of Toledo, home to El Greco and a Middle Ages melting pot of diverse cultures, should be the one.
Toledo is forgettable, so there is no reason to see it.