 Ivanović was born into a family with a sporting history, his father Rade played for local team FK Srem as a defender.Ivanović played youth football for various teams in his home town Sremska Mitrovica.Until the age of 15, he played as a striker, before switching to his current position in defense.