 James Abbott was the third son of Henry Alexius Abbott, a retired Calcutta merchant of Blackheath, Kent, and his wife Margaret Welsh, the daughter of William Welsh of Edinburgh.Abbott was educated at a school in Eliot Place, Blackheath and at the East India Company Military Seminary in Addiscombe, Surrey.A number of his siblings achieved distinction, notably Augustus Abbott, Sir Frederick Abbott, Saunders Alexius Abbott and Keith Edward Abbott.