 Pitt began his career with Premier League club Chelsea, progressing through the youth ranks at Stamford Bridge.Pitt joined First Division side Portsmouth on a three-year contract on 16 June 2001 when his former Chelsea coach Graham Rix took over as manager at Fratton Park, turning down the offer of a new three-year contract at Chelsea.Pitt made the switch for a tribunal set fee of £200,000 plus add-ons, and he finished his first season with the club, 2001–02, with 41 appearances and three goals.