 After spending two seasons on the fringes of the first-team, he became a regular starter for the U's in the 2004–05 season, being rewarded a new two-year contract in March 2005 and leading him to be named as Colchester United 'Player of the Year' for 2005.In 2006, an FA Cup run led to Colchester being paired against Baldwin's former club Chelsea, allowing him to play against the club he joined as a ten-year-old.Colchester would lose the game 3–1 after taking the lead.