 Chilton was born in London to working class parents (his father was a musical hall comedian) and was evacuated to Northamptonshire, where he began playing the cornet at the age of 12.He switched to trumpet at 17 and after doing national service in the RAF (1950–1952) he formed his own jazz band, playing at Butlins in Skegness with a troupe that included comedian Dave Allen.He worked in Bruce Turner's Jump Band from 1958-1963.