 His father, also called Joe, had played football for Hartlepools United in the 1920s.Rayment began his football career as a youngster with First Division club Middlesbrough, while doing an electrical engineering apprenticeship at a local company.Having spent the 1951–52 season in the fourth team, and beginning the 1952–53 in the third eleven, Rayment made his first-team debut on 20 September 1952, at the age of 17, in a 4–1 defeat away to Liverpool.