 Father and son, both accounted "notorious womanizers", had a bitter relationship.His father once challenged Luttrell to a duel, but he declined, observing that his father was not a gentleman.Luttrell, described as "strong in body, if not in mind", achieved a reputation for bravery as a soldier during the Seven Years' War, becoming Deputy Adjutant-General of the British Forces in Portugal.