 Lowther married (Charlotte) Alice Parke (d. 1908), daughter of James Parke, 1st Baron Wensleydale on 17 December 1853.They had seven children including James William Lowther, 1st Viscount Ullswater (1855–1949), Sir Gerard Augustus Lowther, 1st Baronet (1858–1916) and Sir Henry Cecil Lowther (1869–1940).Lowther was the younger brother of Henry Lowther, 3rd Earl of Lonsdale.