 Lady Jane Taylor survived her husband until 1920.Taylor's brothers-in-law included Arthur Hay, 9th Marquess of Tweeddale, Admiral of the Fleet Lord John Hay, and the Liberal member of parliament George Hay, Earl of Gifford, while his sisters-in-law were married to James Broun-Ramsay, 1st Marquess of Dalhousie, a Governor-General of India, Arthur Wellesley, 2nd Duke of Wellington, Sir Robert Peel, 3rd Baronet, and Simon Watson Taylor, of Erlestoke, Wiltshire.Taylor's five children were Constance Mary Jane (married Ronald William Murray, died 1950), Millicent Lilla Harriet (born 1867 died 1948), Evelyn Beatrice Charlotte (died 1944 unmarried), Florence Virginia Mathilde (died 1952 unmarried) and Richard Edward Montagu Taylor (1872–1953).