Fisher was born in Osmington, Dorset, the son of clergyman John Fisher (1788–1832) who was Vicar of Osmington and Canon of Salisbury. Young Fisher was named after Saint Osmond, the patron saint of the church where his father served.