 Margolies was born in Kroza, Russian Empire (now Kražiai, Lithuania) in April 1851 and received his rabbinical training at the yeshivas in Kroza and Białystok.He received semicha from Rabbi Yomtov Lipman Heilpern, the Oneg Yom Tov.He became the rabbi of Sloboda at age 26, serving there until he was summoned to Boston in 1889 to serve as the chief rabbi for that city's Orthodox Jewish community.