 Slangenburg's relations with the Duke of Marlborough were not good, which in 1705 led to his removal from his post and replacement by Ouwerkerk, who had a far better personal relationship with Marlborough.Van Baer rebuild his ancestral mansion the "Slangenburg" near Doetinchem into a considerable castle, fitting his status.His wife is memorialized in many paintings there by Gerard Hoet.