Eukaryotic deubiquitinases are important regulators of ubiquitin signaling and can be subdivided into several structurally distinct classes. The ZUFSP family, with ZUP1 as its sole human member, has a modular architecture with a core catalytic domain highly active against the ubiquitin-derived peptide RLRGG, but not against ubiquitin itself. Ubiquitin recognition is conferred by additional non-catalytic domains, making full-length ZUP1 active against long K63-linked chains. However, non-mammalian ZUFSP family members contain different ubiquitin-binding domains in their N-terminal regions, despite their high conservation within the catalytic domain. Here, by working with representative ZUFSP family members from insects, fungi and plants, we show that different N-terminal domains are associated with different linkage preferences. Biochemical and structural studies suggest that the acquisition of two family-specific proximal domains have changed the default K48 preference of the ZUFSP family to the K63 preference observed in ZUP1 and its insect homolog. Additional N-terminal zinc finger domains promote chain cleavage without changing linkage-specificity.