Heterochromatin, typically marked by histone H3 trimethylation at lysine 9 (H3K9me3) or lysine 27 (H3K27me3), represses different protein-coding genes in different cells, as well as repetitive elements. The basis for locus specificity is unclear. Previously, we identified 172 proteins that are embedded in sonication-resistant heterochromatin (srHC) harbouring H3K9me3. Here, we investigate in humans how 97 of the H3K9me3-srHC proteins repress heterochromatic genes. We reveal four groups of srHC proteins that each repress many common genes and repeat elements. Two groups repress H3K9me3-embedded genes with different extents of flanking srHC, one group is specific for srHC genes with H3K9me3 and H3K27me3, and one group is specific for genes with srHC as the primary feature. We find that the enhancer of rudimentary homologue (ERH) is conserved from Schizosaccharomyces pombe in repressing meiotic genes and, in humans, now represses other lineage-specific genes and repeat elements. The study greatly expands our understanding of H3K9me3-based gene repression in vertebrates.