Transposable elements (TEs) and repetitive sequences are ubiquitously present in eukaryotic genomes which are in general epigenetically silenced by DNA methylation and/or histone 3 lysine 9 methylation (H3K9me). RNA-directed DNA methylation (RdDM) is the major pathway that initiates de novo DNA methylation in Arabidopsis and sets up a self-reinforcing silencing loop between DNA methylation and H3K9me. However, a key issue is the requirement of a basal level transcript from the target loci to initiate the RNA-based silencing. How the heterochromatic silenced loci are transcribed remains largely unknown. Here, we show that JMJ24, a JmjC domain-containing protein counteracts H3K9me to promote basal level transcription of endogenous silenced loci in Arabidopsis. JMJ24 functionally resembles the fission yeast JmjC protein Epe1. The transcript promoted by JMJ24 is, at least in part, processed to small RNA to initiate the RdDM. Genome-wide transcriptome profiling indicates that transcript levels of TEs are more likely regulated by JMJ24, compared with protein-coding genes. Our data suggest that JMJ24 plays a conserved role in promoting basal level transcription of endogenous silenced loci to reinforce the silencing. We also provide evidence of a physical association between JMJ24 and RNA-dependent RNA polymerase 2 (RDR2), which represents an evolved property of the RNA silencing pathway. 