Epigenetic mechanisms can be influenced by environmental cues and thus evoke phenotypic variation. This plasticity can be advantageous for adaptation but also detrimental if not tightly controlled. Although having attracted considerable interest, it remains largely unknown if and how environmental cues such as temperature trigger epigenetic alterations. Using fission yeast, we demonstrate that environmentally induced discontinuous phenotypic variation is buffered by a negative feedback loop that involves the RNase Dicer and the protein disaggregase Hsp104. In the absence of Hsp104, Dicer accumulates in cytoplasmic inclusions and heterochromatin becomes unstable at elevated temperatures, an epigenetic state inherited for many cell divisions after the heat stress. Loss of Dicer leads to toxic aggregation of an exogenous prionogenic protein. Our results highlight the importance of feedback regulation in building epigenetic memory and uncover Hsp104 and Dicer as homeostatic controllers that buffer environmentally induced stochastic epigenetic variation and toxic aggregation of prionogenic proteins. 