 At one point, both Han and Mu were removed from their offices for over their diversion of imperial construction personnel from their project of repairing the palace at the secondary capital Jinyang to repairing their own homes.At the time of the major attack by Northern Zhou in 577, Gao Wei initially battled Northern Zhou forces but eventually fled back to the Northern Qi capital Yecheng (鄴城, in modern Handan, Hebei), and it was at that time he recalled Han back to government.Han subsequently fled with Gao Wei south of the Yellow River and was one of the few officials who did not surrender to Northern Zhou.