Only members of the imperial family and high-ranking priests are allowed past the second of the four fences.
There are four fences, and you can only go past the second one if you are a member of the imperial family, or a high-ranking priest.