if , he thought , longsword was going to begrudge him every small moment with olwen , then he might as well move to the manor permanently and see his friends angry face only when the time came each year for him to give his service .