The only objects not in causal contact are those for which there is no event in the history of the universe that could have sent a beam of light to both. For example, if the universe were not expanding and had existed for 10 billion years, anything more than 20 billion light-years away from the earth would not be in causal contact with it. Anything less than 20 billion light-years away would because an event occurring 10 billion years in the past that was 10 billion light-years away from both the earth and the object under question could have affected both.