he turned back to the fishing , keeping his face turned away from the other , knowing that sooner or later the penny would have to drop , and the less time they spent facing each other , the better the chances of postponing the recognition .