First, even if punishment is inevitable, it falls not on the owners at the time when the sin is committed, but on the owners at the time when the sin is discovered.
Sins are often only discovered long after they have been committed. 