The decision marked a setback in the agency’s push to hold executives accountable in connection with the U.S. housing meltdown and financial crisis, and came after a jury in July returned a mixed verdict in the initial trial in the case.