Pollack was born on July 9, 1938, in New York City, and was brought up in Woodmere, Long Island by a Jewish family that was in the women's garment business. He was a valedictorian of his class at Lawrence High School and graduated from Princeton University in 1960. He then received his master's in nuclear physics at University of California, Berkeley in 1962 and his Ph.D from Harvard in 1965, where he was a student of Carl Sagan. He was openly gay. Dorion Sagan told how his father came to the defense of Pollack's partner in a problem with obtaining treatment at the university health service emergency department.