Dr. John B. Calhoun, an ecologist who saw in the bleak effects of overpopulation on rats and mice a model for the future of the human race, died on Sept. 7 while on vacation in Hanover, N.H. He was 78 and lived in Bethesda, Md.

The cause was a stroke after a mild heart attack, said Kathleen Kerr, a colleague who disclosed the death this week.

In a 40-year career, mostly at the National Institute of Mental Health, Dr. Calhoun demonstrated that as population density increased, social behavior degenerated.

Among other findings, he developed the concept of universal autism -- in which all members of the last generation of mice in an increasingly crowded environment are incapable of the social behavior that would allow them to produce the next generation. And he described a phenomenon in which some mice become "beautiful ones," maintaining their physical appearance, but doing little else, as the population swells.

In one of Dr. Calhoun's experiments, a square steel box, nine feet on each side, contained 2,600 mice, about 16 times what would be considered normal density. He determined that rodents rapidly developed a hierarchy when thrown together in such huge numbers, with those closest to the food supply growing most rapidly and, because of their size, assuming higher social status.

Dr. Calhoun was born in Elkton, Tenn., and received a bachelor's degree from the University of Virginia in 1939 and a doctorate in zoology from Northwestern University in 1943.

After studies on Norway rats at Johns Hopkins University, and on mice at the Jackson Memorial Laboratory in Bar Harbor, Me., he joined the mental health institute in 1954. In 1963, he organized the unit for research on behavioral systems at the Laboratory of Brain Evolution and Behavior, a division of the N.I.M.H. He continued as chief of the unit until his retirement in 1986.

His work was the inspiration for a children's book by Robert C. O'Brien, "Mrs. Frisby and the Rats of NIMH" which became the basis for a popular animated film, "The Secret of NIMH," released in 1982.

Dr. Calhoun was a cheerful man who maintained a sense of humor about his work and the stereotypical image of scientists studying rats in mazes, Mrs. Kerr said. But his work had its frustrations as well, she noted, because its implications for the future of the human rat race were often met with studied disregard.

But Dr. Calhoun was convinced that his mice and rat populations were an accurate model for humans. "He didn't regard it as hypothesis any more, he regarded it as factual," Mrs. Kerr said.

He is survived by his wife of 52 years, Edith Gressley Calhoun of Bethesda; two daughters, Catherine Calhoun of Brooklyn and Dr. Cheryl Calhoun of Waterville, Me., and two grandchildren.