The recession has claimed millions of jobs and permanently sidelined many workers. That could leave the economy less productive than it was before, sapping growth, while also imposing large social costs.

Terry Hipolito opted to retire rather than keep looking for work. Photo by David McNew for The Wall Street Journal

At 9.4%, the unemployment rate is double what it was before the recession started. But even that doesn't wholly reflect the hit the labor market took. That's because the unemployment rate includes only unemployed people who are looking for a job, not those workers who retired, went back to school, applied for long-term disability or simply gave up as a result of the recession.

Take Terry Hipolito, of Placentia, Calif. Even though he just turned 71, he reckons he would still be working as a software developer if the recession hadn't intervened. The last job he had—contract work for Pacific Investment Management Co.—ended in mid-2008.

"I looked around for a job for a while, and saw nothing that was close to what I was interested in," he said. "I can get Social Security and my wife is retired, so I'm not going to bust my hump for something that's irritating."

The broadest way to look at U.S. employment is the employment-to-population ratio—the number of people who are employed as a percentage of the population age 16 or older who aren't serving in the Armed Forces or in institutions such as prison or nursing homes. It has fallen significantly, dropping to 58.3% from 62.7% over the past three years. That has brought it to its lowest levels since the early 1980s, when far fewer women were in the labor force. Over time, an improving job market will raise the employment-to-population ratio.

Tanchel Reasor, left, and Stacie Stemmerman in a class at Southwestern Oregon Community College in Coos Bay, Ore. Benjamin Brayfield for The Wall Street Journal

The biggest problem areas, said Harvard University economist Lawrence Katz, are young people who haven't entered the labor force and a large number of people who have applied for long-term disability benefits after losing their jobs.

The employment-to-population ratio for people aged 20 to 24 has fallen to 60.1% from 67.1% over the past three years. That is in part because many young people opted to continue their education in response to the recession. But there are a large number of jobless young men, in particular, who are neither looking for work nor going to school. Because they aren't building skills, if they do enter the work force many will be a step behind their peers. Some may also turn to crime, though crime rates have fallen since the recession's outset.

The sharp deterioration in the job market during the downturn also seems to have affected applications to the Social Security Disability Insurance program, which rose to 2.94 million last year, up from 2.19 million in 2007.

"If you lose your job, and you already have an existing condition, where do you go then for health coverage?" said Bobby Peterson, executive director of Madison, Wisc. public-interest law firm ABC for Health, which helps people seeking disability benefits through the application process. About half of all disability-insurance applicants are eventually awarded benefits, and only rarely re-enter the labor force.

On the plus side, many people aren't in the labor force because they have opted to go to school instead—a move that will make them more productive workers when they finally get back on the job.

When Tanchel Reasor, a 33-year-old mother of two who lives in Coos Bay, Ore., lost her job as a human-resources manager at a Macy's last February, she found few opportunities for work that paid well.

"I live in a smaller town. To find another job as an HR manager is pretty much impossible," she said. In June, she started taking classes at Southwestern Oregon Community College, and is on track to complete an associates degree as a medical assistant by next fall. After that, she plans to go back to work while continuing to take classes, with the aim of eventually getting a nursing degree.

Write to Justin Lahart at