Friday, October 30, 2009

Selectivity of US colleges and returns to elite education

Below is some work on US college admissions by economist Caroline Hoxby. There has been a recent increase in the number of American students who are interested in attending an elite university, possibly far from home. This accounts for almost all of the increased competitiveness of college admissions, and the effect is confined to only a small subset of elite schools. Selectivity at typical state universities has not increased.

These observations are consistent with my own experience. Growing up in Iowa, I remember many academically talented students from my high school (often children of professors) simply attending the local state university by default. Today, those same kids would be angling for admission to the Ivy league, Stanford, MIT, Caltech, etc.

However, it's not clear what caused this increased interest in elite education...

Near the end of the paper Hoxby reviews the literature on the returns to elite higher education. Her estimates of the rate of return are on the high side (definitely worthwhile, in her Harvard summa and Rhodes opinion ;-), and she casts some doubt on the Dale and Krueger study which compared elite graduates against others who were accepted to an elite school but chose not to attend. Dale and Krueger concluded that the choice of college had little impact on future earnings, but Hoxby suggests there was something "odd" about the 1 in 10 kids who declined elite admission :-)

See related posts returns to elite education , higher education and human capital.

Note to efficient market / rational agent / Chicago School believers: apparently even leading economists cannot agree on the size of returns to elite education. The economists, who are able to look backward in time and use massive statistics, have a much easier task than an individual student, who has to look forward 30 years to decide which choice is better. (Sound like projecting the cash flows from a bundle of mortgages over the next 30 years?) Therefore, it is hard to believe that individuals are clever enough to act like "rational agents" in solving this problem, or making any other life decision.

The Changing Selectivity of American Colleges

Caroline M. Hoxby
NBER Working Paper No. 15446*
Issued in October 2009


More here from the Atlantic Monthly:



... Which brings us back to the Krueger-Dale thesis. Can we really be sure Hamilton is nearly as good as Harvard?

Some analysts maintain that there are indeed significant advantages to the most selective schools. For instance, a study by Caroline Hoxby, a Harvard economist who has researched college outcomes, suggests that graduates of elite schools do earn more than those of comparable ability who attended other colleges. Hoxby studied male students who entered college in 1982, and adjusted for aptitude, though she used criteria different from those employed by Krueger and Dale. She projected that among students of similar aptitude, those who attended the most selective colleges would earn an average of $2.9 million during their careers; those who attended the next most selective colleges would earn $2.8 million; and those who attended all other colleges would average $2.5 million. This helped convince Hoxby that top applicants should, in fact, lust after the most exclusive possibilities.

"There's a clear benefit to the top fifty or so colleges," she says. "Connections made at the top schools matter. It's not so much that you meet the son of a wealthy banker and his father offers you a job, but that you meet specialists and experts who are on campus for conferences and speeches. The conference networking scene is much better at the elite universities." Hoxby estimates that about three quarters of the educational benefit a student receives is determined by his or her effort and abilities, and should be more or less the same at any good college. The remaining quarter, she thinks, is determined by the status of the school—higher-status schools have more resources and better networking opportunities, and surround top students with other top students.

"Today there are large numbers of colleges with good faculty, so faculty probably isn't the explanation for the advantage at the top," Hoxby says. "Probably there is not much difference between the quality of the faculty at Princeton and at Rutgers. But there's a lot of difference between the students at those places, and some of every person's education comes from interaction with other students." Being in a super-competitive environment may cause a few students to have nervous breakdowns, but many do their best work under pressure, and the contest is keenest at the Gotta-Get-Ins. Hoxby notes that some medium-rated public universities have established internal "honors colleges" to attract top performers who might qualify for the best destinations. "Students at honors colleges in the public universities do okay, but not as well as they would do at the elite schools," Hoxby argues. The reason, she feels, is that they're not surrounded by other top-performing students.

Tuesday, October 27, 2009

Do flu vaccines work?

The more I learn about the quality standards of medical research, the less I believe what my doctor tells me! Notice how hard it is, even in a "scientific" context to oppose the conventional wisdom.

Related posts: bounded cognition, Bouchard against group think, the future of innovation.

More from Gary Taubes on diet and nutrition issues, and WIRED on placebo mysteries.

As a postdoc I briefly dated a woman who was doing graduate work at Harvard's school of public health. I was a bit surprised at her low opinion of the scientific and statistical prowess of medical doctors, at least until I thought back to all the pre-med students I had taught while in grad school ;-) She pointed out to me that most MD's (as opposed to MD/PhD's) aren't scientists at all -- they simply apply what they're taught in medical school.


Nonetheless, in 2004, Jackson and three colleagues set out to determine whether the mortality difference between the vaccinated and the unvaccinated might be caused by a phenomenon known as the “healthy user effect.” They hypothesized that on average, people who get vaccinated are simply healthier than those who don’t, and thus less liable to die over the short term. People who don’t get vaccinated may be bedridden or otherwise too sick to go get a shot. They may also be more likely to succumb to flu or any other illness, because they are generally older and sicker. To test their thesis, Jackson and her colleagues combed through eight years of medical data on more than 72,000 people 65 and older. They looked at who got flu shots and who didn’t. Then they examined which group’s members were more likely to die of any cause when it was not flu season.


The results were also so unexpected that many experts simply refused to believe them. Jackson’s papers were turned down for publication in the top-ranked medical journals. One flu expert who reviewed her studies for the Journal of the American Medical Association wrote, “To accept these results would be to say that the earth is flat!” When the papers were finally published in 2006, in the less prominent International Journal of Epidemiology, they were largely ignored by doctors and public-health officials. “The answer I got,” says Jackson, “was not the right answer.”

... THE MOST vocal—and undoubtedly most vexing—critic of the gospel of flu vaccine is the Cochrane Collaboration’s Jefferson, who’s also an epidemiologist trained at the famed London School of Tropical Hygiene, and who, in Lisa Jackson’s view, makes other skeptics seem “moderate by comparison.” Among his fellow flu researchers, Jefferson’s outspokenness has made him something of a pariah. At a 2007 meeting on pandemic preparedness at a hotel in Bethesda, Maryland, Jefferson, who’d been invited to speak at the conference, was not greeted by any of the colleagues milling about the lobby. He ate his meals in the hotel restaurant alone, surrounded by scientists chatting amiably at other tables. He shrugs off such treatment. As a medical officer working for the United Nations in 1992, during the siege of Sarajevo, he and other peacekeepers were captured and held for more than a month by militiamen brandishing AK-47s and reeking of alcohol. Professional shunning seems trivial by comparison, he says.

“Tom Jefferson has taken a lot of heat just for saying, ‘Here’s the evidence: it’s not very good,’” says Majumdar. “The reaction has been so dogmatic and even hysterical that you’d think he was advocating stealing babies.” Yet while other flu researchers may not like what Jefferson has to say, they cannot ignore the fact that he knows the flu-vaccine literature better than anyone else on the planet. He leads an international team of researchers who have combed through hundreds of flu-vaccine studies. The vast majority of the studies were deeply flawed, says Jefferson. “Rubbish is not a scientific term, but I think it’s the term that applies.” Only four studies were properly designed to pin down the effectiveness of flu vaccine, he says, and two of those showed that it might be effective in certain groups of patients, such as school-age children with no underlying health issues like asthma. The other two showed equivocal results or no benefit.

... In the flu-vaccine world, Jefferson’s call for placebo-controlled studies is considered so radical that even some of his fellow skeptics oppose it.

Saturday, October 24, 2009

Eric Baum: What is Thought?

Last week we had AI researcher and former physicist Eric Baum here as our colloquium speaker. (See here for an 11 minute video of a similar, but shorter, talk he gave at the 2008 Singularity Summit.)

Here's what I wrote about Baum and his book What is Thought back in 2008:

My favorite book on AI is Eric Baum's What is Thought? (Google books version). Baum (former theoretical physicist retooled as computer scientist) notes that evolution has compressed a huge amount of information in the structure of our brains (and genes), a process that AI would have to somehow replicate. A very crude estimate of the amount of computational power used by nature in this process leads to a pessimistic prognosis for AI even if one is willing to extrapolate Moore's Law well into the future. Most naive analyses of AI and computational power only ask what is required to simulate a human brain, but do not ask what is required to evolve one. I would guess that our best hope is to cheat by using what nature has already given us -- emulating the human brain as much as possible.

This perspective seems quite obvious now that I have kids -- their rate of learning about the world is clearly enhanced by pre-evolved capabilities. They're not generalized learning engines -- they're optimized to do things like recognize patterns (e.g., faces), use specific concepts (e.g., integers), communicate using language, etc.

What is Thought?



When I first looked at What is Thought? I was under the impression that Baum's meaning, underlying structure and compact program were defined in terms of algorithmic complexity. However, it's more complicated than that. While Nature is governed by an algorithmically simple program (the Standard Model Hamiltonian can, after all, be written down on a single sheet of paper) a useful evolved program has to run in a reasonable amount of time, under resource (memory, CPU) constraints that Nature itself does not face. Compressible does not imply tractable -- all of physics might reduce to a compact Theory of Everything, but it probably won't be very useful for designing jet airplanes.

Useful programs have to be efficient in many ways -- algorithmically and computationally. So it's not a tautology that Nature is very compressible, therefore there must exist compact (useful) programs that exploit this compressibility. It's important that there are many intermediate levels of compression (i.e., description -- as in quarks vs molecules vs bulk solids vs people), and computationally effective programs to deal with those levels. I'm not sure what measure is used in computer science to encompass both algorithmic and computational complexity. Baum discusses something called minimum description length, but it's not clear to me exactly how the requirement of effective means of computation is formalized. In the language of physicists, Baum's compact (useful) programs are like effective field theories incorporating the relevant degrees of freedom for a certain problem -- they are not only a compressed model of the phenomena, but also allow simple computations.

Evolution has, using a tremendous amount of computational power, found these programs, and our best hope for AI is to exploit their existence to speed our progress. If Baum is correct, the future may be machine learning guided by human Mechanical Turk workers.

Baum has recently relocated to Berkeley to pursue a startup based on his ideas. (Ah, the excitement! I did the same in 2000 ...) His first project is to develop a world class Go program (no lack of ambition :-), with more practical applications down the road. Best of Luck!

Friday, October 23, 2009

Pessimism of the Intellect, Optimism of the Will

I often get questions about the slogan on my homepage:

Pessimism of the Intellect, Optimism of the Will.

Some recognize that it's a quote from Antonio Gramsci and mistakenly think it means I'm a Marxist.

In fact, the meaning of the slogan is quite simple.

Pessimism of the Intellect means, simply, be a scientist: see the world as it really is, not as you might like it to be. Try to identify and, if necessary, overcome hidden biases or prior assumptions. Always ask yourself: What assumption am I making? What if it is incorrect? How do I know what I know?

In many cases, the correct answer is: I don't know. Never be afraid to admit you don't know.

Optimism of the Will means, have the courage to attempt difficult things. Sometimes, Will can overcome the odds.

I'll fight them, I'll fight them until I die. --- The Old Man and the Sea, Ernest Hemingway

Thursday, October 22, 2009

Evolution, Design and the Fermi Paradox




It seems more plausible that T is much larger than 5 Gyr, in which case we were lucky, in a sense I will explain. Inflationary cosmology predicts a very large universe (much larger than what is currently visible to us), so that complex life is likely to exist somewhere in the universe. Conditioning on our own existence (a use of the weak anthropic principle), we should not be surprised to find ourselves lucky -- the few Earth-like planets that manage to evolve life must do so before their suns die. Intelligent beings, while not likely to evolve on any particular Earth-like planet, are likely to observe an evolutionary history that took place over a fraction of 10 Gyr.

Why should T be so large? At present we are unable to make quantitative estimates for the rate of evolution from first principles. It is entirely possible that certain evolutionary steps were highly improbable, such as the appearance of the first self-replicating complex molecules. One can also imagine abstract fitness surfaces with local maxima that trap the system for exponentially long periods of time.

I would not be surprised to find that T is exponentially larger than 5 Gyr. Godel went so far as to propose: "... a mathematical theorem to the effect that the formation within geological times of a human body by the laws of physics (or any other law of a similar nature) starting from a random distribution of the elementary particles and the field, is about as unlikely as the separation by chance of the atmosphere into its components."

The framework described above makes the following predictions:

1. The overwhelming majority of Earth-like planets are devoid of life, thereby resolving the Fermi Paradox.

2. Improved understanding of evolution will uncover highly improbable steps -- that is, improbable even over billions (or perhaps 10^100 !) of years. The fact that life on Earth climbed these steps might suggest intelligent design or divine intervention, but is better explained by the anthropic principle.

See related post evolutionary time scales.

Note added: This idea came to me after reading some discussion of ID and the question of improbable steps in evolution. (Here improbable means, for example, that even in an Earth-size population the multiple simultaneous mutations required jump across a particular fitness valley are unlikely to occur in 5 Gyr given the known mutation rate.) It occurred to me that under certain assumptions what might appear to be ID could actually be due to selection bias -- not taking into account the possibility that complex life is rare even on Earth-like planets. It turns out that the idea is not new -- it goes all the way back to Brandon Carter's 1983 paper which (I believe) coined the term "anthropic principle"! See John D. Barrow and Frank J. Tipler, The Anthropic Cosmological Principle, p. 557 for more. Also see this 1998 paper by Robin Hanson (thanks to a commenter for pointing it out). If one assumes many improbable required steps, one can deduce an upper estimate on the remaining time over which favorable conditions can persist on Earth. Note it appears the remaining lifetime of the sun is less than 1 Gyr, so the coincidence is tighter than I suggested in the original post. As the number of required improbable steps increases the likelihood that we would evolve just before time runs out (favorable conditions end) becomes very high -- this is quantified in the two references above.

The point which I have (still) not seen discussed much is that biologists need not be so defensive about improbable steps in evolution. I sense an almost reflexive -- i.e., prior-driven -- response to any claims of improbability. (On the other hand, perhaps biologists just know more about the details: the claim would be that the historical record does not resemble a typical one that would be generated by a chain of improbable events. I think this question requires further study.) But the main takeaway from this analysis is that improbability does not imply design or intervention.

Tuesday, October 20, 2009

Affirmative action: the numbers

Princeton sociologist Thomas Espenshade has a new book on elite college admissions. He is an author of an earlier study which quantified the advantages and disadvantages (''racial spoils''?) conferred by ethnic group at virtually all universities (with only a few notable exceptions such as Caltech). Already, some results from the analysis in the book have leaked out, and the numbers are not pretty -- see below.




To clarify, 1150 is similar to the average at a typical state university, 1460 similar to the average at a top Ivy, and 1600 is obtained by only a few thousand high school seniors each year (note, this is post-1995 re-centering of the SAT). Cognitively, these are very different groups of kids.

Related posts: Asian-Americans hurt by affirmative action, the ugly truth, the price of admission, Asians at Berkeley, the new white flight.

See these slides for more results. The bottom slide below suggests that if race-blind admissions were enacted the percentage of Asian students at elite universities would jump from 24 percent to 39 percent. Note that 39 percent is similar to the current Asian populations at Caltech and Berkeley, two elite institutions with (roughly) race-blind admissions; the former due to meritocratic idealism, the latter thanks to Proposition 209.

(Click for larger versions.)

Universe Splitter -- iPhone app

New! For your iPhone :-)

Thanks to a correspondent who runs a quant hedge fund and who once worked on phase transitions in the early universe ;-)

Universe Splitter

Tough decision? There's no need to choose — Split the universe and do both!

Scientists say that every quantum event plays out simultaneously in every possible way, with each possibility becoming real in a separate universe. You can now harness this powerful and mysterious effect right from your iPhone or iPod Touch!

How? Whenever you're faced with a choice — for example, whether to accept a job offer or to turn it down — just type both of these actions into Universe Splitter©, and press the button.

Universe Splitter© will immediately contact a laboratory in Geneva, Switzerland, and connect to a Quantis brand quantum device, which releases single photons into a partially-silvered mirror. Each photon will simultaneously bounce off the mirror and pass through it — but in separate universes.*

Within seconds, Universe Splitter© will receive the experiment's result and tell you which of the two universes you're in, and therefore which action to take. Think of it — two entire universes, complete with every last planet and galaxy, and in one, a version of you who took the new job, and in the other, a version of you who didn't!*

Monday, October 19, 2009

Posner: How I became a Keynesian

Somehow I missed this! Thanks to a reader for pointing it out to me.

Posner was as captured by Chicago School nonsense as anyone else, but at least we learn that he can perform a Bayesian update (i.e., learn from reality) -- posteriors need not be wholly determined by priors :-)

Strangely, I don't see any discussion of this article on the Becker-Posner blog. How do Gary Becker and Robert Lucas feel about the recent apostasy of their colleague?

How I Became a Keynesian, by Richard Posner

... I had never thought to read The General Theory of Employment, Interest, and Money, despite my interest in economics.


... The dominant conception of economics today, and one that has guided my own academic work in the economics of law, is that economics is the study of rational choice. People are assumed to make rational decisions across the entire range of human choice, including but not limited to market transactions, by employing a form (usually truncated and informal) of cost-benefit analysis. The older view was that economics is the study of the economy, employing whatever assumptions seem realistic and whatever analytical methods come to hand. Keynes wanted to be realistic about decision-making rather than explore how far an economist could get by assuming that people really do base decisions on some approximation to cost-benefit analysis.

... It is an especially difficult read for present-day academic economists, because it is based on a conception of economics remote from theirs. This is what made the book seem "outdated" to Mankiw--and has made it, indeed, a largely unread classic. (Another very distinguished macroeconomist, Robert Lucas, writing a few years after Mankiw, dismissed The General Theory as "an ideological event.") The dominant conception of economics today, and one that has guided my own academic work in the economics of law, is that economics is the study of rational choice. People are assumed to make rational decisions across the entire range of human choice, including but not limited to market transactions, by employing a form (usually truncated and informal) of cost-benefit analysis. The older view was that economics is the study of the economy, employing whatever assumptions seem realistic and whatever analytical methods come to hand. Keynes wanted to be realistic about decision-making rather than explore how far an economist could get by assuming that people really do base decisions on some approximation to cost-benefit analysis.

The General Theory is full of interesting psychological observations--the word "psychological" is ubiquitous--as when Keynes notes that "during a boom the popular estimation of [risk] is apt to become unusually and imprudently low," while during a bust the "animal spirits" of entrepreneurs droop. He uses such insights without trying to fit them into a model of rational decision-making.

An eclectic approach to economic behavior came naturally to Keynes, because he was not an academic economist in the modern sense. He had no degree in economics, and wrote extensively in other fields (such as probability theory--on which he wrote a treatise that does not mention economics). He combined a fellowship at Cambridge with extensive government service as an adviser and high-level civil servant, and was an active speculator, polemicist, and journalist. He lived in the company of writers and was an ardent balletomane.

... The third claim that I am calling foundational for Keynes's theory--that the business environment is marked by uncertainty in the sense of risk that cannot be calculated--now enters the picture. Savers do not direct how their savings will be used by entrepreneurs; entrepreneurs do, guided by the hope of making profits. But when an investment project will take years to complete before it begins to generate a profit, its prospects for success will be shadowed by all sorts of unpredictable contingencies, having to do with costs, consumer preferences, actions by competitors, government policy, and economic conditions generally. Skidelsky puts this well in his new book: "An unmanaged capitalist economy is inherently unstable. Neither profit expectations nor the rate of interest are solidly anchored in the underlying forces of productivity and thrift. They are driven by uncertain and fluctuating expectations about the future." Only what Keynes called "animal spirits," or the "urge to action," will persuade businessmen to embark on such a sea of uncertainty. "If human nature felt no temptation to take a chance, no satisfaction (profit apart) in constructing a factory, a railway, a mine or a farm, there might not be much investment merely as a result of cold calculation."

But however high-spirited a businessman may be, often the uncertainty of the business environment will make him reluctant to invest. His reluctance will be all the greater if savers are hesitant to part with their money because of their own uncertainties about future interest rates, default risks, and possible emergency needs for cash to pay off debts or to meet unexpected expenses. The greater the propensity to hoard, the higher the interest rate that a businessman will have to pay for the capital that he requires for investment. And since interest expense is greater the longer a loan is outstanding, a high interest rate will have an especially dampening effect on projects that, being intended to meet consumption needs beyond the immediate future, take a long time to complete.

... An ambitious public-works program can be a confidence builder. It shows that government means (to help) business. "The return of confidence," Keynes explains, "is the aspect of the slump which bankers and businessmen have been right in emphasizing, and which the economists who have put their faith in a ‘purely monetary' remedy have underestimated." In a possible gesture toward Roosevelt's first inaugural ("we have nothing to fear but fear itself"), Keynes remarks upon "the uncontrollable and disobedient psychology of the business world."

See also my talk (for physicists) on the financial crisis. Some related posts on Keynes. Even more on Keynes (12th Wrangler) here.

Sunday, October 18, 2009

Efficient Frontier and William Bernstein

Last week we had polymath William Bernstein (finance thinker, author, portfolio manager, former neurologist and Berkeley PhD in chemistry) on campus for a seminar. I'd been a reader of his site Efficient Frontier for many years, and when I learned by chance that he lives in Oregon, invited him to visit.

Here is an Amazon link to Bernstein's books -- ranging from practical investing advice to histories of global trade and of the birth of prosperity in the modern world.

Below is an excerpt from the preface of his new book.

... I have come to the sad conclusion that only a tiny minority will ever succeed in managing their money even tolerably well.

Successful investors need four abilities. First, they must possess an interest in the process. It is no different from carpentry, gardening, or parenting. If money management is not enjoyable, then a lousy job inevitably results, and, unfortunately, most people enjoy finance about as much as they do root canal work.

Second, investors need more than a bit of math horsepower, far beyond simple arithmetic and algebra, or even the ability to manipulate a spreadsheet. Mastering the basics of investment theory requires an understanding of the laws of probability and a working knowledge of statistics. Sadly, as one financial columnist explained to me more than a decade ago, fractions are a stretch for 90 percent of the population.

Third, investors need a firm grasp of financial history, from the South Sea Bubble to the Great Depression. Alas, as we shall soon see, this is something that even professionals have real trouble with.

Even if investors possess all three of these abilities, it will all be for naught if they do not have a fourth one: the emotional discipline to execute their planned strategy faithfully, come hell, high water, or the apparent end of capitalism as we know it. “ Stay the course ” : It sounds so easy when uttered at high tide. Unfortunately, when the water recedes, it is not. I expect no more than 10 percent of the population passes muster on each of the above counts. This suggests that as few as one person in ten thousand (10 percent to the fourth power) has the full skill set. Perhaps I am being overly pessimistic. After all, these four abilities may not be entirely independent: if someone is smart enough, it is also more likely he or she will be interested in finance and be driven to delve into financial history.

But even the most optimistic assumptions — increase the odds at any of the four steps to 30 percent and link them -- suggests that no more than a few percent of the population is qualified to manage their own money. And even with the requisite skill set, more than a little moxie is involved. This last requirement — the ability to deploy what legendary investor Charley Ellis calls “ the emotional game ” — is completely independent of the other three; Wall Street is littered with the bones of those who knew just what to do, but could not bring themselves to do it.

It's a sobering thought that so few people make competent money managers, given that most Americans now manage their own retirement accounts. During dinner, we refined his estimate (taking into account correlation) of the fraction of competent investors to about 1 in 1000.

Although I had thought about this question in the past, and even identified the various factors, it hadn't occurred to me until dinner with Bernstein that this was a good reason to suggest finance as a career to someone who has the right interests (history, finance theory, markets -- relatively easily acquired, as these subjects are fascinating), personality factors (discipline, controlled risk taking, decisiveness -- not so easily acquired, but can be improved over time) and intelligence (not easily acquired, but perhaps the threshold isn't that high at 90th percentile). No one factor is particularly rare, but the combination certainly is. As long as the signal to noise ratio in a financial career is reasonable, a person with all these qualities can be confident of success. While only a few people will ever run a hedge fund, it would seem that almost everyone could benefit from help managing their money -- if the advisor has all the qualities listed above.

Thursday, October 15, 2009

Calvin Trillin on Wall Street smarts

Calvin Trillin, one of my favorite New Yorker writers and a keen observer of American society, writes in the Times. It's true that the quants built machines (mortgage finance) that were too complex for their masters to operate. Related posts: The new new meme, The best and brightest, The bubble algorithm ...

(This may sound crazy, but does Trillin read this blog?)

NYTimes: Wall Street Smarts



The statement came from a man sitting three or four stools away from me in a sparsely populated Midtown bar, where I was waiting for a friend. “But I have to buy you a drink to hear it?” I asked.

“Absolutely not,” he said. “I can buy my own drinks. My 401(k) is intact. I got out of the market 8 or 10 years ago, when I saw what was happening.”

He did indeed look capable of buying his own drinks — one of which, a dry martini, straight up, was on the bar in front of him. He was a well-preserved, gray-haired man of about retirement age, dressed in the same sort of clothes he must have worn on some Ivy League campus in the late ’50s or early ’60s — a tweed jacket, gray pants, a blue button-down shirt and a club tie that, seen from a distance, seemed adorned with tiny brussels sprouts.




He looked at me the way a mathematics teacher might look at a child who, despite heroic efforts by the teacher, seemed incapable of learning the most rudimentary principles of long division. “You are either a lot younger than you look or you don’t have much of a memory,” he said. “One of the speakers at my 25th reunion said that, according to a survey he had done of those attending, income was now precisely in inverse proportion to academic standing in the class, and that was partly because everyone in the lower third of the class had become a Wall Street millionaire.”


“That actually sounds more or less accurate,” I said.

“Of course it’s accurate,” he said. “Don’t get me wrong: the guys from the lower third of the class who went to Wall Street had a lot of nice qualities. Most of them were pleasant enough. They made a good impression. And now we realize that by the standards that came later, they weren’t really greedy. They just wanted a nice house in Greenwich and maybe a sailboat. A lot of them were from families that had always been on Wall Street, so they were accustomed to nice houses in Greenwich. They didn’t feel the need to leverage the entire business so they could make the sort of money that easily supports the second oceangoing yacht.”

“So what happened?”





“But you still haven’t told me how that brought on the financial crisis.”

“Did you ever hear the word ‘derivatives’?” he said. “Do you think our guys could have invented, say, credit default swaps? Give me a break! They couldn’t have done the math.”

“Why do I get the feeling that there’s one more step in this scenario?” I said.


“So having smart guys there almost caused Wall Street to collapse.”

“You got it,” he said. “It took you awhile, but you got it.”

The theory sounded too simple to be true, but right offhand I couldn’t find any flaws in it. I found myself contemplating the sort of havoc a horde of smart guys could wreak in other industries. I saw those industries falling one by one, done in by superior intelligence. “I think I need a drink,” I said.

He nodded at my glass and made another one of those eyebrow gestures to the bartender. “Please,” he said. “Allow me.”

I highly recommend Trillin's book Remembering Denny, a meditation on life, success, complexity, the unknowability of others, and fate. Also see Amazon reviews.

Calvin Trillin's "Remembering Denny" is a Cheeveresque rumination on the unfulfilled potential of Trillin's Yale classmate, Denny Hansen. While at Yale, Hansen was so highly thought of that he was profiled in LIFE magazine and his classmates used to kid each other about which cabinet position they'd fill once Hansen had been elected President. After Yale, however, Hansen failed to live up to the high expectations everyone--friends, family, teachers, coaches--had for him. Trillin's book is a delicate examination of what that meant, both for Denny and for his constellation of friends and well-wishers.

Denny doesn't come alive as vividly as might be hoped here, but Trillin does an outstanding job of sketching this young man's life in terms of a larger picture about America. In a country where success on every level is much prized, Trillin subtly but thoroughly plumbs the reasons why Denny didn't succeed--at least not to the extent everyone thought he would. This uncharacteristically somber book is absorbing and thought-provoking, even if it doesn't quite reach the goals Trillin seems to have set for himself in the beginning chapters.

Signals from the market in Guangzhou

This NYTimes article, reported from the world's largest export tradeshow in Guangzhou, China, illustrates that

1. we're past the bottom (at least globally) for the recession

2. the dollar is in trouble, and everybody knows it

3. low-skill labor in China still has little pricing power; manufacturing outside of China is in for more tough times.

NYTimes: Throngs of buyers from around the world swarmed through the world’s largest trade show at its opening Thursday, underscoring how China’s combination of a cheap currency and low wages is producing a resurgence of exports this autumn.

As wholesalers and distributors from dozens of countries began snatching brochures and exchanging business cards at booths in exhibition halls with five times the floor area of the Empire State Building, Chinese exporters said they were convinced that sales were finally rebounding.

In one of the most telling signs of the recovery, container shipping prices from China to the West have jumped 50 percent in the past two months as exporters have scrambled for space on suddenly crowded decks. Chinese factories are hiring temporary workers to cope with a last-minute surge in orders, many of them from emerging markets but some from Europe and even from the economically weakened United States.



China’s many policies to help exporters, from tax breaks to currency market intervention, have relieved unemployment in China — but at the expense of contributing to it in other countries, and that is starting to fan trade tensions. President Barack Obama imposed tariffs starting at 35 percent on tires from China last month, and anti-dumping and anti-subsidy cases against China are piling up in front of trade tribunals around the world.

But as exporters here reviewed their orders for the coming months, they described a consistent pattern: Sales to emerging markets are recovering rapidly, demand from Europe is starting to rebound as the Chinese currency falls against the euro, and buying interest from the United States remains fairly weak.

The dollar has fallen steeply against the euro, the yen and most other currencies over the past two weeks, with a broad index of the U.S. currency dropping Thursday to its lowest level in 14 months. But the Chinese government has intervened heavily in currency markets to make sure that its currency, the yuan, falls with the dollar so that Chinese goods can maintain their competitive edge.

The result has been a steep slide in the value of the yuan against the euro, the yen, the Australian dollar and many other currencies that is making it cheaper for businesses from Helsinki to Sydney to order from China. The yuan has fallen 16 percent against the euro since early March and 31 percent against the Australian dollar.

“We export a lot to Australia,” said Linda Zhang, a sales manager at Hebei Wanlong Steel Structure, a maker of prefabricated housing, adding that demand from there “is coming back.”

Kimmo Tarkkonen, the chairman of SRS Fenno-El, a distributor of lamps and space heaters based on the outskirts of Helsinki, said that he was signing all his contracts with Chinese suppliers in dollars after concluding that the dollar would continue to fall against the euro, making his purchases even cheaper.

“We’d rather take the risk,” he said, adding that the dollar “is declining all the time, so the risk is minimal.”

Chinese exporters’ representatives at the fair said that they had not been able to raise prices for European customers as the euro rose because there was so much competition from other Chinese companies.

“Our price depends on our cost,” not the euro, said Nicola Hou, a marketer at Meizhou Koway Electronics, a clock radio manufacturer in Meizhou City in southeastern China.

Some Chinese companies, in another effort to remain competitive, have even been cutting prices after wages fell during the recent economic downturn — a decline from which they have not fully recovered. Most of the country has not yet seen a return of the labor shortages that started appearing shortly before the global economic downturn began.

“It’s easy to find workers — China has too much labor,” said Helen Chen, general manager of Yuyao Panasia International Trading in Yuyao in east-central China.

Transport costs are a small fraction of labor costs, but they are now rebounding from a vertiginous plunge over the winter. ...

Sunday, October 11, 2009

Skidelsky at LSE

I highly recommend this brilliant LSE lecture by Robert Skidelsky. Among the topics covered: risk versus uncertainty; Chicago school as the academic scribblers responsible for recent disastrous ideological capture; rational expectations, efficient markets and all that nonsense; economics as a science (not) and the role of mathematics; fiscal versus monetary stimulus; Glass-Steagall; Capital ascendant over Labor and Government: the renewed relevance of Marx; the role of globalization and the neoliberal agenda.

Before attacking me as a socialist pinko (I am not), listen to the talk or at least read the earlier post linked to below.

Keynes and the Crisis of Capitalism
(podcast and video available)

Date: Wednesday 7 October 2009
Speaker: Professor Lord Skidelsky


This event celebrates his latest book, Keynes: The Return of the Master.

For more from Skidelsky on Keynes and the current crisis, see this earlier post.


Saturday, October 10, 2009

Spooks drowning in data

Almost every technical endeavor, from finance to high energy physics to biology to internet security to spycraft, is either already or soon to be drowning in Big Data. This is an inevitable consequence of exponential Moore's Laws in bandwidth, processing power, and storage, combined with improved "sensing" capability. The challenge is extracting meaning from all that data.

My impression is that the limiting factor at the moment is the human brainpower necessary to understand the idiosyncrasies of the particular problem, and, simultaneously, develop the appropriate algorithms. There are simply not enough people around who are good at this; it's not just a matter of algorithms, you need insight into the specific situation. Of equal importance is that the (usually non-technical) decision makers who have to act on the data need to have some rough grasp of the strengths and limitations of the methods, so as not to have to treat the results as coming from a black box.

To give you my little example of big data, on my desk (in Oakland, not in Eugene) I have stacks of terabyte drives with copies of essentially every Windows executable (program that runs on a flavor of Windows) that has appeared on the web in the past few years (about 5 percent of this is malware; also stored in our data is what each executable does once it's installed). Gathering this data was only modestly hard; analyzing it in a meaningful way is a lot harder!




... Where does all this leave us? Aid concludes that the biggest problem facing the agency is not the fact that it's drowning in untranslated, indecipherable, and mostly unusable data, problems that the troubled new modernization plan, Turbulence, is supposed to eventually fix. "These problems may, in fact, be the tip of the iceberg," he writes. Instead, what the agency needs most, Aid says, is more power. But the type of power to which he is referring is the kind that comes from electrical substations, not statutes. "As strange as it may sound," he writes, "one of the most urgent problems facing NSA is a severe shortage of electrical power." With supercomputers measured by the acre and estimated $70 million annual electricity bills for its headquarters, the agency has begun browning out, which is the reason for locating its new data centers in Utah and Texas. And as it pleads for more money to construct newer and bigger power generators, Aid notes, Congress is balking.


Rather than give the NSA more money for more power—electrical and political—some have instead suggested just pulling the plug. "NSA can point to things they have obtained that have been useful," Aid quotes former senior State Department official Herbert Levin, a longtime customer of the agency, "but whether they're worth the billions that are spent, is a genuine question in my mind."


But pulling the plug, or even allowing the lights to dim, seems unlikely given President Obama's hawkish policies in Afghanistan. However, if the war there turns out to be the train wreck many predict, then Obama may decide to take a much closer look at the spy world's most lavish spender. It is a prospect that has some in the Library of Babel very nervous. "It was a great ride while it lasted," said one.

Friday, October 09, 2009

A Nobel for Barry?

Huh? In a few years he might have actually earned one. This just demonstrates the outright political biases of the Nobel Committee. The Nobel citation reads like this (without actually mentioning GWB): "Bush era bad, Obama good. Hope good."

Note I speak as an Obama supporter!

Will pundits on the left have the guts to point out that the emperor has no clothes? This kind of outcome just reinforces the paranoid fantasies of the far-right: that Obama's Harvard magna is fake, that the World Government has been grooming him for leadership since his student days, that Bill Ayers wrote Dreams From My Father, etc.

Speaking more broadly, it seems to me that our obsession with prizes (an offshoot of winner-take-all culture) is unhealthy, and that prizes these days are less and less correlated with actual achievement.

While my faith in the Nobel process is shaken (although commenters have already pointed out the Kissinger prize and of course there is always Modigliani-Miller ;-), my confidence in Obama is not -- he reacted properly.

Mr. Obama said he was "surprised and deeply humbled" by the committee’s decision, ... he said he would accept it as “a call to action.”


Someone just pointed out to me that this opens the door for a string theorist to win the physics prize ;-)

Wednesday, October 07, 2009

Schrodinger's virus

If the creature above (a tardigrade arthropod) can be placed in a superposition state, will you accept that you probably can be as well? And once you admit this, will you accept that you probably actually DO exist in a superposition state already?

It may be disturbing to learn that we live in a huge quantum multiverse, but was it not also disturbing for Galileo's contemporaries to learn that we live on a giant rotating sphere, hurtling through space at 30 kilometers per second? E pur si muove!

Related posts: Many Worlds: A brief guide for the perplexed , Are you gork? , Label: many worlds









Tuesday, October 06, 2009

More Brainpower and Globalization

The Chronicle of Higher Education is running a series of articles on the competitiveness of US higher education versus Asia; see here and related links (may require subscription). See also earlier posts Meritocracy in China , Brainpower and Globalization.

This describes the steps Tsinghua University took to recruit Turing Award winner Andrew Yao, going as far as effectively creating an entire department of theoretical computer science. (Interestingly, Yao first earned a PhD in physics before switching to computer science.)

Andrew Chi-Chih Yao's trajectory suggests a genius's quick ascent to success. Born in Shanghai, he studied hard, earned two Ph.D.'s from American universities, and at age 35 was a professor of computer science at Stanford University. By 2000, when he received the A.M. Turing Award, one of the most prestigious prizes in his field, younger computer scientists were memorizing a principle bearing his name.

But when Tsinghua University invited him to return to China to lead a new, generously financed institute in 2004, he jumped at the opportunity. "I was very excited," he recalls.

In creating the Institute for Theoretical Computer Science, Mr. Yao says, he had free rein from Tsinghua's administrators to determine everything from research topics to personnel.

Once in Beijing, he threw himself into the project, emerging with an institute that is today one of the university's jewels and is increasingly known outside China.

The way Mr. Yao's institute was created illustrates a central element of China's higher-education strategy. When the government decides it wants to do something, it does it, and fast.

Theoretical computer science—the abstract, intensely mathematical subfield that is Mr. Yao's specialty—is poorly financed in the United States. It wasn't originally a target area for China, either. But China has an approach different from that found in the United States: a desire to build outstanding institutions by attracting the leaders in a field—any field.

Mr. Yao was first approached about joining Tsinghua's faculty in 2003 by Chen Ning Yang, a Nobel Prize-winning physicist, who had himself recently begun lecturing at the university.

Through that and later discussions, Mr. Yao came to understand that the university's officials were more interested in hiring world-class talent than they were in building up specific areas.

"They want to catch up in a global way, and therefore it doesn't matter where they get started," Mr. Yao explains.

That hunger translates into money for leading Western-trained scientists willing to relocate. "China is a very exciting place for science and engineering for people who have vision," he says.

'Yao's Class'

But while the chance to build a world-class institute from scratch required little consideration, how exactly to do it in a country with a developing higher-education system was a difficult question to answer.

Mr. Yao recruited his first class of graduate students, only to discover that many of them lacked basic skills. If he wanted quality applicants, he realized, he would have to train them himself. So he established an undergraduate program within the institute, selecting about 100 students each year and then identifying the most talented ones—a group now known as Yao's Class—for intensive instruction.

The fact that he had the freedom to do so reflects another element of China's higher-education strategy, in which a select number of elite universities are allowed to flourish. ...

Friday, October 02, 2009

Baidu, baby, Baidu





Thursday, October 01, 2009

Supermassive black holes and the entropy of the universe

New Scientist has an article about a recent paper by two Australian researchers (, which contains detailed estimates of the entropy of various components of the universe (black holes, neutrinos, photons, etc.). This paper is related to some work I did with with Frampton, Kephart and Reeb: What is the entropy of the universe?. (Discussion on Cosmic Variance.)

Our work did not focus on the numerical values of various contributions to the entropy (we made some simple estimates), but rather what the physical meaning is of this entropy -- in particular, that of black holes; see excerpt below.

New Scientist: Mammoth black holes push universe to its doom

30 September 2009 by Rachel Courtland

THE mammoth black holes at the centre of most galaxies may be pushing the universe closer to its final fade-out. And it is all down to the raging disorder within those dark powerhouses.

Disorder is measured by a quantity called entropy, something which has been on the rise ever since the big bang. Chas Egan and Charles Lineweaver of the Australian National University in Canberra used the latest astrophysical data to calculate the total entropy of everything in the universe, from gas to gravitons. It turns out that supermassive black holes are by far the biggest contributors to the universe's entropy. Entropy reflects the number of possible arrangements of matter and energy in an object. The number of different configurations of matter a black hole could contain is staggering because its internal state is completely mysterious.

Egan and Lineweaver found that everything within the observable universe contains about 10^104 units of entropy (joules per Kelvin), a factor of 10 to 1000 times higher than previous estimates that did not include some of the biggest known black holes (, submitted to The Astrophysical Journal).

If entropy were ever to reach a maximum level, that would mean the heat death of the universe. In this scenario no energy can flow, because everything is the same temperature and so life and other processes become impossible. "Our results suggest we're a little further along that road than previously thought," Egan says.

But although black holes do boost the universe's total entropy, it is not clear whether they will hasten its heat death. Supermassive black holes don't contribute much to the flows of heat that even out temperature throughout the universe, says physicist Stephen Hsu at the University of Oregon in Eugene.

It's true that these black holes will slowly evaporate by releasing Hawking radiation, particles created near the boundary of the black hole. And this radiation could move the universe towards heat death.

Black holes may evaporate via Hawking radiation, tipping the universe towards its heat death. However, it will take some 10^102 years for a supermassive black hole to evaporate. "The entropy inside those black holes is effectively locked up in there forever," Hsu says. So we may have reached a state approaching heat death long before, as stars burn out and their matter decays.

The large result obtained by the Egan and Lineweaver for the entropy of the universe is primarily due to supermassive black holes. How do we interpret that entropy? Here is an excerpt from our paper (excuse the latex).

Note the entropy used in this paper describes the uncertainty in the precise quantum state of a system. If the system is macroscopic the full quantum state is only accessible to a kind of ``super-observer'' who is unaffected by decoherence \cite{decoherence}. Individual observers within the system who have limited experimental capabilities can only detect particular decoherent outcomes. These outcomes arise, e.g., from an effective density matrix that results from tracing over degrees of freedom which are out of the experimenter's control (i.e., which form the ``environment''). In \cite{BID} the experimental capabilities necessary to distinguish decoherent branches of the wavefunction, or, equivalently, the precise quantum state of Hawking radiation from a black hole, are discussed. It is shown that a super-observer would either need (at minimum) the capability of making very precise measurements of accuracy $\exp(- M^2 )$ (see also the proposal of Maldacena \cite{eternal} for a specific measurement to determine whether black hole evaporation is unitary), or alternatively the capability of engineering very precise non-local operators, which measure a large fraction of the Hawking radiation at once, including correlations (i.e., as opposed to ordinary particle detectors, which only measure Fock state occupation numbers and are not sensitive to phase information).

An observer who lacks the capabilities described in the previous paragraph would be unable to distinguish the states in the $S = M^{3/2}$ subspace in Fig.~\ref{figure1} from those in the larger $S = M^2$ subspace, assuming the unitary evaporation resembles, in gross terms, Hawking evaporation, with the information hidden in correlations among the emitted quanta. In that case, the future uncertainty for ordinary (non-super) observers might be better characterized by the larger $S = M^2$ entropy. Putting it another way, an ordinary (non-super) observer is forced (due to experimental limitations) into a coarse grained description of the radiation; they cannot distinguish between most of the radiation states, and for them the $S = M^2$ entropy is appropriate. For a super-observer, however, due to unitary evolution, the uncertainty in the quantum state does not increase. For them, black holes do not have greater entropy than the precursor states from which they formed.

For the super-observers described above, the large black hole entropies in Table I do not reflect the actual uncertainties in the (current and future) state of the universe and are in that sense misleading. A black hole of mass $M$ whose formation history is typical for our universe (e.g., it originated from gravitational collapse of a star or galactic core) satisfies the bound S [less than] M^{3/2} \cite{MI}. Thus, re-evaluating the numbers in Table I, the total entropy of all black holes in our universe is not bigger than the total matter entropy: the dominant uncertainty in the precise state of the universe, at least as far as arises from known physics, is, in fact, due to CMB photons or neutrinos.


Final mysterious comment, maximally compressed for the cognoscenti: assuming unitarity, black holes do not push us closer to heat death (equilibrium) in the multiverse, but can contribute (albeit very slowly) to the (coarse grained) heat death experienced by a non-super observer (i.e., an observer subject to decoherence). See here for more on equilibrium in the multiverse.

Blog Archive