Spotting the Signal

photodune 4170906 stock market chart l

Blacklight Helps Discover Sparse Clues for Stock Performance

Why It's Important:  From the smallest retirement investors to the highest-tech hedge fund manager on Wall Street, we’re all trying to anticipate the market.

Should I buy stock A? If I do, when should I sell it? Finance researchers have long suspected the most talented traders use their experience and intuition to pick out market details—performance of key stocks, for example—that help them guess whether a particular stock of interest is likely to go up or down. But how do they identify such “predictor” stocks, and how few predictors do you need? Or could it be the idea of predictor stocks is a myth?


Our intuition is that a lot of these signals are appearing and [experienced traders] know what they mean, but they’re too fleeting to get picked up by most traditional statistical methods…Where the supercomputer comes in is that we were able to analyze the…data among some 7,600 variables relatively quickly. —Adam Daniel Clark-Joseph, University of Illinois

How PSC Helped: University of Illinois finance professors Adam D. Clark-Joseph, Alex Chinco and Mao Ye teamed up to study whether “sparse signals”—small numbers of predictor stocks—actually exist in the markets. With help from PSC staff, Clark-Joseph and his coauthors leveraged the large shared memory and numbers of processors of PSC’s Blacklight to test this idea using nine months of minute-by minute New York Stock Exchange returns for thousands of stocks. Out of the roughly 2,200 stocks they analyzed each month, the researchers could statistically identify about 12 predictor stocks at a time that helped forecast the performance of a given “target” stock. But these predictors were only briefly relevant to the target, with 90 percent of predictor stocks remaining relevant for four minutes or less. Adding the fleeting predictor-stock information improved forecasts by a factor of nearly 1.5 compared with standard forecasting methods that target a stock’s historical performance alone. Even more interesting, more or less the same group of predictor stocks were relevant to numerous other stocks at a given moment.

Future work, including using PSC’s Bridges system, will explore the deeper structure that could underlie these fleeting relationships between seemingly unrelated stocks.


Over roughly the past decade, PSC’s systems have pioneered the use of massive, shared memory to allow researchers to make many calculations against vast datasets in parallel. In 2009, work by Alessandro Acquisti and collaborators at Carnegie Mellon University using PSC’s Pople system—PSC’s first large-memory supercomputer—sounded a clarion warning. Publicly available information on the Internet, they found, can be used in some cases to predict individuals’ social security numbers. Pople’s combination of many processors and large shared memory allowed the team to run many variations of their alorithms against the same data in parallel. The fact that many of us may be vulnerable to identity theft without actually posting sensitive information to the Web gained national attention.

System status

  Bridges status

Bridges is up and running normally.

Featured Projects

All about Bridges

Bridges 4c stacked

Allocation requests now being accepted.


Data Exacell (DXC)



XSEDE Service Provider

image gallery
