


I’ve Turned Off My Ad Blocker
Edit Story

The Sociology Behind Data Silos In Pharma R&D

David Shaywitz

The data paradox vexing pharma companies – and many other healthcare organizations – is that, on the one hand, they seem to be awash in data, yet these data seem to be adding up to something that, if not less than the sum of the parts, also doesn’t seem to be much more.


Pharma researchers collect data for very specific purposes (see here, here), often with pre-specified analyses that reveal a particular molecule can or can’t proceed. Data are collected with a very specific question in mind, and generally in a fashion that allows this very specific question to be answered robustly, in an accurate, reliable, and repeatable fashion, suitable for inclusion in a data submission package for appropriately critical regulators to review and evaluate.

So what’s the problem?

The fundamental limitation of this approach is captured in a classic scene from the end of Raider’s of the Lost Ark:

Brody: Where is the Ark?

Eaton: I thought we'd settled that. The Ark is somewhere very safe.

Jones: From whom?

Brody: The Ark is a source of unspeakable power and it has to be researched.

Eaton: And it will be, I assure you, Dr. Brody, Dr. Jones. We have top men working on it right now.

Jones: Who?


This is basically the status of most pharma (and other healthcare) data today – carefully, often painstakenly collected (ideally with fewer adverse events), and then after the intended analysis, effectively locked in a crate and stuck in a warehouse somewhere.

According to many data experts who come from technology backgrounds, there is a huge missed opportunity when pharma teams silo data (which were collected to answer a specific, defined question) rather than make the data easily available for further exploration or query.

Let’s first contrast this with what happens at companies like Google and Facebook, and then consider whether this analogy is fair or appropriate.

Some of the best-known tech platforms are set up entirely to generate crunchable data – Facebook and Google, for example offer services that attract users (connection, search), and earn profits almost entirely by offering narrowly targeted ads based on data automagically collected from users thronging to the site. Amazon uses a very similar model, using user data to precisely target product offerings to the most likely purchasers. Each of these systems is designed from the ground up to maximize automatic data collection and analysis; clever algorithms designed by data scientists are used to extract insights (read: how to micro-target users) from reams of data.

If algorithms designed by data scientists can crunch search data to target users, the logic goes, why not design other algorithms to achieve something a bit more high-minded, like helping to create new medicines? This is the digital dream showing up on so many pitch decks.

There are at least two major hurdles to overcome, however.

The first is that before data can be analyzed, they must be collected, curated or cleaned, and organized in some fashion, and this turns out to be extremely difficult, as I discussed recently, inspired by this Casey Ross article in Stat. An excellent Kaveh Waddell piece in Axios this week also speaks to much the same problem.

(In the context of this post, it’s assumed that appropriate consent for data use was provided by the patients/subjects, who obviously represent the original source of data, and in whose name the research is generally conducted.)

Here, I’m going to focus on a different but equally real problem, which involves human motivation. A little context: historically, most biomedical data was collected by physicians or scientists for their own specific purposes.   Doctors wanted to remind themselves what their patients looked like, presumably to guide care they will provide in the future. Scientists wanted to remind themselves what their experiments looked like and what they demonstrated, so they could evolve their approach and ultimately share results with scientific community. The important consideration is that the person collecting the information was the one analyzing the information, and hoping to directly benefit from its use.

This perspective endures, though it’s challenged on many fronts, generating increased tension between those who want to collect data so they can analyze the data themselves, and those who want to analyze data collected by others; the “data parasite” debate ignited in 2016 was a recent manifestation of this tension.

Within pharma organizations, individual teams are responsible for delivering various data packages, which collectively comprise a product’s FDA submission.   In general, these teams want to do their intended analyses as carefully and as accurately as possible, and then move on to the next analysis; they use systems which are designed to deliver and help document the relevant aspects of the analysis.

Enter data scientists, who point to all the individual analyses and say “wouldn’t it be so much more powerful if data were collected in a way that allowed it to be more easily accessed, aggregated, and analyzed by others – data scientists, for instance – who might find new and unexpected insights here?

Publicly, the various teams respond “absolutely, we endorse this,” but privately, most are unmoved at best. From the perspective of those responsible for delivering the primary analysis – the original reason the data were collected – there is generally no perceptible upside for investing precious and limited time and capital (and quite perceptible downsides) in high-falutin data initiatives.

C-suite executives can read management consultant digital transformation white papers then preach big data and AI and data liquidity until the cows come home. The only way this is realistically going to change is through a combination of:

  • Demonstrating that collecting data in a fashion more amenable to reuse will yield unarguably valuable insights (which every startup in the space asserts – largely unpersuasively, at least so far). But truly compelling and resonant real use cases could dramatically change the calculus.
  • Dramatically reducing the burden of switching to a better system. This can be more challenging than it seems, because the data elements those designing the “ideal” system often want to include, to facilitate (futilely) the system’s ability to answer any and all future questions, will inevitably be far broader and denser than the data elements current users might supply to answer the specific questions they are trying to answer now.

In the clinical oncology area, Flatiron basically pursued a version of (b) in their efforts to transform clinical data into clinical trial-style data by minimizing the work required of busy providers, and employing an army of human data extractors to convert the information entered by practitioners into the sorts of data expected by clinical trials (e.g. line of therapy, time on therapy, etc.). Presumably, the demonstrated “regulatory grade” quality of the insights generated justified the effort and resources required.

Today, many efforts to evolve the way data are managed within pharma seem about as promising as deciding the way to get someone who doesn’t speak your language to understand you is by asking your question louder. It might feel better, but it’s not going to help.

On the other hand, approaches that enable existing data to become more useful, together with a greater willingness to facilitate access to traditionally siloed data, and, most importantly, evidence that there’s really valuable new insight to be harvested from these data could profoundly change the game for pharmas. And if pharmas can’t figure out how to do this, it may only be a matter of time before someone else does.

