The Science of a DREAM Challenge
Opening up biomedical discovery.
DREAM Challenges are crowdsourcing challenges designed to evaluate model predictions and pathway inference algorithms in systems biology and medicine. The DREAM challenges aim to address fundamental questions in bridging theory and experimentation.
The questions posed by DREAM Challenges are not simple; to deduce the structure of a biological network based on experimental data, for example, researchers test a variety of algorithms. DREAM Challenges allow researchers to compare the strengths and weaknesses of these methods and provides a sense of reliability. It is interesting to note that the aggregated community predictions are extremely robust and usually more accurate than even the best performing algorithms. This effect is often referred to as the “wisdom of crowds”.
Collaborative discovery to accelerate science.
DREAM Challenges produce results by crowd sourcing its wisdom from a crowd of “experts”. We have found that DREAM Challenges promote:
(1) Data democratization through our large network of researchers
(2) Acceleration of research
(3) Rigor of assessment by evaluating blind predictions
(4) Creation of a community that jointly works on the same data
(5) Further collaboration among the researchers while writing the overall paper that results from the challenge, ie publishing the results.
Challenges for Scientists by Scientists.
A community of scientists develop and run the DREAM Challenges. It is a scientific research competition developed by scientists for scientists. So compared to others doing crowdsourcing challenges the involvement of our researchers goes far beyond the “administration” of a challenge. It includes:
(1) developing baseline models to insure the existence of signal in the data and hence the viability of the challenge
(2) normalization and curation of data
(3) development and understanding of scoring and metric statistics
(4) assistance with writing of manuscript from the challenge results for publication
(5) scientific (not just logistical) advisement through all stages of the challenge.
Open data. Open discovery.
In 2013 we began a collaboration with Sage Bionetworks, and DREAM Challenges utilizes their Synapse platform. Synapse allows reproducibility of the research process by providing a collaborative environment for users to share data, code, results, and the analysis provenance linking these research artifacts to one another. Users can interact with these services through the Synapse web portal, or through one of our supported analytical clients (R, python, and command line). Synapse offers DREAM Challenge organizers and participants features such as real-time leaderboards, provenance tracking and community forums incentivize continuous participation and enable teams to build upon one another’s work to evolve more accurate predictive models. Through Synapse the hope is to allow broader participation of the research community in the DREAM Challenge’s open science and provide a meaningful impact to both discovery and clinical research.
Through this active scientific involvement superior results have resulted. We have over 60 publications in top journals. It is a commitment to great science that we welcome you to be a part of.
For more information on the Science of DREAM Challenges, please enjoy the following presentation given by Gustavo Stolovitzky at the NIPS 2014 Workshop – Challenges in Machine Learning workshop (CiML 2014) .
Please contact us to learn more details on how we set up and run a challenge.