From Sight to Insight: Visualization tools yield biomedical success stories

They're more than just pretty pictures adorning office walls and presentation slides. Beamed into operating room computer monitors, they're guiding the scalpels of brain surgeons. Dancing across laboratory workstations as animations of twisting, folding proteins, they're helping scientists design new drugs. Streaming through three-dimensional virtual reality rooms, they're letting researchers probe metabolic networks and biochemical pathways by literally walking among them.

Visualization tools are reshaping clinical practice and driving biomedical discovery.

The computer-based technology comes in two flavors, depending on the kind of data being displayed. The first, known as scientific visualization, starts with spatial data from lab experiments or clinical procedures--for example, molecular structures or x-rays--and renders it in a visual display that's easier for researchers to analyze. The second, dubbed information visualization, takes abstract, non-spatial data such as protein interaction networks or genomic sequences and organizes it into a visual form that allows investigators to more easily conceptualize the diverse information.

Such capabilities were mere fantasy two decades ago, when scientists would shell out $200,000 for a VAX computer whose drawing speed hovered at a sluggish 2,000 triangles per second. But thanks to remarkable computer graphics advances driven largely by the computer gaming industry, "that's all changed," says Terry Yoo, PhD, a program director who oversees visualization research at the National Institutes of Health. "The capacities of a Silicon Graphics system that I would have bought eight years ago for $60,000 is now in my son's PC at home, and I bought it at CompUSA for $300." Today, many home computers can draw millions of triangles per second.

Riding the wave of computing progress, visualization researchers about 30 years ago started getting data from scientists and clinicians and turning it into nice pictures. For example, when computed tomography (CT) emerged as a medical imaging tool in the 1970s and volumetric image acquisition became possible by the 1990s, graphics engineers designed methods to convert those heaps of mathematically reconstructed slices into more naturallooking 3D images of the anatomy.

Such graphics wowed colleagues and journal reviewers but, for many years, didn't actually help researchers and doctors do their work, says Chris Johnson, PhD, professor of computer science and director of the Scientific Computing and Imaging (SCI) Institute at the University of Utah in Salt Lake City. "I think only recently have we been able to create visualizations that biomedical scientists are starting to find effective and useful." Johnson credits improved algorithms and faster hardware that enable graphics to more readily handle the complexity of biological systems.

The March Toward Scientific Visualization

BRAIN MAPPING White matter connections can be visualized by diffusion tensor (DT)- MRI 3D tractography. Courtesy of Martha Shenton, Harvard University. For more information, visit the Laboratory of Mathematics in Imaging Web site A key step toward successful biomedical visualizations came in 1987 when Bill Lorensen, PhD, and Harvey Cline, PhD, of General Electric introduced the Marching Cubes algorithm, which treats the space between adjacent data points as small cubes. Applied to a set of brain scans, for example, the algorithm "marches" through each cube, deciding whether its corners lie inside or outside the brain structure to be visualized. In cubes that are intersected by the structure's boundary, tiny adjacent triangles are used to approximate the surface. Using standard video-game display techniques, a computer assembles this triangle mesh into pictures of three-dimensional anatomy.

Other reconstruction methods, known as volume rendering, avoid the step of building geometric meshes. Instead, these approaches make 3D visualizations directly from the image data either by assigning opacity values to each and stacking slices on top of one another, or by tracing the 3D path of light rays through the image data.

Although radiologists may prefer to scrutinize the original 2D slices, some feel they glean additional information from 3D reconstructions of CT and magnetic resonance imaging (MRI) data. They want the value that is added by visualization technology.

Other scientific visualization efforts bypass the problem of extracting structures from images. For instance, 3D protein structures may come from a variety of sources such as X-ray crystallography, Nuclear Magnetic Resonance (NMR) spectroscopy or even ab initio simulation.

In all of these cases, much of the challenge of scientific visualization is in providing added value to the enduser by choosing the best visual representation and optimizing the user's interaction with the model.

When Visualization Makes a Difference: Showing Surgeons Where to Cut

ASSISTING SURGEONS Different MRI techniques displayed in the same brain guide surgeons by revealing tumor (green – conventional MRI), motor cortex (golden—functional MRI) and white matter connections extending from motor cortex (ivory – DT MRI). Courtesy of Carl-Fredrik Westin, Harvard Medical School.

After years of crafting dazzling displays for clinicians and scientists, Johnson knew that scientific visualization was making a real contribution when those pictures became a valuable tool for surgery. In 2001, interactive visualization techniques developed by his SCI Institute team helped a neurosurgeon plan a challenging tumor operation on a young girl at Primary Children's Medical Center in Salt Lake City.

Donning stereo glasses, the surgeon saw the girl's MRI brain scans rendered in 3D on the Institute's eight by ten foot wide screen. Using immersive visualization techniques that better reveal the spatial orientation of vital anatomical structures, "he could stick his head up in her brain and look around," Johnson says.

Though the technology provided valuable perspective on the 3D relationships between key brain areas, certain important subtleties remained invisible.

But the power of 3D visualization of MRI images would soon reach new heights. In the 1990s, scientists began trying to create 3D maps of key nerve pathways--features that were indiscernible by conventional MRI--by tracking the random motion of water molecules within the brain. Exquisite tissue changes measured by this technique, called diffusion tensor (DT)MRI, could potentially serve as red flags for certain degenerative diseases.

But scientists have faced tough obstacles en route to these grand goals. In conventional MRI, each cube, or voxel (3D pixel), is assigned a gray value. In DT-MRI, however, the diffusion data for each voxel is too complex to convey with a single number, explains Carl-Fredrik Westin, PhD, who directs the Laboratory of Mathematics in Imaging (LMI) at Brigham and Women's Hospital and is an associate professor of radiology at Harvard Medical School in Boston.

To get at how diffusion translates into tissue structure, consider the spherical pattern produced by the unrestricted motion of an ink drop in a jar of water. In similar fashion, water molecules in cerebral spinal fluid travel freely in all directions. But within highly organized tissues such as heart muscle or white matter tracts in the brain, water's motion becomes highly constrained to the direction of the fibers. Scientists can represent these diffusion properties as a 3x3 matrix of numbers, but this delineation is neither visual nor intuitive. "The challenge," Westin says, "is how you show this to the doctor." By devising algorithms that cast these matrices into 3D ellipses, he and others have succeeded in mapping nerve pathways that connect key brain areas. The process is known as DT-MRI tractography.

Currently, once or twice each week, tractography maps are streamed into operating rooms to guide brain surgeries at Brigham and Women's Hospital.

The ability to see critical white matter tracts fused with real-time conventional MRI data showing a brain tumor's location allows neurosurgeons to more confidently extract tumors while leaving critical nerve connections intact.

Three and a half years ago, DTMRI scans showed that a 23-year-old woman's tumor had crept into areas of the brain responsible for motor function. "This information was essential in defining the target for tumor resection and sparing the infiltrated    motor fibers," says IonFlorin Talos, MD, a Harvard radiology instructor and member of the team that performed the first few tractography-guided brain operations. "The patient made a full recovery post-surgery."

DT-MRI mapping has also revealed that certain neural networks are disrupted in schizophrenics. Such insights are possible today because multiple types of data--from conventional MRI, functional MRI and DT-MRI--can be visualized in the same person at once. "This gives us a much better window on the brain than any one imaging tool by itself," says Martha Shenton, PhD, director of the Psychiatry Neuroimaging Laboratory at Brigham and Women's Hospital and professor of psychiatry and radiology at Harvard Medical School.

Seeing Tumors Recruit Blood Vessels: New Frontiers in CT Imaging

While MRI works well for visualizing soft tissues, CT brings blood vessels into sharper view--a boon for Charles Keller, MD, and others who study how tumors recruit complex vascular net-works. Keller, a pediatric oncologist at the University of Texas Health Science Center in San Antonio, and colleagues have engineered a mouse model for alveolar rhabdomyosarcoma, a human skeletal muscle cancer. Unlike xenograft systems, in which human-derived tumor cells are transplanted underneath the skin of a mouse, Keller's mice sprout tumors from muscle, intertwined with bone--a more natural rendition of the disease process.

But bone poses problems for visualization. Most methods for making volume-rendered 3D reconstructions from CT slice data use 1D transfer functions, which assign pseudocoloring based on the measured CT data value (density) for each voxel. In visual displays built from these values, researchers can often recognize different biological structures. For instance, a fat-cell tumor stands out because it's less dense than the surrounding soft tissue. Bones, however, can not always be distinguished from contrast-enhanced blood vessels based on density alone. So Keller teamed Bones and Vessels compared to commonly used 1D transfer functions (top), 2D transfer functions (bottom) generate CT reconstructions that more clearly distinguish blood vessels from bone in the mouse. Courtesy of David Weinstein and Gordon Kindlmann, Scientific Computing and Imaging Institute, University of Utah; Charles Keller, Children's Cancer Research Institute, University of Texas at San Antonio.up with visualization experts at the University of Utah's SCI Institute to craft a better strategy for clarifying bone-vessel distinctions without resorting to difficult and non-robust image segmentation algorithms.

Instead of using 1D transfer functions, which color the anatomy solely according to material density, the researchers created a 2D transfer function that incorporates a second property--degree of "edge sharpness." Because bones have more distinct boundaries than blood vessels, the new algorithm can visually separate bones from blood vessels with greater accuracy.

The fine specificity comes with a price. Compared with 1D transfer functions, multi-dimensional transfer functions are harder to design and consume more computing power, says Johnson, whose SCI Institute team developed the BioImage visualization software used in Keller's research. Nevertheless, BioImage runs efficiently in real-time on workstations costing less than $5,000.

The technology has sparked new insight into how muscle tumors spread and thrive. "Now that we have these new contrast agents and powerful software tools in hand," Keller says, "we're getting an appreciation for how aggressive these tumors are in luring new blood vessels to their leading edge while simultaneously leaving a trail of dying tissue in their wake."


Drug Design By Visualizing Life's Smallest Units

Visualization tools for molecular models are helping researchers design new drugs by allowing them to "see" the tiniest drivers of biological activity--molecules and atoms. To understand a molecule's function, researchers need to probe its inner workings by studying how its atoms are arranged as well as how they interact with each other and with their surroundings.

For small, simpler molecules like water, ball-and-stick models can offer a basic view of how the molecule looks and behaves. But for huge biological structures--nucleic acids, viruses, and protein complexes, for instance--Tinker Toys won't cut it. That's where molecular modeling software such as Chimera ( at the University of California, San Francisco (UCSF)--comes to the rescue.


Such tools allow researchers to see 3D renditions of molecules on the computer screen. With a few clicks of the mouse or turns of a joystick, they can zoom in for close-up views of specific binding regions. Or flip the molecule to expose atoms that form bonds with neighboring proteins.

These capabilities are boosting drug design research. To design a small molecule that cramps the activity of a specific protein, scientists are turning to DOCK, UCSF's molecular matchmaking algorithm that scans databases for compounds that could bind with that protein. Scored according to size, shape, affinity for water and other properties, promising compounds can be viewed interactively using the Chimera extension, ViewDock.

DRUG DESIGN Chimera's interactive 3D displays of ligand-target binding help researchers design new drugs. Courtesy of Greg Cook, North Dakota State University in Fargo. For more information, visit the UCSF Chimera Web page: perfect matches rarely crop up, the 3D visualizations can reveal how a bit of chemical fine-tuning--for example, replacing a hydrogen atom with a nitrogen--could make a molecule a much better fit. "You have to understand the whole machine so that you can tweak it to get it to do what you're trying to accomplish," says Thomas Ferrin, PhD, professor of pharmaceutical chemistry at UCSF and director of the UCSF Computer Graphics Lab, which created Chimera five years ago. About 34,000 copies of Chimera have been distributed freely to government and academic institutions, and to companies for a licensing fee. Gregory Cook, PhD, an organic chemist at North Dakota State University in Fargo, has used Chimera to design molecules that block a tumor growth-promoting enzyme called matrix metalloproteinase (MMP)-3. "Just being able to see the shape and being able to zoom in and see the protein structure," Cook says, "you get a better feel for what drug shapes you want to try."

Although such technology can augment human ingenuity, it will never replace it, says Ferrin. "Computers are not creative. It's humans who are creative. But computers are very good at repetitive calculations." Chimera's goal, he says, is to "take problems and divide them into spaces where computers can do what they're good at and humans can do what they're good at."

Animating Molecules

MOLECULAR MOVIES Yale’s Database of Macromolecular Movements calculates  a plausible trajectory between two  structures of calmodulin—one where  the alpha helix is straight (image below) and another where the helix is bent at its hinge residue (image above). The program includes an  algorithm, FlexOracle, that predicts hinge locations. Courtesy of Samuel Flores, Yale University.Designing a molecule to fit a protein's binding groove can seem a bit like scrounging the universe for a missing jigsaw puzzle piece. Now consider doing this for puzzle pieces in constant motion. That's a more realistic picture of the challenge biologists face. Understanding a protein's function means knowing how it bends, twists and reshapes itself in different situations. To study protein dynamics, researchers have turned to publicly available software such as the Database of Macromolecular Movements (, developed by Mark Gerstein, PhD, associate professor of biomedical informatics at Yale University in New Haven, Connecticut and colleagues. The database's Morph Server takes user-submitted protein structure files and interpolates between them to produce plausible trajectories leading from one structure to another. More than 17,000 movements have been compiled so far.

In summer 2005, physics PhD student Samuel Flores and other Gerstein group members added a new tool to the system: FlexOracle. This program predicts the location of hinges in proteins with moving domains. To create an animation of the hinged protein's motion, the program applies forces to one domain while holding the rest fixed in place. By calculating the force needed to move the domain in each direction, Flores explains, the algorithm derives a "path of least resistance" that approximates the protein's natural movement. In an early demonstration of FlexOracle's value as a discovery tool, the program correctly predicted a known hinge in calmodulin, a calcium-binding protein involved in many key biochemical events within the cell. What's remarkable, Flores says, was FlexOracle's success in placing this hinge in the middle of an alpha helix, a type of protein secondary structure that rarely contains hinges.

The Morph Server brings these predictions to life by allowing investigators to make movies that show how the proteins twist and fold around the presumed hinge. While viewing the animations on a computer, users can enlarge the molecule to focus on specific atoms and rotate the structure three-dimensionally to see movements from different vantage points.

Visualizing "Shapeless" Information

Anatomic and molecular visualizations benefit from the fact that brains and molecules are physical objects easily portrayed in pictures. "Everything comes in nice, neat Cartesian coordinates," says Yoo of the NIH.

Harder to visualize are genome sequences, biochemical pathways and other biological information lacking inherent shape or dimension. Stockpiles of such data are growing faster than ever, fueled by an explosion of high-throughput sequencing and gene analysis techniques.

Biologists trying to see the forest through the trees face a formidable challenge. "Just picture an Excel spreadsheet with 45,000 lines and 20 columns," says Karen Duca, PhD, a research assistant professor at Virginia Bioinformatics Institute, at Virginia Polytechnic Institute in Blacksburg. "There's no possible way a human can mine those data points for any meaningful patterns without some sort of tool."

From Spreadsheets to Pathways

With new information visualization software such as Cytoscape, researchers are taking a stab. Cytoscape ( is an open-source bioinformatics platform that puts genome data into context with the real biological workhorses-- proteins. Let's say you use a microarray, or DNA chip, to get a snapshot of all the genes that get turned on or off in liver cells treated with a particular drug. What you really want to know, says Trey Ideker, PhD, one of Cytoscape's developers, is whether there are "particular biochemical pathways or protein complexes in the cell that connect the genes that are upor down-regulated together."

Cytoscape integrates gene expression data with protein-protein and proteinDNA information from interactions databases. The software groups genes of interest in biologically relevant clusters and allows users to assemble pathways. This image shows a yeast network of protein-protein interactions (blue dashed edges) and transcription factor-DNA regulatory interactions (purple arrows). These are the significantly differentially expressed genes of a wild-type strain both with and without a mutagenic growth condition (red: up-regulated, green: down-regulated). ”Node sizes are scaled by significance of differential expression.” Courtesy of Trey Ideker, UC San Diego.Scores of protein-protein and protein-DNA interactions already abound in databases. But sifting through this information is tedious. "It's about as meaningful as a bunch of As, Ts, Cs and Gs on a page when looking at gene sequences," says Ideker, assistant professor of bioengineering at the University of California, San Diego.

By querying data banks of molecular interactions, Cytoscape can take that huge spreadsheet of microarray data and display clusters of liver genes whose activity changed with drug treatment. For example, genes involved in respiration would appear as a network of colored nodes linked by lines reflecting protein-protein interactions or arrows representing protein-DNA binding.

Cytoscape users can "personalize" these networks by adding or deleting nodes, and by changing node colors or line thickness to reflect certain properties relevant to their own research. Earlier versions of the software, though, often required users to write computer code to leverage these capabilities. "I must confess that 'user-friendliness' is currently not our strongest suit," Ideker says.

It's a concern shared by others developing information visualization technology. "Bench biologists have enough to do running their experiments," says Duca of Virginia Tech, who has done usability tests on bioinformatics tools. "Learning elaborate software just adds an extra burden. The best software is intuitive and easy to follow."

Cytoscape's newest release, scheduled for November 2005, allows users to compare networks by displaying them as side-by-side panels. Computer engineer Iliana Avila-Campillo, at the Institute for Systems Biology in Seattle, hopes improvements in the system's graphical user interface will make it "as easy as having a Microsoft wizard that takes you step-by-step to create your own network."

Networks: Getting an Inside View

But a daunting problem remains: No matter how easily such models can be built, sheer biological complexity still makes them horribly convoluted.

"The graphs look like spaghetti," says Julie Dickerson, PhD, assistant professor of electrical and computer engineering at Iowa State University, upon seeing a 2D display of metabolic and regulatory networks in Arabidopsis, a model organism for plant biology research. "They're overloaded. There's so much information you're trying to convey in a single diagram."

A researcher navigates Arabidopsis biochemical pathways and metabolic networks using MetNet3D, a new virtual reality-based information visualization tool. Courtesy of Julie Dickerson, Iowa State UniversityThat's why she and systems biologist Eve Wurtele, PhD, a professor in Iowa State's department of genetics, development and cell biology, have turned to virtual reality as a way to improve those visual displays. They hope their interactive 3D system, MetNet3D, will help users navigate the bewildering webs of genomic, biochemical and metabolic data without oversimplifying the information.

In a recent test run, Wurtele used MetNet3D to explore clusters of Arabidopsis genes that are active during the day but not at night. Sporting 3D stereo glasses, she and colleagues walked among swarms of biochemical pathways and metabolic networks streaming around them in a 10-cubic-foot, virtual-reality room. During their exploration, they punched keys on a handheld PC to select specific genes and get more detailed information about them. Able to display more than 500 nodes as spheres, cubes, cylinders and cones, MetNet3D can organize the elements into pathways or by sub-cellular compartments, and users can modify their colors and edges. "I can do it all easily, just clicking buttons and toggles," Wurtele says. MetNet3D isn't yet ready for prime time, but developers have already ironed they punched keys on a handheld PC to select specific genes and get more detailed information about them.


MetNet3D isn't yet ready for prime time, but developers have already ironed out a slew of technical kinks, and usability tests with the exploratory software were scheduled to begin in fall 2005. Because it uses three open-source application program interfaces, MetNet3D can run on a variety of virtual reality and visualization platforms, including conventional desktop computers. Several American and European research groups have downloaded the software and started to use it in their labs.

On the Horizon

As bioinformatics data becomes denser, future information  visualization tools might use vibrations and motion to help researchers zero in on  particular nodes and connections. Courtesy of Colin Ware, University of New Hampshire. Moving beyond colors, shapes and lines, some in the visualization field are toying with newer methods to portray rich bioinformatics networks. "As the information gets denser, you have to use more cues to help people home in on what's important," Duca says.

Colin Ware, PhD, a computer scientist who builds visualization systems at  the University of New Hampshire in Durham, has developed an interactive graph visualizer that uses motion to highlight key connections within large networks. When a user clicks on a node, neighboring nodes and the lines that connect them start vibrating. "Motion works over a larger visual field  than color," Ware says. "If you have a small point of color, you don't see it in the periphery. But if it's moving, you do." Ware's system hasn't yet entered the biomedical realm, but some think it could be useful for visualizing bioinformatics data.

LigandScout, a software platform for virtual screening of small molecule pharmaceuticals, borrows a concept from photography to create "semantic depth of field:" it steers the user's attention toward the most relevant elements by blurring less germane parts of the image.

UCSF's molecular modeling program, Chimera, might someday include a platform that incorporates sound into its visual displays. The toot of a trombone, for instance, could suggest van der Waals forces within a molecule while a tuba represents areas of repulsion, with high or low pitch linked to the magnitude of force or repulsion.

A Promising Outlook

If these sorts of tools offer a taste of what's to come, experts forecast a bright future for visualization technology. High resolution imaging; high-throughput biological assays; databases and knowledge-bases of unprecedented scale; and systems biology approaches are all driving the need for scientific and information visualization in 21st-century biomedicine.

Much has happened already. The pretty pictures have come off office walls and entered operating rooms. Reams of bioinformatics data are appearing on computer screens as organized, interactive networks.

"I think we're going to see a golden age of computers and visualization in biomedicine," says Johnson of the University of Utah. "And what we've seen is just the tip of the iceberg. It's just going to explode in the next 10 years."


Post new comment

Enter the characters shown in the image.