Projects in Scientific Computing: At the Frontier of Physics and Chemistry

  Magnetic Moments
Teraflop Software
Models Complex
Magnetic States

Where would we be without magnets? Refrigerators would be boring, and that's just the surface of the icebox. Bulk magnets similar to those that transform refrigerators into photo displays play a little-noticed role in modern life, but they're essential in power generation, electrical motors, the sensors in cars, and many home appliances.

Other magnetic materials, thin films and composites, are ubiquitous in consumer electronics. The growth of home entertainment, from cassettes to videos, has been driven by progress in magnetic-recording technology, and the information age we're living in would still be a futurist dream (and programmers would still be shuffling punch cards) were it not for amazing magnetic devices like the read-heads on disk drives.

Despite these advances in technology, we know less than you might think about how magnetism really works, in other words, what's going on at the level of atoms and electrons. "Even refrigerator magnets are very complicated," says Malcolm Stocks. "There are big pieces of the underlying physics we don't understand." The Department of Energy has designated this underlying physics as part of a four part "grand challenge" in materials, methods, microstructure and magnetism, and with DOE support, Stocks, a physicist at Oak Ridge National Laboratory, leads a team of scientists working to unlock magnetism's puzzles.

As a member of this team since 1993, Yang Wang of the Pittsburgh Supercomputing Center has helped develop software that simulates the interactions between electrons and atoms in magnetic materials. "This is key to understanding magnetism at the theoretical level," says Wang. Using this software — the locally self-consistent multiple scattering (LSMS) method — Stocks' team has proposed new understanding of complex, disordered states of magnetism. In 1998, furthermore, LSMS achieved the distinction of being the first research software to break the teraflop barrier in supercomputing.

Breaking the Teraflop Barrier

On Nov. 9, 1998, running on a 1,480 processor CRAY T3E system at SGI/Cray, LSMS sustained performance of 1.02 teraflops (trillions of calculations per second). Although special-purpose computers running benchmark software had run this fast before, this marked the first time an actual research program ran at teraflop speed. For an earlier 657 gigaflop run with the same software, Stocks' team — which includes researchers from Oak Ridge, the National Energy Research Scientific Computing Center (NERSC), the University of Bristol (UK) and PSC — won the 1998 Gordon Bell Prize for best achievement in high-performance computing.

Wang's contribution played a crucial role. "He took the code to Pittsburgh and made it run on the T3E," says Stocks. Beginning in 1993, when Wang was a post-doctoral fellow at Oak Ridge, he and Stocks worked to develop LSMS on Oak Ridge's Intel Paragon system. Wang came to PSC in 1996, and in this position collaborated with Oak Ridge scientist Bill Shelton to translate LSMS to run on PSC's CRAY T3D, predecessor to the T3E.

LSMS was designed from the start to exploit parallel systems like the T3D and T3E, where hundreds or thousands of processors can team up on one large computing task. Much of the trick in using these systems well is finding a way to divide the work into more or less equal shares. LSMS assigns each atom in a "unit cell" — the irreducible unit of atomic structure for solid materials — to a separate processor.

A powerful innovation of LSMS's software design is its "scaling," which is linear: as the size of a problem enlarges, i.e., the number of atoms in a unit cell increases, computing time increases by only the same multiple. This is a major accomplishment. With conventional methods for solving the governing quantum equations, the amount of computational work increases like the cube of the number of atoms (N3), severely limiting the problems that can be addressed.

LSMS can simulate unit cells ranging from hundreds to thousands of atoms, well beyond most existing methods, and close to the range of realistic magnetic materials. The Nov. 9 calculation simulated a 1,458 unit cell of iron. "This is not much smaller than nano-particles being made today," says Stocks.

The raw computing power of the T3E was key to the Nov. 9 breakthrough. Each individual T3E processor is a very fast Alpha chip, a 600 megahertz chip from Digital Equipment Corp. Combining 1,458 of these processors with the high efficiency of LSMS showed decisively that teraflop performance, described only a few years ago as a goal of the future, is now supercomputing reality.

Investigating Noncollinear Magnetism

Along with setting speed records, LSMS is enabling Stock's team to launch new investigations into the theory of magnetism. Developing a microscopic understanding of the dynamics of metallic magnets has been an abiding scientific challenge, and the current challenge, explains Stocks, is to track what happens to the "magnetic moments" of atoms under complex conditions.

Aligned moments create a magnetic field (left). Parallel non-aligned moments (center) are "antiferromagnetic" — i.e., there's no magnetic field. Random moments (right) exemplify noncollinear magnetism.

Up till now, to a large degree, computational studies have looked at how magnetism works only under limited conditions, so-called ground states, at absolute zero temperature. Under these conditions, the magnetic moments of the atoms — which occur due to unpaired electrons in the outermost orbits — are aligned, all pointing in the same direction or at least parallel to each other.

In real-world conditions, obviously, magnets aren't maintained at absolute zero, and when temperatures rise or under other conditions, such as the influence of an external magnetic field, the magnetic moments change direction. At high enough temperature, the moments become randomly disordered, canceling each other and eliminating the magnetic field. The objective of LSMS is to track such disordered magnetic states, called noncollinear magnetism. "Now is the time to do real-world problems," says Wang, "where the magnetic moments aren't all up or down, which is what LSMS and parallel systems allow us to do."

LSMS relies on density-functional theory (for which physicist Walter Kohn won the 1998 Nobel Prize in chemistry) to describe the interactions between electrons and atoms in magnetic materials. DFT alone, however, can only solve aligned states of magnetism. To simulate noncollinear states, Stocks' team developed a complex variation of the basic theory, called "constrained DFT," that applies a separate magnetic field to each atom in a unit cell.

In a recent calculation, the group applied its constrained DFT model to a 512-atom unit-cell of iron. The method keeps track of how a change in the magnetic moment of one atom affects other atoms in the vicinity, and solves Schrodinger's equation, the fundamental equation of quantum theory, for each atom of the unit cell. The results of this calculation, using a 512-processor CRAY T3E at NERSC, demonstrate the reliability of LSMS and validate it as powerful new tool for studying the finite temperature properties of metallic magnets.

Results from computing the magnetic moments for a 512-atom unit-cell of iron above its Curie temperature, when the magnetic field begins to break down. As the inset (right) shows, the magnetic moment at each atom (arrowhead vectors) has a corresponding constraining field (translucent cones) as a result of the constrained DFT model. Color indicates vector magnitude.

Download larger version of main image(928K) or inset(189K)

In future calculations, Stocks, Wang and their colleagues plan to address a number of problems in magnetic multi-layers and magnetic nano-particles. Their work is basic research, aimed not at technology per se but at new scientific understanding. At the same time, however, the potential to help bring about significant new technologies is real. Even small improvements in the information density of magnetic-storage devices, for instance, could radically transform computing. "If you could even slightly increase the maximum energy product of a permanent magnet," says Stocks, "it would have enormous effect on technology."

Researchers Malcolm Stocks, Oak Ridge National Laboratories
Yang Wang, Pittsburgh Supercomputing Center
Hardware CRAY T3E
Software LSMS
Related Material
on the Web
Locally Self-Consistent Multiple Scattering (LSMS) Method .
Pittsburgh Scientist Helps Crack Teraflop Barrier
References G. M. Stocks, B. Ujfalussy, Xindong Wang, D. M. C. Nicholson, W. A. Shelton, Yang Wang, A. Canning & B. L. Gyorffy, "Towards a Constrained Local Moment Model for First Principles Spin Dynamics," Philosophical Magazine B 78 (5 & 6), 665-73 (1998).

B. Ujfalussy, Xindong Wang, Xiaoguang Zhang, D. M. C. Nicholson, W. A. Shelton, G. M. Stocks, A. Canning, Yang Wang, B. L. Gyorffy, "High Performance First Principles Method for Complex Magnetic Properties," Proceedings of the ACM/IEEE Supercomputing 98 Conference, Orlando Fla., Nov. 7-13, 1998, IEEE Computer Society, Los Alamitos, CA 90720-1265 (CD-ROM).
Writing: Michael Schneider
HTML Layout/Coding: R. Sean Fulton
