Computing the Cosmos 

One of the biggest computer simulations ever run is illuminating the deepest mysteries of the universe
By Alexander Hellemans & Madhusree Mukerjee

IF COSMOLOGISTS WERE TO MAKE A MOVIE of the universe's entire history, the show
would start, of course, with the scorching blast of the Big Bang. The
universe-absolutely every bit of mass we can detect or even infer today-would
expand at unfathomable speeds, going from smaller than a proton to larger than
a galaxy in the blink of an eye. As the expansion continued, the universe would
cool down, and by the time the opening credits of the movie finished scrolling,
a superhot soup of elementary particles would fill the whole cosmos, ready to
cook the first protons and neutrons.But what would happen next?  The fact is,
cosmologists are still working out the rest of the plot-what exactly took place
during the more than 13 billion years since that primeval blast. [For this
article, in keeping with the current trend in international scientific
publishing, IEEE Spectrum uses the words "billion" to mean 1e9 and "trillion"
to mean 1e12.] A particular piece of the story that has kept researchers
scratching their heads is how galaxies formed and evolved. How did that
amorphous particle soup transform itself into billions and billions of galaxies
of breathtakingly different shapes and sizes? Why did these galaxies gather
together in clusters, and clusters of clusters, embedded along unimaginably
enormous structures of matter shaped like bubbles, filaments, and sheets?  To
answer these and other fundamental questions in cosmology, an international
group of scientists from Canada, Germany, the United Kingdom, and the United
States has been working on an ambitious project whose goal is to simulate on a
supercomputer the evolution of the entire universe, from just after the Big
Bang until the present.  The group, dubbed the Virgo Consortium-a name borrowed
from the galaxy cluster closest to our own-is creating the largest and most
detailed computer model of the universe ever made. While other groups have
simulated chunks of the cosmos, the Virgo simulation is going for the whole
thing. The cosmologists' best theories about the universe's matter distribution
and galaxy formation will become equations, numbers, variables, and other
parameters in simulations running on one of Germany's most powerful
supercomputers, an IBM Unix cluster at the Max Planck Society's Computing
Center in Garching, near Munich.  4 200 000 000 000: NUMBER OF CALCULATIONS PER
SECOND THE VIRGO CONSORTIUM SUPERCOMPUTER CAN PERFORM  Late this year, the
group plans to begin storing all of its output data in public repositories
available to researchers around the world [see article (1) "Downloading the Sky
<0804vobs.html> " above]. This accessibility, according to Simon D.M. White, a
research director at the Max Planck Institute for Astrophysics who leads the
German participation in the Virgo Consortium, will allow researchers to compare
each simulated universe to its ultimate benchmark: the universe itself, as
observed with ground and space telescopes.  If the simulation produces a
bizarre universe that doesn't resemble ours, the assumptions that underpin the
simulation are probably flawed or in need of adjustment. On the other hand, if
the virtual cosmos is like the one we see, researchers will know they are on
the right track. In this way, they hope to see some of the deepest mysteries of
the cosmos solved on their computer screens. 

BY PEERING INTO GREAT DISTANCES with powerful telescopes, astronomers have
discovered a startling pattern of, literally, cosmic proportions: galaxies
gather into clusters of varying sizes that don't float in isolation in the
universe but rather are linked to one another by long tendrils of matter.
What's more, these agglomerations of clusters are collected onto incredibly
huge bubbles, filaments, and sheetlike structures, millions of light-years in
size.  These structures, the grandest that are known, form a three-dimensional
cosmic web that fills the universe. If you could shrink the cosmos until a
galaxy cluster were the size of a grain of sand, a chunk plucked out of this
universal web would resemble a piece of a kitchen sponge; the air pockets in
the sponge would represent the huge cosmic voids that contain almost no
matter.  To find out how and when this giant clumpy web formed, the Virgo group
is simulating how matter dispersed in space over almost the entire course of
the universe's existence. 

TO BEGIN CREATING THEIR MODEL of the universe, the researchers faced two basic
questions: at what moment, precisely, should they start the simulation? And
what are the universe's conditions at that very moment? Fortunately,
cosmologists believe they have these answers.  According to the inflationary
universe theory put forward in the 1980s by Alan Guth, of the Massachusetts
Institute of Technology, and Andrei Linde, of Stanford University, the universe
swelled at an extraordinarily rapid rate during a tiny fraction of a second
immediately after the Big Bang. This exponentially fast expansion amplified
minute, quantum-scale fluctuations that existed in the primordial energy field
that filled the very early universe. These fluctuations caused matter to clump,
and later, gravity created denser and denser aggregates.  The result is that
these aggregates, which began as unimaginably small energy fluctuations in that
primeval universe much smaller than a proton, ultimately evolved into the giant
structures that compose the universe's spongelike web of matter. Even more
surprising, most of the mass in this web of matter is not the ordinary stuff we
know that makes up galaxies, stars, planets, and people.  After many
experiments and calculations throughout the past decade, most cosmologists now
agree on the astounding fact that some 85 percent of the matter in the universe
consists of a mysterious substance known as dark matter that cannot be seen
directly. They infer its presence by tracking the motions of stars and
galaxies: stars are attracted to the centers of galaxies, and galaxies to the
centers of galaxy clusters, by gravitational forces that are far greater than
visible matter alone can possibly account for. Something else must be out
there.  This shadowy substance is made up not of the familiar quarks,
electrons, and their derivatives-atoms and molecules-but of some particle that
has so far eluded experimenters. Candidates include axions, photinos,
neutralinos-all yet to be discovered-among other particles predicted by
theorists. The upshot is that, because dark matter is not visible, what
astronomers have observed are the contours of the universe's great web,
revealed by the light of the stars and galaxy clusters that formed onto the
web's nodes, the junctions at which large amounts of matter accumulate. It's
kind of like inferring the shape of a Christmas tree in a pitch-dark room from
the positions of the lights strung on it. These stars, galaxies, and other
objects that we can see were born from dense aggregates of normal matter
embedded in the dark matter of the web.  2 000 000 000 000 000 000 000 000 000
000 000 000 000: MASS IN KILOGRAMS OF EACH OF THE 10 BILLION SIMULATION POINTS
IN THE VIRGO CONSORTIUM'S MODEL OF THE UNIVERSE  One of the major cosmological
features that the Virgo team relies on to formulate its simulations is
something called the cosmic microwave background, a feeble radiation remnant of
the Big Bang that astronomers have now studied in great detail. This radiation,
a key piece of evidence supporting the inflation theory, was emitted 380 000
years after the Big Bang when protons combined with free electrons to form
neutral hydrogen atoms.  If the cosmic particle soup were absolutely smooth,
with evenly distributed hydrogen atoms, this radiation would also be smooth all
over the place-always the same wherever you look. But as cosmologists pointed
their detectors to different parts of the sky, they found small variations in
the cosmic microwave background. These variations were recently minutely
detailed by NASA's Wilkinson Microwave Anisotropy Probe, whose first scientific
results were made public early last year. In a triumph of modern cosmology, the
measured variations correspond precisely with the predictions of inflation
theory.  The cosmic microwave background, therefore, gives cosmologists a
fairly good picture of the distribution of matter when the universe, with an
estimated current age of 13.7 billion years, was still in its infancy, only 380
000 years old. That's the starting point the Virgo group has chosen for its
simulations. The main one, the first of a series, dubbed the Millennium Run,
was completed this past June. When data is fully processed within the next few
months, the Millennium Run will reveal with unprecedented detail how the
cosmos's broad distribution of matter came to be. "It will be the mother of all
simulations," says Carlos S. Frenk, a cosmology professor at the University of
Durham, United Kingdom, who leads the British participation in the Virgo
Consortium. 

ON THE CHARMING CAMPUS of the Max Planck Institute, it's not difficult to find
its supercomputer center: just follow the sound of the loudest air
conditioners. They're chilling the computer cluster-two long rows of
refrigerator-size black boxes occupying a shiny white room-used by the Virgo
astrophysicists, as well as by other associated research groups.  The machine,
a cluster of powerful IBM Unix computers, has a total of 812 processors and 2
terabytes of memory, for a peak performance of 4.2 teraflops, or trillions of
calculations per second. It took 31st place late last year in the Top500 list,
a ranking of the world's most powerful computers by Jack Dongarra, a professor
of computer science at the University of Tennessee in Knoxville, and other
supercomputer experts.  But as it turns out, even the most powerful machine on
Earth couldn't possibly replicate exactly the matter distribution conditions of
the 380 000-year-old universe the Virgo group chose as the simulation's
starting point. The number of particles is simply too large, and no computer
now or in the foreseeable future could simulate the interaction of so many
elements.  So the fundamental challenge for the Virgo team is to approximate
that reality in a way that is both feasible to compute and fine-grained enough
to yield useful insights. The Virgo astrophysicists have tackled it by coming
up with a representation of that epoch's distribution of matter using 10
billion mass points, many more than any other simulation has ever attempted to
use. 

THESE DIMENSIONLESS POINTS have no real physical meaning; they are just
simulation elements, a way of modeling the universe's matter content. Each
point is made up of normal and dark matter in proportion to the best current
estimates, having a mass a billion times that of our sun, or 2000 trillion
trillion trillion (239) kilograms. (The 10 billion particles together account
for only 0.003 percent of the observable universe's total mass, but since the
universe is homogeneous on the largest scales, the model is more than enough to
be representative of the full extent of the cosmos.)  Cosmologists will let
these massive points interact exclusively through gravity, which prevails over
all the other forces at the scale of the simulation. The points will be evenly
distributed in space many millions of light-years apart from one another,
except for small variations in their positions that mimic the density
inhomogeneities of the early universe. These slight displacements resulting
from the inflationary expansion ensure that some pairs of particles find
themselves closer to each other and therefore move even closer, until they join
up. Ultimately, this flow of matter will end up weaving the universe's web,
with its dark-matter filaments, clumps of galaxies, and gargantuan voids.  But
if all pieces of matter in the universe are attracting one another
gravitationally, will the universe eventually collapse upon itself in a reverse
Big Bang? It's a question that has nagged at researchers since Sir Isaac
Newton's time.  The answer emerges from a discovery a few years ago that jolted
scientists' concept of the cosmos. For decades cosmologists have been studying
the universe's expansion by looking at the light emitted by the most distant
observable galaxies. The distance this light has to travel is constantly
increasing as the universe expands and galaxies, the Earth, and everything else
get farther and farther apart. As a result, the light's wavelength is
"stretched" and thus shifted toward the red part of the spectrum (owing to a
phenomenon called the Doppler effect).  Measuring this redshift,
astrophysicists can calculate the speed of galaxies' motion and can thereby
infer the Hubble constant. Named after astronomer Edwin Hubble, who in 1929
found that the universe is expanding, this number describes how fast any two
points in the fabric of space-time are currently moving apart. In 1998
observations of distant supernovae demonstrated that the expansion of the
universe is not slowing down but accelerating-news that stunned cosmologists. 

AFTER A FEW THOUSAND TIME STEPS, THE SIMULATED UNIVERSE SHOULD COME TO RESEMBLE
THE ONE WE LIVE IN  As early as 1917, Albert Einstein had conceived a solution
to a similar puzzle. He realized that if the attraction of matter prevailed in
the universe, it would ultimately pull the universe inward into collapse. So he
postulated a hypothetical quantity, the cosmological constant, to provide
repulsion and hold the universe steady. Although Einstein later rejected his
cosmological constant, calling it his greatest blunder, theorists have now
resurrected a similar entity-dark energy-to explain the ever-faster expansion
of the universe.  This dark energy, whose mere existence still baffles many
physicists, repels everything, including itself, and therefore cannot clump.
Instead, the dark energy spreads into a sort of haze that fills the universe
and pushes against its outer limits. The true nature of dark energy remains, if
anything, even more elusive than that of dark matter.  Cosmologists resort to a
two-dimensional analogy to explain the outcome of this expansion-clumping
balance: think of ants living on a balloon that is constantly being blown up.
Much as they would all like to get together, the best they could manage would
be little ant congregations scattered around the balloon. In other words, the
pull of gravity still causes matter to clump, despite the repulsion of dark
energy, but it clumps into clusters separated by great voids in which the
density of matter is extremely low.  In the Virgo simulation, the virtual
universe, contained in a cube, expands a thousand times, until each of its
sides grows to more than two billion light-years.  The present rate of the
expansion is dictated by the Hubble constant, whose value the scientists get
from the latest observations. In the end, the simulation produces a virtual
cube with enough room for the largest cosmic structures in the universe and
enough detail to catch the cosmic web's formation in the act. That's the hope,
anyway. 

ON A BLISSFULLY SUNNY HOLIDAY, the Max Planck astrophysics institute is mostly
empty, except for a few self-conscious scientists slinking in and out of
offices, clearly embarrassed to be caught working on such a beautiful day. One
of them is astrophysicist Volker Springel, a researcher who designed much of
the algorithm that puts the virtual universe's mass in motion. The software he
and his colleagues developed calculates the gravitational interactions among
the simulation's 10 billion mass points and keeps track of the points'
displacements in space. It repeats these calculations over and over, for
thousands of simulation time steps. But what seems to be merely a repetitive
calculation task turns out to be a computational nightmare.  Unlike other
forces, such as the strong force that binds quarks into protons or neutrons and
that doesn't extend beyond an atomic nucleus, gravity never dies off
completely, no matter how far apart an assortment of bodies are from one
another. The simulation, therefore, has to calculate the gravitational pull
between each pair of mass points. That is, it has to choose one of the 10
billion points and calculate its gravitational interaction with each of the
other 9 999 999 999 points, even those at the farthest corners of the universe.
Next, the simulation picks another point and does the same thing again, with
this process repeated for all points. In the end, the number of gravitational
interactions to be calculated reaches 100 million trillion (1 followed by 20
zeros), and that's just for one time step of the simulation. If it simply
chugged through all of the thousands of time steps of the Millennium Run, the
Virgo group's supercomputer would have to run continuously for about 60 000
years.  This computational barrier is known as the n2 bottleneck. In general
terms, to know how an assembly of n points interacts gravitationally, you have
to compute a total of n x (n - 1) interactions, or approximately n2 when n is
large. To overcome this barrier, the Virgo group had to use some tricks. 
First, the researchers divided the simulated cube into several billion smaller
volumes. During the gravitational calculations, points within one of these
volumes are lumped together-their masses are summed. So instead of calculating,
say, a thousand gravitational interactions between a given particle and a
thousand others, the simulation uses an algorithm to perform a single
calculation if those thousand points happen to fall within the same volume. For
points that are far apart, this approximation doesn't introduce notable errors,
while it does speed up the calculations significantly.  This approach, however,
does not work well at short distances, because it blurs the effects of
relatively small clumps of matter. In other words, the simulation loses
resolution, and researchers can miss significant events. So Springel developed
new software with what is called a tree algorithm to simplify and speed up the
calculations for this realm of short-distance interactions. Think of all 10
billion points as the leaves of a tree. Eight of these leaves attach to a stem,
eight stems attach to a branch, and so on, until all the points are connected
to the trunk. To evaluate the force on a given point, the program climbs up the
tree from the root, adding the contributions from branches and stems found
along the way until it encounters individual leaves.  This trick reduces the
number of required calculations from an incomputable n2 to a much more
manageable n log10n, says Springel, who combined the two algorithms in a single
program. With such calculation shortcuts, he says, the Millennium Run took 26
days to run on the Max Planck Institute supercomputer. It yielded 20 TB of data
in 64 snapshots of the virtual universe, from birth to its present state. 

TOTAL DARKNESS REIGNED in the universe in a cosmic dark age that began a few
million years after the Big Bang and lasted for hundreds of millions of years.
The universe was expanding and cooling, and matter was gathering in the
gigantic strands and nodes of the cosmic web. No stars existed to illuminate
the cosmos.  Then the birth of stars and galaxies heralded a new phase, in
which a whole new menagerie of heavy atoms came into existence and new
celestial objects multiplied in all corners of the cosmos. This led to planets,
and eventually even to people, who would marvel at it all.  Over many decades,
astrophysicists have built up a picture of how these events unfolded. Gravity
compresses dark matter and normal matter, the latter in the form of hydrogen
and a little helium, into a dense mass at a node in the cosmic web. This
compressed cloud of gas and dark matter is not static but swarming with motion.
Since dark matter has no electromagnetic interactions, it cannot radiate and
therefore always retains its kinetic energy. But normal matter-hydrogen and a
few other particles-radiates some of this energy and yields to the pull of
gravity. It settles into the center of the clump, moving in ever-tighter
circles, like water swirling down a bathtub drain.  Angular momentum arranges
the gas into a ragged spiral. When it is dense enough, stars start to form:
hydrogen nuclei begin fusing, creating helium and heavier elements and giving
off light. The brilliantly studded spiral disk of such an infant galaxy, now
typically having a radius of 10 000 light-years, is embedded within a far
larger halo of dark matter. Our Milky Way, for example, has a spiral disk with
a 50 000-light-year radius surrounded by a spherical halo of dark matter with a
radius that is 10 times as great.  Gravity continues to draw nearby cosmic
aggregates of matter ever closer, so that the newly born star disks frequently
smash into one another. Some astrophysicists believe that most large galaxies,
with their many different shapes, are formed as a result of collisions.
Although most of the stars in the colliding galaxies pass by one another,
gravity pulls them out of their neat disks into chaotic orbits, creating an
irregular galaxy.  Eventually, the galaxies coalesce completely, their stars'
orbits being entangled so thoroughly that they constitute a smooth elliptical
galaxy. Any remaining hydrogen is reenergized in the collision, so that, in
time, it radiates again, settles toward the center, and then forms a new disk
around the elliptical core.  In this way, collisions can lead to a large spiral
galaxy with a central bulge. The Milky Way has such a central bulge containing
very ancient stars, some 10 billion years old; it almost certainly resulted
from one or more collisions roughly eight billion years ago. The beautiful
spiral arms, home to our sun, are a relatively recent acquisition, most of the
stars within them being only a few billion years old.  That's precisely the
kind of story the Virgo astrophysicists hope to see unfolding in their
simulation. The recently completed Millennium Run gave them the universe's
broad distribution of matter as dictated by gravity. In upcoming simulations,
other forces will come into play. Onto the web of matter the scientists will
graft the electromagnetic aspects of normal matter, which by radiating photons
allows gas to cool down and condense into spiral disks that originate stars. At
the same time, hydrodynamic pressure, which ultimately derives from the fact
that two atoms cannot overlap each other because of repulsion between their
electrons, redistributes matter along the cosmic web's strands and nodes.  If
all goes well, the cosmic web will then light up, becoming populated by
galaxies-spiral, elliptical, irregular-as well as nebulae, stars, and star
clusters. Stars and supernovae, the spectacular explosions by which stars die,
will feed energy and heavier elements, such as oxygen and carbon, back into the
surrounding gas. These effects, in combination, should lead to galaxies that
look a lot like the ones we actually see. And as the galaxies move, they will
collide and reconstitute. Little by little, the cosmic web will acquire
galaxies of various shapes and sizes, each within a neighborhood that
corresponds with its history.  After a few thousand time steps, the simulated
universe should come to resemble the one we live in, with its staggering
diversity in the forms and density accumulated matter takes. "The idea is to
make a small number of physically motivated assumptions and let those rip, and
see how close you get to the real world," says Virgo member John A. Peacock, a
cosmology professor at the University of Edinburgh, in the United Kingdom.  The
Virgo scientists hope their simulation will be detailed enough to be
statistically representative of the real universe. That means that while they
won't be able to point to a galaxy and say, "This is the Milky Way," they will
be able to search their virtual universe and find a section of it that looks
like the Milky Way. And then they will be able to follow this section back to
study its history and learn about how the real Milky Way came to be.  In the
end, the simulations will help cosmologists refine their theories, which, like
all theories, attempt to portray reality accurately. Discrepancies will point
to deficiencies in theorists' understanding of star and galaxy formation, or
possibly even in observers' interpretations of their data. The exchanges will
let scientists continually refine both the virtual universe and the
observational strategies used with real telescopes. 

FOR MILLENNIA, human beings have been contemplating the heavens, trying to
decipher the celestial mysteries. From Ptolemy and Aristotle in ancient Greece
to Einstein and Hubble, stargazers have come up with ever more elaborate models
of the universe. But as it turns out, the current generation of cosmologists is
tackling the kinds of problems they may not be able to solve by jotting down
equations on paper or peering through telescopes.  They have made remarkable
progress with their cosmological theories, but many fundamental questions
remain. When were the first stars really born? Why do galaxies have the shapes
and sizes they have? How much dark matter, exactly, is out there? What is in
the huge labyrinthine voids entangled in the cosmic web? For some of these
questions, supercomputers may provide the long-sought answers.