When UCLA found out that a
planned upgrade to its brick and mortar data center was going to surpass
the original budget estimate by several million dollars, it began
thinking "inside the box"--and chose cargo-container computing.
- By Dian Schaffhauser
- 08/11/11
The latest and potentially most powerful research data center at the
University of California Los Angeles
(UCLA) was delivered on the back of a flat bed truck from Austin, TX.
It was put in place with a crane. The facility resembles the kind of
container used to run portable recycling centers. But when it's fully
loaded and performing research in physics, economics, genomics, and
biochemistry, among other disciplines, the Performance Optimized
Datacenter, or POD, will support more than 1,500 compute nodes. This far
exceeds the campus' traditional "brick and mortar" data centers in
sheer computing power, and yet it fits into a compact space of 40' x 8'.
The story of UCLA's "pod" began in January 2008. At that time, 18
clusters of high performance computing were being delivered from two
primary locations, the Math Sciences data center and the
Institute for Digital Research and Education
(IDRE) data center. Between the two, researchers could run jobs on any
combination of about 800 nodes. But capacity was fairly well tapped out,
according to Bill Labate, director of UCLA's
Academic Technology Services and managing director of IDRE.
IDRE works as a kind of shared service. "People 'buy' nodes and
then we integrate them," said Labate. Technically speaking, users don't
really buy the nodes and hand them over; IDRE does the purchasing, based
on well researched specs that provide high value for a low price;
operations are ultimately funded by the 175 research groups in 64
departments and centers on campus that partake of the center's services
and equipment.
A Plan for Growth
Given that the IDRE center had physical space available, the
Institute put together a proposal to increase computing capacity. It
brought in a third-party data center engineering firm to do a quick
estimate of what the project would take. The needs were substantial. The
goal was to squeeze in 1,600 computer nodes; but the existing power
infrastructure could only support about 400 nodes. Somehow, the center
would have to be revamped to support the dramatically increased wattage
required by those additional nodes and the cooling they'd require. The
consulting firm's estimate came to $4.6 million. Based on that number,
the university granted IDRE funds to do its build-out.
That's when IDRE knuckled under to sort out various configurations
for the project. The center brought in another consulting firm to do a
more detailed cost estimate. This time, however, the estimate for 1,600
nodes exceeded $7 million. A second proposal for an 800-node build-out
came in at $4 million.
"We were stuck almost $3 million low in budget," Labate said. "We
went through a whole series of [scenarios]: What if we only do this many
nodes capacity? What if we go to these hot aisle containment systems?
It got down to where it was almost ridiculous to do something with this
brick and mortar data center."
Computing in a Box
At the same time modular data centers, also called
"containerized data centers," were getting media attention. Sun
Microsystems had previewed "Project Blackbox," in late 2006. In mid-2008
both
HP and
IBM
began publicizing their modular data centers. All promised to reduce
energy usage, a major consideration in data center expansion. According
to a 2010 study by Gartner, energy-related costs account for about 12
percent of overall expenditures in the data center, a share that's
expected to rise as energy costs themselves rise.
Labate began having conversations with people at other institutions
that were trying these new kinds of containerized set-ups, including
UC San Diego and
Purdue University, both major research universities. "We started thinking, these modular data centers could be a viable alternative."
Due diligence led the university to settle on HP's pod for four
compelling reasons: density, price, flexibility, and energy efficiency.
The pod could contain 1,500 nodes, nearly the same count as the planned
data center build-out; but the price would be only $2 million versus $7
million.
When IDRE was in the middle of its shopping, it discovered that not
all modular data centers are alike. "Some of the modular data centers
required you to have specific equipment--and as you can imagine, the
equipment was specific to the vendor of that particular unit," noted
Labate. "That would have forced us to standardize forever on that
particular node, which is something we would never want to do. With the
HP pod, we can put anything we want in there."
That flexibility is important. The various compute clusters on
campus currently carry four brands of equipment in all kinds of
configurations. But on a regular basis, IDRE goes out to bid on
computing nodes to find the optimal combination of feature set for
price. The current choice happens to be HP, he explained. "But that's
not to say there might not be another vendor in the future that comes
along that meets that price/performance curve."
As of August 2010, the minimum standards for those computer nodes are:
- 1U, rack-mounted; half-width preferred (two nodes sit side by side in the slot);
- Dual six-core 2.6 GHz Intel Core i7 or Xeon 2.66GHz CPUs;
- 4 GB of memory per core;
- 160 GB to 250 GB hard drive per node;
- A Gigabit Ethernet port, DDR/QDR InfiniBand interconnect, and PCI-Express slot; and
- Three-year warranty.
Preparing for Pod Arrival
UCLA placed the order for an HP 40c pod in October 2010. The
vendor could have shipped it within six weeks, but, as Labate pointed
out, "You have to build a site to put this thing on." Rather than going
through the exercise of accepting delivery of a 43,000-pound retrofitted
cargo container, stashing it somewhere, then hauling it out again for
final placement, the campus told HP to keep the pod in its Austin
factory until the site preparation was done.
| | 
A crane lowers UCLA's pod into its new home, a former loading zone. | |
Working closely with the facilities and capital planning staff
members, as well as HP, IDRE identified a former loading zone and
storage area on the main campus that fit several criteria: The pod would
be hidden so as not to mar the aesthetics of the other buildings on
campus; it was sizable enough to accommodate not just one but two pods,
for the day when UCLA decides to increase its high performance computing
capacity; and the site could accommodate the major upgrade to power and
chilled water that would be necessary for the running of both the new
pod and the future pod.
To support the 110,000 pounds the pod would weigh once the
equipment was in place, workers poured a concrete slab that Labate
estimated to be between two and three feet thick.
| | 
The portable data center in place. Loaded, it will weigh about 110,000 pounds. | |
Labate noted that HP would have delivered the pod equipped with its
computing components, ready to plug in and add to the network. That's
the way it's promoted on the Web site: "fully integrated and tested ...
as part of a complete data center solution." "We didn't want to go that
route," he said. "It came from Austin to L.A. on a truck, and I didn't
want to subject my equipment to that stress."
Insulation in the Extreme
Once the site was done and the pod delivered, a crane lifted the
pod off the flatbed and onto the slab, and it was ready to outfit. On
the interior, the pod holds 22 racks, each 50U tall or about 88 inches,
along the 40-foot wall. Opposite is a narrow aisle wide enough for
accessing and moving equipment.
To control the climate, blowers on the ceiling force air
conditioning downward; hot exhaust goes out the back and rises up,
enters the coolers, gets cooled, and sinks down again. Equipment located
on the "hot aisle" side of the pod, where all the hot exhaust blows, is
actually accessible from the outside. The pod is outfitted with sensors
for temperature, chilled water supply, and humidity, as well as leak
detectors under the water mains and overhead condensate drip pan.
Describing the pod as "really solid," Labate observed, "When you
walk inside, with all of this equipment running and 36 blowers--it's
extremely loud. You go outside, and you can't hear a thing. You can't
feel anything on the outside. You don't feel cold when you touch the
metal."
He estimated that the highly maximized environment saves about a
third more power than that which would have been required by a brick and
mortar data center.
"It's very Spartan," Labate said. "It's a purpose-built data center
with extremely tight engineering for the purpose of being highly energy
efficient. It's not something you want 10 or 15 people to try to get
into. It's strictly made for the equipment, not the people."
Proximity to the other data centers was fairly unimportant to the
placement decision, Labate said. All of the centers are linked over the
campus networking backbone for Ethernet connectivity and interconnected
by wide area Infiniband for input/output.
Because the pod's monitoring systems--both environmental and on the
computing gear--can be managed remotely, Labate anticipated weekly
visits to the pod to handle work that needs to be done. "The No. 1 thing
is keep as many nodes up and running as possible. If we have a
catastrophic failure, we're going to go fix it. But if a hard drive goes
out or memory goes bad, we'll queue those up and send somebody out
there once a week."
Mishaps and Skeptics
In the 10 weeks the pod has been in place, IDRE has loaded
about 250 nodes into the racks--between 15 percent and 20 percent of
capacity--which is just about where it needs to be right now. During a
recent check, Labate said, the entire system was running at 95
percent--about 733 jobs--across all three data centers. "When you run on
this system, you have no idea where you're running, nor do you need to
know. That's all handled in the background with a scheduling system."
But there's still a bit of "teething" going on with the new
environment. It hasn't been hooked into the campus alarm or fire systems
yet. And, Labate added, more people need to get inside to learn about
the new layout. In its short time on campus, that newness has already
caused a mishap. Twice, the same vendor contracted to maintain the
FM-200 fire suppression system has accidentally discharged the gas.
"They know how to do FM-200. They're not familiar with the actual pod
configuration."
As might be expected, the idea of a pod-based data center was
off-putting to some of the data center technical crew, Labate said.
"When the rubber meets the road, they have to be the ones to maintain
it," he pointed out. "They have to work in it, put equipment in it."
IDRE sent a couple of people to the Austin factory to watch how the
container was built. HP brought a pod to a local movie studio and
members of IDRE took a piece of equipment there to see how it would fit.
Eventually, the skeptics warmed up to the new approach. "They started
to realize what it is," he said. "This isn't a building. It isn't a
rehearsal studio. It isn't an office. It's a data center, period. That's
all it is. It doesn't claim to be anything else."
So while most people were skeptical that this was actually a good
decision, Labate noted, "in the end, it turned out to be probably the
best decision we could have made."
About the Author
Dian Schaffhauser is a writer who covers technology and business for a number of publications. Contact her at dian@dischaffhauser.com.
Following are the key points to remember for RAID level 1.
Following are the key points to remember for RAID level 10.
