Cornell mobilizes to fight the power outage

By Bill Steele

When power outages swept the Northeast Thursday, Aug. 14, Cornell responded by partially activating its Emergency Operations Center. The directors of critical operations units assembled at the Cornell Police facility in Barton Hall to facilitate communication among units responding to the emergency.

"People responded immediately and did a very effective job of shutting down systems in a controlled way and redirecting power to areas that were most crucial," said Linda Grace-Kobas, interim vice president for communications and media relations and director of Cornell News Service, who was one of those gathered in Barton.

The Cornell campus was one of the few places in the Northeast that still had electricity. This was because Cornell is served by a special high-voltage transmission line that is not controlled directly by New York State Electric and Gas, explained Phil Cox, director of facilities management.

However, Cox said, the circuit that supplies the Lake Source Cooling Facility was down, cutting off the supply of chilled water for campus air conditioning. The campus has a 4-million-gallon chilled water storage tank, but that would only have lasted a few hours, he said.

"The first priority is making sure people are safe," Cox said. "Then we turn to intellectual property, primarily research animals and plants." Chilled water was redirected to research buildings.

NYSEG asked that no additional load be turned on, so power was cut to about 30 buildings. This allowed facilities staff to start up two of the old chillers that had been in use before the Lake Source Cooling Facility was built, and late in the evening more chilled water was available.

At around 2 a.m., power to downtown Ithaca was restored and the Lake Source Cooling Facility could be restarted. "We were sufficiently stable by around 3:30 that a lot of us went home," Cox said. Nevertheless, he added, everyone remained on alert for the possibility that the campus might lose all power.

The chilled water cutback also sent Cornell Information Technologies into emergency mode.

"Shortly before 5 p.m. we recognized that we weren't getting any chilled water into our air conditioning," said Rick MacDonald, CIT director of systems and operations. "We had no trouble at CCC, where there are fewer computers, but temperatures started to rise at the server farm in Rhodes Hall."

The server farm houses about 300 computers. Staff was called in to be prepared for shutdowns. To bring the temperature down, CIT asked Theory Center technicians to shut down their cluster supercomputers, which share space in the server room. "We were very pleased and appreciative that they did that, and we could keep other equipment up an hour or so longer," MacDonald said.

But by 8 p.m. the room was up to about 80 degrees, and temperatures of some equipment enclosures were up to 100, so other equipment was shut down, including e-mail servers. The campus was without e-mail for about two hours, MacDonald said.

At about 10 p.m., chilled water started coming through, and some computers were restarted. Then Facilities Management asked CIT to shut down again to conserve power, and eventually everything was shut down except for communications-related units, including e-mail and Web servers. Things stayed that way until around 5:30 a.m. Friday, when power returned. Everything was back to normal by about 8:30 a.m.