The rockefeller university

Overflow. Two views of floodwater from Hurricane Sandy: the FDR Drive, near RRB (left) and 62nd Street, near Scholars Residence.

by ZACH VEILLEUX

When water from the East River first spilled over its banks and began washing onto the FDR Drive late in the evening of October 29, the several dozen or so administrators and essential personnel monitoring the storm knew it was not likely to stop at the curb. The lowest levels of several campus buildings were vulnerable — and some damage was going to be inevitable.

Wind and water. Flooding and gusty winds caused damage in several places on campus. Damaged wiring in the Smith Hall Annex backup sever room (top); flooding in the parking garage and bike room beneath Faculty House (left); waterlogged electrical equipment in Flexner (right).

Over the previous three days several dozen members of the Plant Operations and Custodial teams had been working to prepare the campus for high winds, flooding and a possible blackout. A 20,000-gallon diesel fuel tank had been topped off, outdoor furniture was secured, doors and windows were checked, roof and storm drains were cleared, sandbags were piled against sub-basement doors and security cameras were repositioned to monitor flood-prone locations. Turner Construction, the contractor for renovations under way in Flexner and Welch Halls, ensured that all loose construction materials were brought indoors and that barriers and fences around the work sites were secure. Arrangements were made for key personnel to work extra shifts and stay over in guest rooms. Messages were sent telling non-essential personnel to stay home. The university was as prepared as it could be.

But flooding like that had never happened before, at least not in the past few decades. Nobody could predict how high the water would come, or exactly how it would enter buildings. The nearly 13-foot storm surge downtown had been dramatic — but how far the waters might rise six or so miles upstream was anyone’s guess. The lowest floors of several university buildings have ground-level access to the FDR Drive, useful for moving heavy equipment like boilers and generators in and out of mechanical spaces. Significant flooding from the river would threaten electrical circuits that feed labs, water pumps that provide domestic water and the electronic controls that regulate boilers.

So as the river rose — it peaked at 10:15 p.m. 32 inches above the FDR pavement — and began pouring into the Flexner mechanical room, Smith Hall Annex D level and the boiler room located beneath the hospital, the strategy was to mitigate the damage as effectively and safely as possible and wait for the water to recede.

The first area to see damage was in Smith Hall Annex. A backup server room located on the D level was filling with floodwater, threatening equipment. Starting at 8 p.m. IT personnel began an organized shutdown of the equipment, turned off the underfloor power distribution system and elevated equipment that could easily be relocated. The D level facility houses redundant servers that take over if the primary equipment, located in Weiss high above the water, should fail. Because the D level room, which once served as the university’s only data center, was identified as being potentially susceptible to flood damage several years ago, it is now used only for backup equipment. (The Weiss server room, located on the third floor, was built in 2007 and houses the primary equipment.) As an additional layer of protection, the university’s mission-critical data is also backed up off-site.

“Although the shutdown of mirrored servers resulted in some minor issues with latency, no data was threatened and all critical campus services remained fully functional throughout the storm,” says Armand Gazes, director of IT operations and networking, who oversees the university’s data facilities and was on site throughout the storm.

Meanwhile, the situation was also becoming critical in Flexner, where, at about 10 p.m., water reached the bottom of newly installed circuit breakers — huge electrical panels that route hundreds of thousands of watts of electricity from Con Edison to equipment in Flexner and Smith Halls. Jim Schaeffer, manager of the maintenance department and the university’s chief electrician, recommended they be shut down.

“If water gets into those circuit breakers while they are live, they can short out and be damaged beyond repair. But if they are shut down preemptively, they will work fine once they dry out,” says Alex Kogan, associate vice president for plant operations and housing. “So the decision was made to turn them off and switch the north campus over to emergency power provided by our diesel generators, which use a different distribution system located a floor above.”

Although the plan hit a small hitch — an emergency circuit accidentally mislabeled during installation wasn’t immediately switched on, leaving a few labs without emergency power until the problem could be correctly diagnosed by university electricians — the outage proved to be short and normal power was restored at about 1:30 a.m.

By morning, as the extent of the damage throughout the region became clear, Rockefeller was operating in a mostly normal state. Although the floods had damaged some equipment and the campus had lost two trees, both near the President’s House, a thorough survey of the buildings indicated no structural damage or loss of scientific assets. There were no injuries to personnel and no loss of research animals. Although Internet connectivity was unstable on Tuesday due to the primary service provider’s loss of both primary and backup power, and the backup provider’s unusually high load, the university’s internal network was fully functional.

“Considering the magnitude of the storm and the amount of damage it inflicted in our area, we are fortunate to have escaped with relatively little loss,” says Marc Tessier-Lavigne, the university’s president. “Thanks to the hard work and dedication of nearly 100 personnel, many of whom worked long hours and faced tough commutes, the university was able to reopen just 48 hours after Sandy departed.”

Staff from Plant Operations, Security, Human Resources, Information Technology, the Comparative Bioscience Center, Laboratory Safety and Environmental Health, Custodial and Housing all contributed to storm preparation and recovery efforts. In the hours and days following the storm, there was much work to be done to get everything back to normal — from picking up fallen branches to replacing waterlogged equipment. Cleanup and repairs will take several weeks, and when those are done, decisions will need to be made about how and where to fortify or relocate susceptible equipment.

The damage totaled roughly $500,000. Losses include damaged water and fuel pumps; three damaged elevators, two of which are freight cars serving the boiler room and are not heavily used; stored network and electrical equipment in the Smith Hall Annex server room; and two broken windows, in Welch and Bronk. Flooring and sheetrock damaged by the water need to be replaced. And at Faculty House, spare appliances and building materials kept in a lower-level storage room were damaged by flood waters.

“Although we got through this event well, and were even in a position to offer assistance to institutions in the area that sustained much worse damage, there are always lessons learned, and my team is already working on implementing improvements that will help us when the next disaster strikes,” says Mr. Kogan.

Inflate in case of flooding

While the cleanup work from Sandy continues, Plant Operations personnel are also making plans for the next time the East River floods. Among several improvements being considered is the purchase of a Tiger Dam system (left), which was demoed near the entrance to Founder’s Hall in early December. Filled with water from a fire hose or pumped out of the river itself, these 50-foot-long, 18-inch diameter tubes can be stacked in a pyramid formation to block floodwater. More effective and easier to deploy than sandbags, they would likely have prevented much of the damage that occurred to the lower levels of the Power House and Flexner Hall with Sandy. “This is one of several products we’re investigating to better protect vulnerable areas from the river,” says Alex Kogan, associate vice president for plant operations. “Our goal is to have better options to prevent water infiltration before we are next faced with this type of disaster.”