Campus power failure likely caused by crane activity

by ZACH VEILLEUX

Dead lines. In Founder’s Hall, cables feeding current from the damaged conduit are cut to effect temporary repairs.

An electrical failure that caused power outages in seven labs and dozens of offices in parts of Flexner Hall, Nurses Residence and The Rockefeller University Hospital on March 29 was likely prompted by the weight of a construction crane on 50-year-old conduit that had recently been excavated. The university’s electricians believe the conduit, which contains lines carrying high-voltage current from an electrical vault near the university’s 66th Street gate to the basement of Founder’s Hall, cracked when the crane’s weight caused new topsoil surrounding it to compress.

“Although the exact sequence of events that followed is unknown, it is likely that a combination of shearing force and moisture damaged the lines over a period of two weeks, eventually causing a short that tripped two 4,000-amp circuit breakers, knocking out power,” says Alex Kogan, associate vice president for plant operations. The damaged lines fed the first and second floors of Flexner Hall, all of Nurses Residence and the first, second and third floors and the A and B levels of the hospital.

While many labs are wired for backup power, which is produced by diesel generators that come on automatically in the event of a power failure, labs in unrenovated areas of Flexner, Nurses Residence and the hospital are not. When the power failed, critical equipment, including freezers that preserve biological samples, shut down in the Funabiki, Kapoor, Krueger, Leibowitz, Libchaber, Pavlides and Tomasz labs.

The power outage was first reported by a security guard on routine patrol at 1:18 a.m. March 29. Security quickly notified the power house, and the engineer on duty, unable to determine why the power had failed, then notified Con Edison. While Security began the process of alerting occupants of the affected areas, using both Send Word Now — the university’s emergency notification system — and manual telephone calls, electricians and engineers from Plant Operations as well as outside contractors and Con Ed emergency service technicians began trying to locate the cause of the failure. Jim Schaefer, manager of the maintenance shop and the university’s chief electrician, arrived on campus at 3:45 a.m. and consulted with Mr. Kogan via telephone. The first Send Word Now alert was transmitted at 4:03 a.m.

By 7 a.m., 15 Plant Operations personnel and six outside electrical contractors were working on the problem, and Con Ed had determined that the fault was not in their lines. Coordinating with representatives from the affected labs, Plant Operations staff supervised by Assistant Director of Plant Operations Andy Gallina worked to provide temporary power to the affected areas, running extension cords to freezers and in some cases relocating freezers to areas with power. Meanwhile, Mr. Schaefer isolated the problem to the circuit breakers in the electrical vault. When replacing the breakers with spares didn’t work, Mr. Schaefer determined that there was a short in the lines and that they would need to be circumvented in order for power to be restored.

“At approximately 1 p.m., I authorized Andy to purchase a 600-amp switch, which we do not stock, and cabling that would be needed to create a new feed using excess current supplied to an unaffected portion of the hospital,” says Mr. Kogan. The new wiring, which runs beneath the hospital, was in place and power was restored to all areas by 7:45 p.m. Occupants of the buildings were notified that power was back on via Send Word Now.

The three buildings affected will continue to draw power from the new wiring until the exact location of the failure is pinpointed and permanent repairs are made. Preliminary investigations — from line-monitoring equipment and exploratory excavations begun April 24 — indicate that the conduit that failed was located directly above new conduit laid several months ago as part of the project to construct a new electrical vault beneath the 66th Street parking lot.

“Though we won’t know all the details until the excavation work is complete, the theory is that work in this area left the conduit vulnerable,” Mr. Kogan says. Repairs to the conduit, which was to be taken out of service once work on the new vault is complete, will take place in the coming weeks in order to provide redundancy for the temporary feed from the hospital. In addition, work is under way to provide limited backup power to certain areas of Flexner Hall, Nurses Residence and the hospital where capacity is available.

Meanwhile, Plant Operations and Security have begun a review of their procedures to determine whether responses to emergencies could be improved. Among the changes to be implemented in response to this incident are:

• Earlier communication of power emergencies to key administrative personnel and earlier and more detailed notification of building occupants via Send Word Now, including contact information so that lab personnel know whom to contact for questions and updates
• The creation of a master blueprint with the location of all critical equipment as well as a prioritized list of freezers that Plant Operations personnel can refer to in order to best direct their resources
• Additional procedures for Security staff to follow during power emergencies, including commencing an immediate and thorough search of the entire campus, a visual inspection of all freezers to be documented in the security log and updates to the Send Word Now templates

“The Security personnel on duty responded well to this event, but there’s always room for improvement, particularly in terms of how people are notified,” says Jim Rogers, director of security. “We’re going to make sure that there are clear guidelines for them to follow so that they know when they need to wake someone up in the middle of the night.”