BellSouth’s R-Force is ready when disaster strikes

BellSouth’s R-Force is ready when disaster strikes – BellSouth Corp’s disaster recovery group

Gale Miklos

It’s not only Boy Scouts who like to be prepared.” Bellsouth is one company that won’t be caught off guard if a disaster hits its operating region.

In May, for example, the company tested its ability to respond and recover from a disaster when it pretended that a hurricane hit Perdido Bay in the Florida panhandle, wiping out a central office. A team of BellSouth employees and a small fleet of tractor-trailers responded to the simulated disaster by heading for Perdido Bay with a new central office on wheels” in tow. The next day, just four hours after their arrival, the Bellsouth R-Force supplied dial tone from the portable CO. Later, a microwave link was established to a nearby CO, and the R-Force was ready to begin restoring priority circuits and facilities.

This simulated disaster dramatically showed the readiness and quick response of the R-Force, thanks to Bellsouth’s one-of-a-kind portable CO.

This project was conceived after Hurricane Hugo roared through the Southeast, knocking out commercial power and threatening Bellsouth’s facilities in North Carolina and South Carolina. The telco is now ready and waiting if the unthinkable should happen–a major network switching or transmission failure.

Bellsouth’s new disaster recovery trailers provide a quick, flexible and temporary response to a catastrophic network failure until all the supplies and equipment needed to make permanent repairs can be rounded up by the company’s equipment vendors. Vendors have told Bellsouth that it would need a minimum of 14 days to restore a seriously damaged 20,000-line CO. After witnessing the havoc wreaked by Hurricane Hugo and the Hinsdale, Ill., CO fire, Bellsouth decided to find a way to begin restoring service more quickly if disaster strikes again in its service area. This commitment to quick service restoration led to the CO on wheels.

Bellsouth’s custom-built, air-conditioned trailers are no slap-dash, let’s-stick-some-extra-equipment-in-a-truck solution. The stark white trailers were specially designed to carry almost all the equipment and supplies a telco crew would need to restore telephone service.

The trailers contain a 235-kW diesel generator; several kinds of digital loop carrier equipment; miles of fiber and copper cable; connectorized distributing frame blocks; a Northern Telecom DMS-100 SuperNode with enhanced network and integrated services digital network capabilities; test equipment; distribution frames and bays; and other items. Complete with built-in, gel-cell batteries to back up the switching equipment one trailer provides power, one contains transmission equipment, and the other two contain switching and related gear.

Built by switch vendor Northern Telecom to BellSouth’s specifications, the switching and transmission trailers feature special external “belly bays” underneath the trailer floors that provide quick and easy exterior access to banks of line connectors and other equipment. These connectors run inside to the switch and peripherals so repair technicians can work outside the trailers to connect trunks, lines and digital facilities while CO installers work inside to establish dial tone. With more than 11,000 lines, 3000 trunks and various digital facilities to put to work, using connectors was the only way to go.

The main switching trailer includes a stand-alone DMS-100 digital switch; 2400 analog lines; 1200 AT&T SLC 96 digital lines; 2000 DMS-1 Urban lines; and 2400 digital trunks. Bellsouth currently is investigating wiring options that would permit the use of line controllers as trunk controllers, providing an additional 2400 digital trunks.

The trailer also comes with software so the switch can be used as a signal transfer point, an E-911 tandem switch, a local switch, a toll switch, or both a local and toll switch. Therefore, the unit can replace virtually any of the 1600 stored-program control switches in Bellsouth’s network, including switches from other manufacturers.

Other equipment in the main trailer includes a SuperNode-30, a 36-port link peripheral processor, an E-NET enhanced network with 15,360 network circuits, two input/output equipment bays, two digital trunk equipment bays, two subscriber module equipment bays, a line group equipment bay, two line-concentrating equipment bays, five trunk module equipment bays, a power distribution center bay and a maintenance administration position.

The second, or extended-switch, trailer carries DMS-100 ISDN equipment that can provide 320 2B1q lines or 480 analog lines and 20 primary rate interface lines or 480 digital trunks. This trailer can handle up to 5120 analog lines and another 480 digital trunks and also has space for more frames or bays.

The R-Force’s transmission trailer connects calls that originate or terminate in the portable CO to the outside world. This unit includes a fiber interface via AT&T DDM-1000S; a Tellabs 532 TCS digital cross-connect system; SLC 96 and DMS-1 Urban digital line terminals; D-4 channel banks; an office repeater bay; a Stratum 2 clock; and DSX-1 and DSX-3 panels. All the transmission equipment terminates on the DSX panels and belly-bay connectors so that the telco has complete flexibility to configure the equipment to meet the needs of the local interoffice facilities and cabinetized microwave equipment.

Because the lines switched by the disaster recovery trailers are likely to be heavily used, the portable office was engineered to handle common channel signaling system 6, or roughly double the call-carrying capacity of a traditional 10,000-line CO. In other words, those agencies telephone service restored by the Bellsouth R-Force are likely to be able to get their calls through.

Simulated Disaster

Like any real emergency, the May exercise began with calls from Bellsouth’s Florida Emergency Center to the Atlanta-based members of the R-Force, who perform on-call duties in addition to regular assignments. Within hours, team members had shut down their equipment-which is kept fully operational at a suburban Atlanta CO–and were on their way to Perdido Bay, where Bellsouth was hosting a meeting of its interexchange carriers. Three of the four trailers and a digital microwave radio system were trucked down separately.

Intended to simulate the loss of a small switching office, the exercise was enacted in a remote hotel parking lot where neither communications nor commercial power was available. When members of the advance team arrived from Georgia, they first established a ground field for the trailers and portable microwave tower, using cellular telephones to communicate from the site.

The uncooperative weather during the simulation helped set the stage: heavy rain and lightning continued throughout the first day. The trailers pulled into the hotel parking lot at 8:15 a.m. on May 8, and by 11:45 a.m. the primary switching trailer was providing dial tone. A microwave link with the Pensacola, Fla., CO-some 10 miles away across Perdido Bay–was established later that day.

Throughout the exercise, the generator trailer, which has a 600-gallon fuel tank capable of operating approximately 30 hours at full load, provided all the necessary ac power to the switch and transmission gear. It was backed up by a locally provided portable generator. No commercial power was used during the event.

The trial proved that there cannot be too many electrical outlets for all of the lights, computers and tools needed during an emergency. BellSouth also is looking at ways to provide simpler connections with a microwave tower.

Overall, the exercise proved that the disaster recovery trailers and the R-Force could rapidly restore a disabled CO.

The trailers were designed using experience gained from a 1989 Bellsouth simulation of a CO failure in Raleigh, N.C. Using a containerized DMS-10 module with a 1200-line capacity as the replacement switch, the team quickly brought together various switching, transmission, microwave and power equipment, but found that some basic incompatibilities made it difficult to get the equipment to work together. There also was not enough work space for the CO installers and outside plant technicians to work together inside the trailer. As if to reinforce the results, the Raleigh experiment ended in a driving rainstorm-the first wave of storms from Hurricane Hugo, which soon cut commercial power to several area COs.

Hugo’s fury drove home the decision to develop the full-blown disaster recovery trailers, which were delivered this past spring, prior to this year’s hurricane season. In addition to Perdido Bay, the R-Force recently participated in an exercise on the streets of downtown Atlanta that gave team members experience working in a congested urban environment.

The four trailers stand waiting-powered 24 hours a day and monitored both by employees from the adjacent CO and remotely by a network operations center, just like other Bellsouth COs.

And that’s just fine with Bellsouth, because the trailers are like a good insurance policy: something you must have, but hope you won’t have to use too often.

COPYRIGHT 1991 PRIMEDIA Business Magazines & Media Inc. All rights reserved.

COPYRIGHT 2004 Gale Group