When the dream job becomes a nightmare

13.09.2006
Network administrators frequently find themselves deeply engrossed in the command-line world of configuring routers and switches. However, properly installing a network goes far beyond creating VLANs and such. Construction, power, pathways, spaces and interfacing with other organizations are just a few of the issues to consider when designing a properly functioning network. The following illustrates some things that can go wrong if little attention is given to variables beyond the network configuration.

The dream job

You landed a dream job as a network administrator for a large regional hospital. Your first assignment was to oversee the design and installation of network services in a new four-story physicians office building planned for construction. The responsibility for all network services, from capacity planning to integration with the existing network to installation and configuration of the switches and routers, was yours.

You attended the first project meeting, where architects met with the hospital administration to determine needs, costs and design layout desires. A few weeks later you received the first set of drawings from the architect with data jack locations noted as dictated by the eventual tenants' needs. From this you designed the network.

The architect had provisioned in the drawings a communications closet per floor, and you determined the number and type of switches needed for each closet based on the quantity of cables terminated per closet. You obtained a quote from the hospital's network equipment vendor and submitted the projected costs for the network electronics.

You obtained a quote for running single-mode fiber-optic cable from one of the hospital core distribution centers to the new building's first-floor closet, designated as the building's Main Distribution Frame (MDF). The interior data copper cabling as well as the single-mode fiber-optic cables from the MDF to each communications closet were are included as part of the construction bid, so you didn't include these costs.

The first surprise

After the construction bid was released the phone system manager informed you that they were planning to deploy voice over IP (VOIP) for all voice communications. You mumbled under your breath as your original electronics estimate was for non-Power over Ethernet (PoE) switches. You resubmitted your cost estimates, now 22 percent higher.

The bid was awarded and construction began. After the frame of the building was completed, you toured the site with the fiber cabling contractor. You both noticed there were no conduits from the new building's MDF to the outside. While the cabling contractor had included in the estimate costs for installing four buried conduits to the building, building penetration was not included. Without this, the fiber-optic cable could not be pulled into the building.

The cabling contractor, like you, assumed it was the architect's responsibility for planning all pathways into the building. The hospital facilities manager assumed it was your responsibility because the network group handled the external conduit system. The general contractor insisted they were only required to run pathways and cable inside the building, per the bid documents.

The cabling contractor's fix was to core drill through the new foundation, at a significant cost. Midway into the project, the costs for data network connectivity had exceeded your original estimate by 35 percent.

You were called into a meeting with the director of Communication Services (your supervisor) to explain the cost overruns. The meeting did not go well, and you were requested to review your estimates for any other possible cost overruns.

You reviewed the network design and were satisfied with it. You examined the core router you planned to connect the new building network to and discovered the interface you planned to use was no longer available.

You asked the core network engineer what happened to the interface that was set aside for the new building and she responded that she didn't know you had wanted to reserve that interface, and that if she'd known she would have reserved it. While there were two more available, she said that both were reserved for other projects.

You obtained from the core engineer an estimate for providing another blade for the router, a cost that increased your already bloated estimate. You had another heart-to-heart talk with your supervisor. He took some of the blame for the cost increases, as he realized he should had worked closer with you on this project from the beginning.

Substantial completion

The building construction reached substantial completion and it was time to install the network electronics. As a proactive action you placed the equipment order two months prior, and had the equipment configured, connected to the new router interface, and burned in for several weeks. In addition, you had the phone system manager connect several voice terminals to the switches to ensure the new VOIP system worked as planned.

You installed the switch in the rack in the new building MDF. To your horror, you discovered that the PoE switches, because of their increased power needs, had a 20-amp type plug. A 20A plug will not fit in a standard receptacle. While it was a minor additional cost to fix the problem, your supervisor was not pleased.

When you returned to your office you had a voice-mail from the subcontractor installing the fire alarm system. The subcontractor requested a fiber-optic circuit between the new building and an adjacent building to connect the fire alarm monitoring system. Your original design was for a 24-strand (12-pair) single-mode fiber-optic cable so while this was a surprise you were relieved that you had overestimated needed capacity.

That relief quickly faded when the fire alarm subcontractor said that their system required multimode fiber connections. You didn't inform your supervisor of your mistake as you figured a couple of mode conditioning (multi to single mode) patch cables would solve the problem.

You installed the new cables but the fire alarm contractor said that they couldn't certify their system with mode conditioning cables. Furthermore, the date for occupancy was drawing near and the local fire marshal would not permit occupancy until the alarm system was operational. The only solution was to pull a multimode fiber-optic cable from the building MDF to the core distribution room, at an additional cost.

Lighting the network

The building was brought online and afterward the supervisor called you into his office. He explained that when he saw your original estimates without a contingency figure, he added 30 percent to the estimate. And he explained that the project manager included an additional 10 percent contingency to the estimates for items not covered by the general contractor. Therefore, the building was completed close to budget.

The supervisor admonished you for not informing him of the fire alarm issue and required you to examine your mistakes and create a plan for avoiding similar errors in the future.

In the next article I will present a few tips how to accomplish that.

Greg Schaffer is the Director of Network Services at Middle Tennessee State University. He has over 15 years of experience in networking, primarily in higher education. He can be reached at newtnoise@comcast.net.