Cox Communications Blues

As you can clearly tell from the picture above, I have recently been working with a telecommunications company. Anyone in the IT industry who has worked with SW Bell, AT&T, Bell South, or any of the old "Ma Bell" companies, knows how excruciatingly painful it is to work with them. As a customer, once you get done you need to do something slightly less painful to ease the transition. So I chose to pull my ears off, since that is way less painful than dealing with a telecommunications company!

Cox Communications had always been one of th easier companies to deal with, but after our recent project at Lakeside Imaging nearl Lake Hefner, I am starting to feel they have been drinking from the same water well as the old Bell (and now AT&T) folks. Our goal was to increase the bandwidth for inbound DICOM images at Lakeside Imaging, and secondary to that was to have more redundancy at their facility in case of a T1 failure. The radiologists at Diagnostic Radiology read several remote hospitals and imaging centers from their location at Lakeside, and all this had been done over a single T1 line from Logix Communications. The T1 line was just not able to handle the increased load, and had been down off and on over the past year for a variety of reasons. In addition, the workload between Diagnostic Radiology's office at Lakeside and their main office in Edmond had been increasing. When the Lakeside or Edmond office T1 line would go down, their entire operations ground to a halt.

So the plan was to install two lines - one from Cox and another from Logix. At first Cox said they could do load balancing between their line and the Logix line using something we call BGP routing. Both lines would be a T1 (1.544 megabits per second) so the Cox sales person and sales engineer thought the customer would get double the bandwidth (3.08 mbs). I had not done a lot of BGP routing, since this is mainly for the Internet backbone providers, but I do have extensive experience in computer networking and TCP/IP so I did a bit of research on BGP prior to the installation. I found out that BGP is primarily for failover and will NOT provide load balancing. Hence, the Lakeside office, even with two T1's, would only have a single T1 worth of bandwidth and the system would fall over to the backup T1 if the primary went down. This did not solve the problem of needing more bandwidth.

So I had to show the Cox people this would not work. They came up with a solution - a 3 mbps Cox line with the Logix T1 as backup. All traffic would go over the Cox line and the Logix line would be a backup. Not ideal but acceptable. The Cox folks promised me and the radiologists that they would handle everything on their side of the firewall, and all I had to do was handle the changes inside the network and reroute all the VPN's. I also had to get an entire Class C subnet for the Lakeside office, and apply to ARIN - the main USA internet registry - for what is called an ASN number. That took a couple of weeks of paperwork.

We scheduled a cutover day in November, but when the day came Cox was not ready. So we rescheduled the cutover day for two weeks later. The morning of the cutover they again were not ready. They wanted me to run cables for their end, their sales rep kept thinking should be doing the coordination but I kept telling her "this is your deal, I handle my side of the firewall, you guys do your side like you promised". We had delay after delay.

The icing on the cake was yesterday. We finally got the firewall switched over to the Cox router, but we were going to plug in the Logix T1 into the Cox router - again, this is their side of the firewall. I agreed to meet a Cox engineer there at 9:00 am. Of course, I get there at 8:45 and wait until 9:30 - no engineer. You know how that goes - "A Cox engineer will be there at your home between 8:00 am and noon". They treat their commercial customers the same way I guess. 9:00 am doesn't mean 9:00 am, it means sometime between 9 and 11 and you should wait on them.

Well, I didn't wait, I left. As soon as I got to the office, Cox called up a bit after 10 am and said "where are you at?" ARRRGGHH All that was needed was to plug a cable into a jack, but they wanted me there to plug in the cable. I finally convinced them to plug in the cable. Got that done.

So 30 minutes later they call and say "OK, now we need to coordinate this with Logix."

I say "Have you talked to Logix at all?"

"No."

AARRRGGGHHH

So Logix has no idea what we are doing, nobody has given them a heads up, nobody even knows if they can do BGP routing. Remember, this is all going on on the COX side of the firewall, not on my side. So I call Logix, conference them in, wait for the holding game for them to get to 3 levels of technicians. I thought it was funny that the Cox engineer was on the phone with me, seeing exactly what us customers have to go through to get through to someone at these telco's that know something - payback. Anyway, we finally get through to someone at Logix that knows something and they say "oh, we cannot do BGP with Cox." Double ARRGGHHH!

I finally blow my stack and told the Cox guy to get with Logix and call me when they get this resolved. I was very tired of doing their end of the deal.

Luckily I got an email from Cox today saying the got it worked out with Logix and everything is up and running. FINALLY! Nearly two months later, at least we got it going. Whoever said this telco stuff is easy?

I am sure anyone with an IT background who has worked with Telco's knows EXACTLY what I am talking about. It is the most excruciatingly difficult thing you can do, working through layers and layers of technicians. I had to deal with a total of 9 people at Cox to get this thing done. Only took one of me. Geez.

If you are a radiology director or healthcare professional, then this is just a tiny glimpse of the challenges involved in setting up teleradiology or an integrated PACS network. At InTelemed, our background is in Internet, DICOM and telecommunications. We know how to set up your radiology network, integrate your different modalities, make all your DICOM talk, set up your telecom and VPN's, and most importantly, we have the battle scars to prove it.

Hope my ears grow back!