0

  0

  0

  0

  0

“If ignorance is bliss, why aren’t there more orgasmic people out there?”

  0

I understand natural disasters (if that is what caused this) but there is absolutely no excuse for executives of T-Mobile to not tell customers of the problem so we could take action to continue to run our businesses.

I am in Colorado using T-mobile hotspot@Home for all my phone needs. For several DAYS the service has been increasingly erratic, undependable and now totally useless.

Yesterday I was vainly attempting to facilitate a webinar/conference call with 25 people across the country and I kept getting dropped from the call.

I looked like an idiot for the simple reason I trusted T-Mobile and the executive management decided to pretend or deny they have any problems with their system.

T-Mobile clearly has a catastrophic technical problem, but it is insignificant in comparison to the PR/Customer relations problem they have created by not telling loyal customers the situation. A simple email would have done, they have my email address and send me other stuff.

Yesterday I could have easily used another phone for the conference call if I had known I needed to, it would have been an irritation but no big deal. Today, after having to call T-Mobile customer service twice and finally finding out some hint of the problem (from very nice polite CS people) I bought another phone solution.

  0

She then goes on to add 100 bonus minutes to my plan – which is Funny because I just told her that the only service that I can use inside my apartment and at work is UMA due to poor coverage area.

I think TMO needs to come out with a press release so we aren’t left in the dark with what is going on.

But it must be pretty big if it’s taking TMO this long to get things fixed.

I WANT MY DANG UMA!

  0

But it is also human nature to see the other side: Thanks Tmo.

  0

  0

  0

  0

  0

  0

  0

  0

  0

When will this thing be fixed? It’s been DAYS!

  0

  0

  0

OK, people, so here is the full, real skinny, from someone who actually works for T-Mo IN the T-Mo datacenter in Bothell. I experienced all this first hand, so I know it’s true.

Monday, coming in to work, the rain (and the real problem, adding to the runoff was a lot of melting snow from last weekend) had caused all kinds of problems all over the city, mostly traffic snarls due to washed out roads. The usually 4-ft wide creek that runs behind the building had turned into a raging river several hundred feet wide in places, and was threatening to overflow the levies that hold it in. Bothell City/PD officials and T-Mo facilities made the decision at @10am to evac the building. Most technical folks went home to VPN in, and some stayed to help sandbag. Sandbagging efforts were started early, both up on the levies as well as around the building itself. Unfortunately as the afternoon wore on and the rain continued, the levies themselves began breaking down and leaking, and water began pouring over into the parking lot for the building, which is surrounded on two sides by levies.

So the real cause of the outage was not flooding IN the datacenter itself, but due to the fact that water had risen outside the building and wound up causing our transformer to literally blow up. It actually exploded. Had never seen that before. So we lost street power. We switched immediately to generator power with no downtime, but basically that only lasted @15 minutes, due mostly to the fact that the generator itself was also becoming submerged. Some teams had a chance to gracefully shut down systems (ours among them), but many systems in the DC just lost power abruptly – not the ideal way to bring down a multi-node oracle cluster… So the Bothell DC lost all power at @4:25pm PST. Confusion reigned for about 20-30 minutes after that.

We have a second Bothell DC that was not flooding (in the Canyon Park area), and all emergency incident operations switched over to there. Some IT systems were able to swing service to systems there, but many (most) were not. Efforts were made to turn up critical servers at the other DC, and bring up a read-only subset of services (mainly we tried to get the website to do something other than just time out…) At one point, as the flood waters continued to rise, and power was still out, there were executive level decisions being considered to physically remove servers from the DC and at the very least take them upstairs (yes, our DC is on the first floor of the building – more on that in a sec). Yes, we actually considered going into the dark DC and by the light of flashlights, un-racking several tons of equipment (mostly drive cages) and trying to move them physically upstairs (no elevators of course) where they would not get water damage. Thank god the water never made it past the front door, or we would have been doing that.

So that was actual outage. Bothell city, Puget Sound Energy, and emergency crews worked all night long and got street power restored to the building at @3:20am 12/4 (PST). So we spent a total of approx 11 hours without power to our main datacenter.

So let me tell you what was in there. Basically most all T-Mo IT systems and administrative function are represented there. Ideally all the systems there have an analog in our other data center (not the other Bothell DC, but our second DC in Tampa, FL). In practice this is not the case – again, more on that in a sec. NO engineering gear is here, so all the switches, all the SS7 gear, all the radio and cell equipment – all that resides in different DC’s that are geographically diverse (both from each other as well as from Bothell). This is why none of you lost cell service or vmail. However… Blackberry service, the T-Mo website, billing reconciliation (note, NOT billing, just reconciliation for billing – sorry, not likely to be getting any free calls), credit card processing, store connectivity, call center connectivity, HR systems, corp email systems, etc are all a different story.

Those systems were affected hard and are still being brought back up. As of noon PST today, most are back up and functioning nominally again, with just a few issues among them. Really, for as much poor planning and bad architecture, and for as hard as these systems went down, I am impressed with our technical teams’ ability to do what needs to be done to provide workarounds and fixes quickly. I also would have predicted a week-long, painful period of getting everything back up and running, but in fact, it has happened much faster than that and its because our technical teams have been working their asses off.

To those that commented that we (as in T-Mo) dont care about our customers or that T-Mo employees themselves did not do anything to help, you couldnt be more wrong. (Specifically JULIO: “Yet another failure by corporate America in pursuit of the almighty dollar while simultaneously not giving a rat’s a$$ about the customers that pay the bills that keep their companies going.”) I can tell you for a fact that MANY T-Mo employees went WAY above and beyond the call of duty, staying at the DC, snadbagging, working all night trying to restore service, etc. Some people even slept at the DC, and managers/leadership went out and bought a bunch of air mattresses on their own dime. I wont go so far as to say ‘heroic’ but I can tell you that a LOT of people worked their ASSES off Monday night and all day Tuesday, trying to get things restored ASAP. So get the full story before you rail on T-Mo for “not caring”. I am not going overboard and talking about “how much t-mobile does for society” or anything sappy like that. I am just saying that the regular folks that work here are people too and we did and are doing everything we can to get things back up and working.

To those that have commented about the fact that we should be geographically diverse and loss of a DC should not take down services the way that it did – well, all I can say is that I couldnt agree more. All said and done, a lot of people (the DR team especially) do have a lot to be embarrassed about. I am not disputing that at all. Yes it was an act of god, yadda yadda. But best practices and correct architecture should always be able to mitigate those things – the fact that our shit was not together enough to handle an outage like this IS inexcusable and I am not making any claims that there are not areas of this company (like all companies) that have their head up their asses. Someone mentioned the “5 9’s uptime” thing, and we (T-Mo IT – not engineering) have been struggling with that anyway, irrespective of flooding. And going back further than that, the choice of location for the DC in the first place is just downright STUPID. (I mean, 1st floor of a shared building in an office park that sits in what amounts to the middle of a swamp? C’mon, retards!) I absolutely will not dispute that. And many of us who work there and have decades of IT experience have been saying it for years. No matter what else happened, the fact is that this was a problem waiting to happen. And Monday, it finally did. What can I say – us worker bees do our best and give our technical recommendations, but C-level’s and execs make their decisions and we are stuck with them. Such is life. Hopefully this will be a BIG wakeup call to some folks who work in Bellevue to re-evaluate what DR means and how to run a world-class IT organization. We will see.

“But someone at t-mobile should have made a backup plan. After 9/11, most real companies have a disaster recovery plan that prevents and works arround even acts of terrorism and acts of God.” You are absolutely correct and T-Mo DID fall down on the job there. I am not making any excuses for that, but all I can say is that most of the underlings here know full well just how fast and loose we play with system availability, and while we do what we can to mitigate it, there is only so much that can be done if the execs and C-level management dont see a problem. Thats all I can say about that.

So there you go. That is the real deal. By COB tomorrow, I imagine this will all be 100% restored. At this moment I would say we are about 95% restored with only a handful of problems lingering. FWIW.

  0

  0

  0

happy holidays

gigi

  0

  0

I just wish I could get the three hours back that I lost today because of stupidly trying to figure out what was wrong with my UMA (since it supposedly wasn’t affected).

So far as comments about customer service, my only other experience in the US is with the “nation’s most reliable network”. I like dealing with CSRs who appreciate their customers and aren’t condescending pricks. So far, tmo has exceeded my expectations in that regard. Very friendly people, and nearly no wait time to speak with someone. I cannot say the same about the “other” carrier, save one friendly rep who gave me his ext and told me I could call him any time. He appreciated the thousands of dollars I’ve spent on his company’s services. Too bad the rest didn’t. Good riddance.

  0

  0

  0

  0

  0