T-Mobile confirms flooding, mass chaos, babies crying
T-Mobz just hit us with the official statement regarding what we published late yesterday evening. We said that their main data center was down due to flooding, and that T-Mobile call centers were also affected, in addition to the main www.t-mobile.com website being offline. Here’s the official statement:
"Severe flooding stemming from a powerful storm has affected a facility in Western Washington that houses some T-Mobile business systems. Our wireless network is operating at normal service levels. Various other T-Mobile operations may be temporarily offline or otherwise impacted. T-Mobile is working around-the-clock to restore the impacted systems. We apologize for any inconvenience this may cause our valued customers."
There you have it! And the site is back up, but looks like account information is temporarily unavailable for the moment.
On Dec 4, 2007 @ 7:26 pm, Galvatron Said:
Oh yummy hope I can py my bill by the time the week os done. Lol hm 6-7th wonder if they got site back up to post that rumore press release on the 6th or 7th regarding ther 3g/umts/hsdpa network.
Permalink | Reply
On Dec 4, 2007 @ 8:34 pm, jdh Said:
One would hope so but no umts bb still not fully in love yet with 3g
Permalink | Reply
On Dec 4, 2007 @ 8:37 pm, xnifex Said:
it looks like the billing is back up but it only has info on minutes/texts used until about 5:15 PM Mountain time.
Permalink | Reply
On Dec 4, 2007 @ 9:13 pm, Jake Said:
That entire area is pretty much underwater. I live only a few miles away, and I’m almost underwater too!
Permalink | Reply
On Dec 4, 2007 @ 9:28 pm, RAS Said:
So they’re in the Pacific northwest, where it’s been known to rain a little. And they have their servers in, what, a basement? Jackasses.
Permalink | Reply
On Dec 4, 2007 @ 9:51 pm, Jeff B. Said:
Awwwww t-mo is so cute. Its like a little baby. Or it could be a smokin’ hott 21 year old Swedish model…your choice.
Permalink | Reply
On Dec 4, 2007 @ 10:23 pm, JohnJ Said:
These writers need to be realistic, the title implies crying but where is the proof. Weather is uncontrollable! Is interesting why flooding caused such drastic outage but was a violent storm. As for people commenting, you are idiots! what does 3G or anything have to do with this situation!?!?
Permalink | Reply
On Dec 4, 2007 @ 10:44 pm, Justin Johnson Said:
Well fortunately they still have the capability to cancel past-due accounts. I had the money ready yesterday but nope, I guess I have to sit without a phone til I can call and pay. They’re gonna have someone b****ing at them when they get back online. (611)
Permalink | Reply
On Dec 4, 2007 @ 11:01 pm, Anthony to the S. Said:
great time to not be a t-mobile user
http://paidandpopular.blogspot.com
Permalink | Reply
On Dec 5, 2007 @ 12:20 am, Tmser Said:
Hey they’re offering no activation fee for customers who activate from now till the sixth
Permalink | Reply
On Dec 5, 2007 @ 12:22 am, Ben Said:
S, get an education or a clue. It is a natural disaster for the most part t-mo’s network is unscathed. they are not shutting ppl off and giving grace to those who are past due and can’t pay the bills due to natural disaster. Um, ya think that lil’ old t-mo is in yer mom’s basement? nope they wouldn’t want to take yer room away.
Permalink | Reply
On Dec 5, 2007 @ 1:10 am, todd Said:
Anthony,
Actually, yes, it’s still a great time to be a T-Mobile user. About 30 minutes ago, I got off the phone with my brother, who also has T-Mobile, and the call was crystal clear. No dropouts, no static, nothing, and I’m probably paying less than people on other carriers. Other companies I’ve used *cough*at&t*cough* can’t even provide that kind of service when their core infrastructure isn’t underwater.
Permalink | Reply
On Dec 5, 2007 @ 1:10 am, bluehorseshoe Said:
I’m surprised that a company which relies on the web site to a certain degree wouldn’t have a backup co-location to turn the switch on. Most companies that run an ecommerce solution have a backup plan with a switch in place. Hard to believe they’d let a chunk of their site reside in one location.
Permalink | Reply
On Dec 5, 2007 @ 1:36 am, todd Said:
bluehorseshoe, not necessarily. The costs of having a hot site (another site that can be up and running shortly) is astronomical. T-Mobile doesn’t really rely on the web much. To keep customers satisfied and to prevent them from leaving, they need to keep the cellular network up and running 24/7, which they have been doing this entire time. And their cell network probably has a ton of redundancy. That’s how they make their money. Sure, people can purchase new plans and phones online, but the cost of running a hot site likely far exceeds the amount of money they may lose these few days from online purchasing being down. You just have to analyze your risks (flooding being one of them), and decide if there’s a cost benefit to taking actions to reduce the risk, and in this case, there probably isn’t such a benefit. It’s cheaper for them to deal with rare, isolated incidents like this once in a blue moon than keep a hot site available at all times. Just pulling some numbers out of my ass here, say it costs them a million dollars a month to have a hot site. That’s 12 million a year, 120 million in a 10 year period. Now assume something like this disaster only happens about once every 10 years, and only costs 50 million to take care of. It makes sense more financial sense to let these disasters take their toll and then recover after that. A company that relied solely on the web to make money would probably have enough redundancy in place to prevent things like this, but the web isn’t where T-Mobile brings in most of their revenue.
Permalink | Reply
On Dec 5, 2007 @ 1:55 am, bluehorseshoe Said:
@ Todd
I’m not sure how their set up, but like I said, I’d be surprised. While your argument is understood, it’s easy to state that if their down for a day or two. What if it was more serious and they were down for a month? Then you have to factor the insurance. I’d be hard pressed to believe that if they had insurance on their web operations that the insurance company wouldn’t dig in for the back up plan. May be cheaper to do the backup.
One thing I know they have a backup plan in place is their databases. Who doesn’t these days.
Permalink | Reply
On Dec 5, 2007 @ 2:02 am, todd Said:
bluehorseshoe, I’m sure they have other places in the country they can temporarily move the datacenter to if needed, they’ve got several corporate buildings throughout the country (regional offices, call centers, etc) and I’m sure some could handle the datacenter if necessary. Or at the very least, outsource a datacenter for a short period. They have to purchase new equipment anyways (assuming what they had got destroyed) and would just set it up in the new location rather than the old one. And then restore those backups, and they’re up and running.
Permalink | Reply
On Dec 5, 2007 @ 2:31 am, Shaman Said:
Actually, not until I saw this bit did I know that something was wrong with tmo. All my services (email, text messaging, voice calls, data, etc) are fully functioning. Go Tmo!
Permalink | Reply
On Dec 5, 2007 @ 7:35 am, M. Ronson Said:
Same here. All my T-Mobile services (phone, voice, mail and SMS) worked fine without any interruption. I didn’t even know there was such a storm in Washington!
Permalink | Reply
On Dec 6, 2007 @ 8:32 am, HTHOMAS Said:
FYI, if anyone plans on getting a black berry (this i’m sure, not sure of the other phones).Don’t until T-Mobile gets this fixed. My boss bought one on Tues. and he will not be able to register the email section of his phone until they fix the issue. His phone happens to be a main source of business and he’s stranded. Each day they tell him wait 48 hours…. still waiting.
Permalink | Reply
On Jan 24, 2008 @ 3:09 pm, KP Said:
T-Mobiles Nework wasn’t the Problem “POWER” was most likely to be the Problem and “No Generator” or Fuel to refill it & or being allowed to enter the Evacuted area(S)
“Sound Logical Maybe?”
Permalink | Reply