T-Mobile Says Outage Wasn't a Cyberattack

T-Mobile experienced a service disruption yesterday, impacting both customers and employees who were unable to access their accounts on the company's systems. Although the problem has since been resolved, initial concerns arose over the possibility of a cyberattack.

T-Mobile has reached out to us to assure us that it wasn't the case. 

 

This was not a cyberattack. This has been resolved – it was an internal technical issue that temporarily impacted some platforms.

According to a Reddit post (cited by The Mobile Report), there is speculation that the outage might have been triggered by a rogue employee running a script. T-Mobile has not confirmed this claim, leaving room for the alternative explanation that it could have been an unintentional error.

The post read:

 

A script was executed that deleted every single namespace managed by the Conducktor platform. A namespace is essentially an abstraction of a cluster of EC2 instances that are leased from AWS. Conducktor manages the leasing, organization, networking configuration, and API orchestration to handle deployment and configuration of AWS stuff in general but its primarily EC2 instances, K8S configuration, some Redis, Elsaticache, Routing/Load Balancers, etc. It’s a lot. Too much to list and it gets complicated in a hurry and I don’t know how to succinctly summarize it. Maybe “Giant magical AWS wrapper” ?

 

But the overall is that this means that every team that owned an application or service, that deployed to AWS via Conducktor, had their stuff nuked. Conducktor is very widely used in Digital for APIs and applications. So most UI applications that are served from a webserver, APIs running on a java server, etc., were impacted as those servers themselves were deployed to EC2 instances managed by Conducktor. This is why this was such a widespread problem across channels (Retail, Care/Telesales, Web and App) as well as across lines of business (Prepaid, Postpaid, Business, Tmobile Money – which nobody knows exists nor should they, etc etc).

 

Quoting from a guy on the bridge, this was done by “A rogue admin ID” …so …I dunno, that smells really bad to me. Like, someone’s going to jail kinda bad.”

During the outage, customers faced difficulties accessing their accounts through both the web and the app, while T-Mobile employees also experienced similar restrictions. Consequently, some customers faced suspensions, preventing them from making necessary account modifications and activations.

Fortunately, the service disruption has been addressed, and we are now awaiting further updates from T-Mobile to provide clarity on the nature of the issue.

Source: The Mobile Report