Home
News
You are here

All T-Mobile systems seem to be down but carrier assures it hasn't been attacked

Updated: Jan 11, 2024, 7:15 PM

1comment

We may earn a commission if you make a purchase from the links on this page.

All T-Mobile systems seem to be down but carrier assures it hasn't been attacked

Update (2): T-Mobile has told us that everything is back to normal and this was not an attack.

This was not a cyberattack. This has been resolved – it was an internal technical issue that temporarily impacted some platforms.

Update (1): Everything seems to be getting back to normal. According to a Reddit post, a rogue employee ran a script that brought the system down. This claim hasn't been verified but if that's what happened, it suggests a bad actor was able to access T-Mobile's internal system and make it dance to their tune. Alternatively, it may have been an honest mistake.

A script was executed that deleted every single namespace managed by the Conducktor platform. A namespace is essentially an abstraction of a cluster of EC2 instances that are leased from AWS. Conducktor manages the leasing, organization, networking configuration, and API orchestration to handle deployment and configuration of AWS stuff in general but its primarily EC2 instances, K8S configuration, some Redis, Elsaticache, Routing/Load Balancers, etc. It’s a lot. Too much to list and it gets complicated in a hurry and I don’t know how to succinctly summarize it. Maybe “Giant magical AWS wrapper” ?

But the overall is that this means that every team that owned an application or service, that deployed to AWS via Conducktor, had their stuff nuked. Conducktor is very widely used in Digital for APIs and applications. So most UI applications that are served from a webserver, APIs running on a java server, etc., were impacted as those servers themselves were deployed to EC2 instances managed by Conducktor. This is why this was such a widespread problem across channels (Retail, Care/Telesales, Web and App) as well as across lines of business (Prepaid, Postpaid, Business, Tmobile Money – which nobody knows exists nor should they, etc etc).

Quoting from a guy on the bridge, this was done by “A rogue admin ID” …so …I dunno, that smells really bad to me. Like, someone’s going to jail kinda bad.

The original story continues below.

###

T-Mobile is experiencing what appears to be a widespread outage, with affected users flocking to outage monitoring site Downdetector and social media platforms Reddit and X to report issues.

Initially it was suspected that the company might be under another cyber attack, but it has confirmed to BleepingComputer and CNET that the outage has been caused by an internal technical issue which is currently being worked on.

T-Mobile seems to be having an issue. Website is down, can’t log into account.

T-Mobile tells @CNET, however, that this is not a data breach or hack. pic.twitter.com/uroMDaOxG4
— Eli Blumenthal (@eliblumenthal) January 11, 2024

Customers are unable to log into their accounts and the company's app is also not working. Users who try to sign into their accounts get a message that says the website has been "unplugged" and are being redirected to an outage site, per The Mobile Report. The carrier's banking service T-Mobile Money is also down.

More worryingly, some customer accounts have been suspended because they were unable to make payments due to the outage.

Oops, somebody unplugged the site. The site is currently unavailable. We're working on it, but in the meantime please give us a call for anything you need."

The breadth of disruption seems wide, with one X user reporting that they saw an error for T-Mobile's Apache Kafka event store and received a warning that the service to route the host address account.t-mobile.com couldn't be found.

P.S. Even assuming that this is caused by an innocent error (which is common), it could still be leveraged by attackers who were waiting for a disruption like this to strike. So regardless of the explanation, monitor and exercise extra care for now if you are a t-mobile user.
— zooko ⓩ (@zooko) January 11, 2024

Apparently, even T-Mobile's frontline teams are seeing login and other errors in many applications, hindering their ability to carry out tasks like process activations and account modifications.

If you reach out to retail and customer care representatives, they will likely tell you that they are experiencing "challenges" that have impacted their ability to process transactions.

We're experiencing system challenges impacting our ability to process nearly all transactions."

Galaxy S24+ with T-Mobile and trade-in: save $600

Get the Galaxy S24+ with a subscription to T-Mobile via Samsung.com, and you can save up to $1,000 with a trade-in from participating carriers (this includes T-Mobile.) Alternatively, choose Samsung's trade-in option to save up to $600 on the S24+.

$600 off (60%) Trade-in

$399 ⁹⁹

$999 ⁹⁹

Buy at Samsung

Galaxy S23 is up to $535 off with T-Mobile and a trade-in

Alternatively, you can trade in a suitable phone and sign up to T-Mobile through Samsung.com to save up to $535 on the Galaxy S23. Extra savings are available via Samsung's Offer Programs.

$535 off (76%) Trade-in

$164 ⁹⁹

$699 ⁹⁹

Buy at Samsung

Save up to $1,050 on Z Flip 5 with T-Mobile at Samsung

Currently, you can get the Z Flip 5 with a subscription to T-Mobile through the official Samsung store at up to $1,050 off, including a $50 instant Samsung Credit with your selected carrier. As you might have guessed, the deal requires a trade-in.

$600 off (60%) Trade-in

$399 ⁹⁹

$999 ⁹⁹

Buy at Samsung

View Full Bio

Anam Hamid is a computer scientist turned tech journalist who has a keen interest in the tech world, with a particular focus on smartphones and tablets. She has previously written for Android Headlines and has also been a ghostwriter for several tech and car publications. Anam is not a tech hoarder and believes in using her gadgets for as long as possible. She is concerned about smartphone addiction and its impact on future generations, but she also appreciates the convenience that phones have brought into our lives. Anam is excited about technological advancements like folding screens and under-display sensors, and she often wonders about the future of technology. She values the overall experience of a device more than its individual specs and admires companies that deliver durable, high-quality products. In her free time, Anam enjoys reading, scrolling through Reddit and Instagram, and occasionally refreshing her programming skills through tutorials.