Verizon is tight-lipped about outage cause, but the embarrassing reason has been figured out
Verizon may have pushed out the wrong update at the worst possible time.
Verizon | Image Credit - NBC News
Wrong update at the wrong time
The outage lasted nearly ten hours and impacted voice, text, and data services. Because the disruptions weren't region-specific, infrastructure failure was an unlikely culprit. Verizon has ruled out a cyberattack, but it didn't provide details about the software problem that wreaked havoc.
Entner believes that Verizon may have pushed out a buggy update at the wrong time. Companies usually roll out major updates during off-hours to keep potential disruptions to a minimum.
It appears that someone at the company deployed a minor change early in the day, which kicked off the blackout around 11:00 am ET and impacted hundreds of thousands of customers.
Lee W. McKnight, an Associate Professor at Syracuse University's School of Information Studies, has chalked up the problem to a failed update to a Virtual Network Function (VNF), which knocked down other VNFs. VNFs are services running as software processes and deployed in the cloud.
The interconnected nature of VNFs explains why the service kept coming back and cutting out intermittently. McKnight believes Verizon's technical staff needs more cloud training to prevent a recurrence.
It appears that someone at the company deployed a minor change early in the day, which kicked off the blackout around 11:00 am ET and impacted hundreds of thousands of customers.
Only eligible devices in high-end markets where Verizon has activated 5G Standalone (5G SA) were affected, which suggests a change was made to the 5G SA core.
It looks like their 5G SA (Standalone) core went down during a minor feature change.When carriers do massive upgrades, they do that between 2 a.m. and 4 a.m. in the morning. A noon start for the crash indicates 'fat fingers' for a smaller change that cascaded through the system.
Roger Entner, Recon Analytics, January 2025
Some users claim that 5G SA was disabled — at least temporarily — in their area after the outage.
The interconnected nature of VNFs explains why the service kept coming back and cutting out intermittently. McKnight believes Verizon's technical staff needs more cloud training to prevent a recurrence.
Like an engine stalling out, if all Verizon VNFs are not in sync [or] well orchestrated, the network is off-key.
Lee W. McKnight, Syracuse University's School of Information Studies professor, January 2025
Given Verizon is being frugal these days and has laid off 13,000 employees to run efficiently, paid training likely sits low on the corporate agenda.
Everything that could have gone wrong went wrong
Since the update was rolled out in the morning, it was likely an insignificant update, but a human error, such as mistyping a command, made it snowball into a widespread outage.
Verizon has been in damage control mode since, issuing a profuse apology to customers and offering a $20 credit.
The outage couldn't have come at a worse time because Verizon just had a rough year and has installed a new CEO with the hope of turning things around. AT&T and T-Mobile capitalized on the carrier's misery by reminding customers that their networks were operating as normal.
5G SA is an independent 5G network that doesn't depend on 4G infrastructure. Verizon initially relied on 4G resources to bring 5G NSA (Non-Standalone) to customers, and began transitioning to 5G SA last year, five years after rival T-Mobile.
The outage couldn't have come at a worse time because Verizon just had a rough year and has installed a new CEO with the hope of turning things around. AT&T and T-Mobile capitalized on the carrier's misery by reminding customers that their networks were operating as normal.
5G SA is an independent 5G network that doesn't depend on 4G infrastructure. Verizon initially relied on 4G resources to bring 5G NSA (Non-Standalone) to customers, and began transitioning to 5G SA last year, five years after rival T-Mobile.
Perception is everything
Verizon has been trying to reverse the trend of customer losses by offering more value, while proclaiming that it's the best. Confusingly, AT&T and T-Mobile have also been claiming the same, making it difficult for customers to decide who to believe. T-Mobile's claims have recently been supported by insight companies Opensignal and J.D. Power.
Interestingly, T-Mobile was victorious by a slight margin, meaning its rivals are not too far behind when it comes to connectivity metrics and perception. However, a large-scale outage could torpedo perception about Verizon's competitiveness, and its reluctance to offer more details is only making things worse.
Follow us on Google News
Things that are NOT allowed:
To help keep our community safe and free from spam, we apply temporary limits to newly created accounts: