Verizon is tight-lipped about outage cause, but the embarrassing reason has been figured out

Verizon may have pushed out the wrong update at the worst possible time.

1comment
verizon outage cause 5G SA update
Verizon | Image Credit - NBC News

Unfortunate timing and a 5G update gone wrong may have been behind Verizon's Wednesday outage, according to market intelligence firm Recon Analytics' founder, Roger Entner. In a statement to CNET, Entner helped fill in the blanks after Verizon revealed only that a software issue had brought its network down.

Wrong update at the wrong time


The outage lasted nearly ten hours and impacted voice, text, and data services. Because the disruptions weren't region-specific, infrastructure failure was an unlikely culprit. Verizon has ruled out a cyberattack, but it didn't provide details about the software problem that wreaked havoc.

Entner believes that Verizon may have pushed out a buggy update at the wrong time. Companies usually roll out major updates during off-hours to keep potential disruptions to a minimum.

It appears that someone at the company deployed a minor change early in the day, which kicked off the blackout around 11:00 am ET and impacted hundreds of thousands of customers.

Only eligible devices in high-end markets where Verizon has activated 5G Standalone (5G SA) were affected, which suggests a change was made to the 5G SA core.


Recommended For You

Some users claim that 5G SA was disabled — at least temporarily — in their area after the outage.

Lee W. McKnight, an Associate Professor at Syracuse University's School of Information Studies, has chalked up the problem to a failed update to a Virtual Network Function (VNF), which knocked down other VNFs. VNFs are services running as software processes and deployed in the cloud.

The interconnected nature of VNFs explains why the service kept coming back and cutting out intermittently. McKnight believes Verizon's technical staff needs more cloud training to prevent a recurrence.


Given Verizon is being frugal these days and has laid off 13,000 employees to run efficiently, paid training likely sits low on the corporate agenda.

Everything that could have gone wrong went wrong

Since the update was rolled out in the morning, it was likely an insignificant update, but a human error, such as mistyping a command, made it snowball into a widespread outage.

Verizon has been in damage control mode since, issuing a profuse apology to customers and offering a $20 credit.

The outage couldn't have come at a worse time because Verizon just had a rough year and has installed a new CEO with the hope of turning things around. AT&T and T-Mobile capitalized on the carrier's misery by reminding customers that their networks were operating as normal.

5G SA is an independent 5G network that doesn't depend on 4G infrastructure. Verizon initially relied on 4G resources to bring 5G NSA (Non-Standalone) to customers, and began transitioning to 5G SA last year, five years after rival T-Mobile.

Do you view Verizon differently after the outage?

Perception is everything


Verizon has been trying to reverse the trend of customer losses by offering more value, while proclaiming that it's the best. Confusingly, AT&T and T-Mobile have also been claiming the same, making it difficult for customers to decide who to believe. T-Mobile's claims have recently been supported by insight companies Opensignal and J.D. Power.

Interestingly, T-Mobile was victorious by a slight margin, meaning its rivals are not too far behind when it comes to connectivity metrics and perception. However, a large-scale outage could torpedo perception about Verizon's competitiveness, and its reluctance to offer more details is only making things worse.
Google News Follow
Follow us on Google News

Recommended For You

COMMENTS (1)

Latest Discussions

by 30zpark • 3
FCC OKs Cingular\'s purchase of AT&T Wireless