NTP server that traveled back in time caused massive Aussie mobile outage

Telstra skipped a patch, didn’t record changes, had no idea it was an accident waiting to happen

What Happened

• During planned maintenance, a Telstra NTP server restarted with its clock set to 2006 and distributed the incorrect time across the network.
• Network equipment then rejected connections because digital certificates appeared invalid, disrupting mobile services, electronic payments, transport systems and some calls to Triple Zero.
• Telstra had not applied an available software update, and an earlier design change had not been properly reviewed, documented or incorporated into the maintenance procedure.

The Cyooda View

This is a near-perfect example of why resilience is rarely about buying more technology.

A time server went back 20 years, certificates stopped validating and critical services began falling over. But the real failure happened much earlier: a known software update was not applied, a design change was not properly documented and maintenance was performed without fully understanding the consequences.

An issue that reportedly could have been fixed for less than $20,000 instead caused an outage likely to cost millions. And the final bill will not be limited to technical remediation. Add customer compensation, regulatory scrutiny, contractual disputes, insurance claims, external investigations and the legal costs of working out who knew what, when, and whether reasonable steps were taken.

Boards often approve large transformation programs while small maintenance and documentation issues remain unresolved. Cybersecurity and operational resilience are built in those unglamorous details. The expensive incident and the legal fallout, is often just the final invoice for years of deferred housekeeping.

Source: The Register Original: Read More