Telstra outage sparks call for enforceable reliability standards

The report into Telstra’s nationwide outage on 8 July, recently completed by Technology Audit Partners (TAP), has emphasised the need for enforceable reliability standards to be established, according to the Australian Communications Consumer Action Network (ACCAN).

As detailed by Telstra CEO Vicki Brady, Telstra staff completed in-person maintenance on a device called a Network Time Protocol server at 2.50 am on 8 July, as part of work to address a backup power fault. This server helps different parts of the mobile network stay in time with each other.

According to Brady, the team followed the relevant procedure — including restarting the server — but because of an underlying software configuration, the server restarted with the wrong date. This is significant because parts of the mobile network rely on accurate timing to work properly — and so as the incorrect date spread through the network, some mobile voice and data services began to fail. At its peak, the issue impacted approximately 45% of all calls and data sessions on the Telstra mobile network.

By that afternoon this issue had been addressed, Brady said — but further investigation identified a subsequent issue affecting some calls to Triple Zero, which was linked to the same software configuration that caused the original outage but required a separate fix. As a result, 611 callers to Triple Zero received an error message, and their phones may have attempted to connect to an alternative mobile network. A solution addressing this issue was not implemented until late morning on 9 July.

TAP’s investigation found the outage was primarily a result of Telstra not treating network timing as a sovereign function requiring the highest levels of oversight and protection, Brady admitted. In regards to the network timing system, TAP identified the following:

• Ownership for this function was not sufficiently defined, which led to weakness in this area and not treating it as a high-risk function of the network for the purposes of design, operations and resource prioritisation.
• The architecture supporting network timing had evolved over time without adequate end-to-end oversight.
• There was insufficient technical expertise for network timing systems and the need for stronger competency and more curiosity to investigate when something looked wrong.
• There were shortcomings in process discipline including change management, configuration control, incident management, documentation and follow-through.
   

Commenting on the findings, ACCAN CEO Carol Bennett said, “At the core of this governance failure is that Telstra did not designate time coordination with the level of importance that it deserves. A function critical enough to bring down calls, payments and emergency services nationwide should never have been left without clear ownership.

“The report also points to insufficient technical expertise supporting this critical system. It is deeply disturbing that a business as profitable as Telstra did not have the in-house capability to properly maintain a function this critical to its network.

“This confirms, again, that we must see clear and enforceable reliability standards in this sector. We must drive practices and processes that will prevent failure in the first place. We need enforceable standards that require telcos to properly resource and govern the critical systems Australians depend on.

“This failure is the latest in a string of telco network outages. At this point it is clear we have a systemic failure of sector governance, and the government must step in to address this problem by setting clear reliability requirements before the next avoidable outage hits.”

Image credit: iStock.com/FG Trade Latin