Date of outage – 3rd January 2023
Duration – Intermittent over 3 days
Services affected – Business voice customers and Giganet phone systems.
Impact to services – Degraded voice quality, calls dropping and packet loss
Overview
On the 3rd January 2023 we were advised by a number of business and internal customers that they were experiencing issues with incoming and outgoing call quality. Following investigation from our Voice and Network Engineering teams it was evident that the issue was affecting a number of business and internal teams.
Following investigation throughout the day and working with our voice partner it was suggested that we reboot the server which would in turn restart a number of services. A change was carried out at 21:00 on the 3/1/23, however this did not resolve the issue.
On the 4/1/23 further analysis was carried out by Technical Teams and our voice partner. The Technical Teams identified sporadic spikes in CPU activity. Having discussed this issue with our voice partner it was suggested that we add additional processing power (CPU) and memory to the affected server.
A change was therefore raised to increase CPU and Memory at 21: 00 on the 4/1/23. This change was carried out successfully, however this did not unfortunately fix the root cause.
On the morning of 5/1/23 a failed disk was replaced in a SAN (Storage Attached Network). Following the replacement of the disk, packet loss and degraded call quality reduced significantly.
The replacement disk / array finished rebuilding at 13:50 on 5/1/23 and following an hour monitoring services we provided an update on our Status Page advising that a fix had been implemented.
Resolution Actions
Server and processes rebooted / restarted
CPU and Memory increased on Server
SAN Disk Replaced
Root Cause Analysis
Failed SAN Disk in Telehouse North causing high CPU usage
Preventative Actions
Root cause has been captured.
Failed SAN issue was not attributed to this issue, however knowledge base article created to allow quicker resolution should this happen again.
Internal Teams are working on internal processes as part of Continual Improvement
PRB0040115
09-01-2023