Category: RESOLVED

Our networking monitoring has alerted us to a failure of several BT wholesale provided leased lines. Initial calls to them advise they have a failure of a core network device. Engineers have been dispatched to site.

circuits with a backup connection will have re-routed.

UPDATE01 01 – 10:00
We have observed a number of connections restore with a number still offline. We are chasing our supplier for an update as they advise this looks to be power related at there rack.

UPDATE01 02 – 11:00
We have been advised this has been resolved and are awaiting and RFO.

We are aware of a problem with one of our media gateways affecting inbound and outbound calls. We are currently investigating as a matter of priority.

UPDATE01 – 13:13
Engineers have found the media gateway to be suffering from a sudden high CPU load. We have been able to stabilize the system remotely and are currently monitoring.

UPDATE02 – 13:20
We have seen a repeat of high CPU demand and are in the process of isolating the service.

UPDATE03 – 13:25
The service has been found and the root fault isolated with remedial works planned over the weekend. We apologies for the issues caused.

Additional media gateways had already been planned for deployment over December to provide additional redundancy, this will now be bought forward.

UPDATE04 – 15:00
We are still seeing some calls fail to route due to stuck sessions on customer trunks. This was caused by the SQL service being overloaded and log entry’s being missed. Work is being planned for tonight to reload the affected services.

UPDATE05 – 20:00
Work has been undertaken to stabilise the platform.

UPDATE06 – 21/11/2016 12:00
Engineers have completed the install of additional hardware and a migration plan will be designed in due course.

Our network monitoring has alerted us to CPU errors being generated on lns01.dsl.structuredcommunications.co.uk

We have raised a ticket with our hardware vendor for them to investigate and advise. The device remains in operation and routing data as expected, however it should be classed as “at rick” during this time due to the possibility of an unexpected failure.

Other LNSs within the pool are unaffected. Should the above LNS fail then circuits will automatically re-terminate on another device.

UPDATE01 – 11:30
Our hardware vendor has reviewed the logs we have provided and concluded the device could be suffering from a physical hardware issue. We are looking at options on replacing this device.

UPDATE02 – 11:52
While options are being reviewed to replace this device, we have taken the decision to remove it from the active termination pool. This means it will not accept any new PPP connection attemps, however will continue to service existing established ones.

UPDATE03 – 13:05
Replacement hardware has been agreed by the vendor. We will advise of a maintenance window in due course.

UPDATE04 -23/11/2016
The LNS was replaced without issue and is now back in service.