We are aware of an issue affecting inbound and outbound call routing on our network that started at 15:55 and are investigating as a matter of urgency. Initial diagnostics and reports show this to be an upstream carrier issue.
UPDATE01 – 16:10 Due to the nature of the outage, Automatic outbound re-routing to an alternative carrier has been intermittent as the carrier suffering the issue has been going on and offline. We have therefore manually re-routed outbound calls and can see them terminating OK. We are continuing to investigate.
UPDATE02 – 16:15 Further investigations are showing our BGP session to the affected upstream carrier as being intermittent. While reviewing the link we have received notice from 1 of our fibre providers that they have suffered a fibre issue in the Docklands area and this is causing there network to re-converge. Outbound calls continue to terminate correctly, however inbound calls via this upstream provider may be intermittent.
UPDATE03 – 16:45 We have seen the link stabilise and inbound calls from the carrier terminating OK. We are continuing to route outbound calls over alternative carriers as a precaution.
On the Friday the 29th of January between 21:00 & 21:30 we will be upgrading our LNS servers to the latest vendor software release.
This work will start by upgrading lns02.dsl.structuredcommunications.co.uk first. We will then drop some DSL sessions on lns01.dsl.structuredcommunications.co.uk to ensure they properly re-terminate on LNS02 with traffic flow as expected. We will then upgrade LNS01, dropping all sessions once complete. Sessions should re-terminate on LNS01. Any sessions that remain on LNS02 after the upgrade ,will be dropped and moved back to LNS01.
UPDATE01 – 29/01/2016 – 20:05 This work is complete. If you still dont have service then please restart your equipment. Some device may need to be powered down for 20 minutes to clear any stale or stuck sessions.
We have been advised that our wholesale provider will be moving some of our Virgin Media circus to a new POP as part of planned capacity upgrade works at Telehouse East on Tuesday 19th Jan 2015 between 0:01 – 06:00.
Circuits with backup will continue service operation with the only interruption seen when the main service goes offline.
We apologies for any inconvenience this may cause.
We are aware that some DSL PPP sessions have dropped and are not re-establishing. We are working with our wholesale supplier to try and resolve the situation.
UPDATE01 – 12:20
We have have tracked down the fault to circuits that are provided to us as “off-net” circuits from Openreach. We have raised an escalation with our supplier and and working to resolve this as a priority.
UPDATE02 – 12:25
We are stating to see off-net circuits re-establish. We have not yet officially received a clear from our wholesale provider, however are pushing for a update.
UPDATE03 – 12:38
We have spoken to our wholesale NOC engineers who have advised that they added a new DSL gateway in to there network pool following a failure of an existing device. Unfortunately this device was not announced to our network by our supplier within our DSL BGP session, Therefore we where unable to route PPP responses back to this device. The device has been removed from our suppliers device pool and services are terminating back to us via known L2TP gateways. Service has now returned to normal. We apologize for any inconvenience caused.