Category: Incidents

We are aware of the national BT DSL outage that is sweeping social media. In addition to this we are now starting to see wholesale DSL services affected, whereby if a connection looses sync or PPP then it wont recover. Further diagnostics shows that BT are not delivering the session to us correctly and is also affecting other wholesale ISPs.

We are aware of an issue affecting inbound and outbound call routing on our network that started at 15:55 and are investigating as a matter of urgency. Initial diagnostics and reports show this to be an upstream carrier issue.

UPDATE01 – 16:10 Due to the nature of the outage, Automatic outbound re-routing to an alternative carrier has been intermittent as the carrier suffering the issue has been going on and offline. We have therefore manually re-routed outbound calls and can see them terminating OK. We are continuing to investigate.

UPDATE02 – 16:15 Further investigations are showing our BGP session to the affected upstream carrier as being intermittent. While reviewing the link we have received notice from 1 of our fibre providers that they have suffered a fibre issue in the Docklands area and this is causing there network to re-converge. Outbound calls continue to terminate correctly, however inbound calls via this upstream provider may be intermittent.

UPDATE03 – 16:45 We have seen the link stabilise and inbound calls from the carrier terminating OK. We are continuing to route outbound calls over alternative carriers as a precaution.

We are aware that some DSL PPP sessions have dropped and are not re-establishing. We are working with our wholesale supplier to try and resolve the situation.

UPDATE01 – 12:20

We have have tracked down the fault to circuits that are provided to us as “off-net” circuits from Openreach. We have raised an escalation with our supplier and and working to resolve this as a priority.

UPDATE02 – 12:25

We are stating to see off-net circuits re-establish. We have not yet officially received a clear from our wholesale provider, however are pushing for a update.

UPDATE03 – 12:38

We have spoken to our wholesale NOC engineers who have advised that they added a new DSL gateway in to there network pool following a failure of an existing device. Unfortunately this device was not announced to our network by our supplier within our DSL BGP session, Therefore we where unable to route PPP responses back to this device. The device has been removed from our suppliers device pool and services are terminating back to us via known L2TP gateways. Service has now returned to normal. We apologize for any inconvenience caused.

We are currently aware of an issue affecting a number of on-net and off-net Vodafone based circuits.

We will continue to work with our suppliers and publish updates as they become available.

UPDATE 01 – 12:10
We have seen sessions restore, however are still pending an update from Vodafone.

UPDATE 02 – 12:40
Telecity have confirmed that this incident was caused by an isolated power issue. Telecity engineers are working to ensure the underlying issue is fully resolved. Until we receive confirmation from Telecity that all work has completed, previously affected services should be considered as at risk.

Further updates will be provided as they are made available from our suppliers.

UPDATE 03 – 14:57
We have been advised via another supplier of the following:

We are aware of an issue at Telecity Sovereign House. The issue has been confirmed to be a wide-spread power outage affecting the ground floor up to, and including the 4th floor. The power appears to be currently intermittently available and engineers are on site investigating the source of the problem. At this time, the DC are unable to advise the cause or why resilient systems failed to take over.

We are aware of a MTU issue on our DSL network affecting lns01.dsl.structuredcommunications.co.uk. This is preventing JUMBO FRAMES from getting to this device and forcing DSL circuits that terminate on our network to obtain a lower MTU (1460-1462) Where end users CPEs and networks are not configured to handle fragmentation correctly this will manifest itself as access problems to various websites.

We are currently working to resolve the problem.

We apologise for any inconvenience this may cause.

UPDATE 01 – 13:56We have isolated the problem to a interconnecting VLAN between LNS01 & CORE02. The VLAN will need to be reconfigured, however this will drop all active DSL sessions that terminate on that device. We are looking at other work around at this time.

UPDATE 02 – 14:20 We have reconfigured the VLAN and interface which has resolved the issue with JUMBO frames and our carriers. Existing L2TP tunnels to our carriers LACs are still negotiated at a lower MTU. Any existing sessions and new session that join that tunnel will still have a lower than normal MTU.

UPDATE 03 – 16:00 Tunnels have been reset and sessions restored. MTUs are now being seen at 1500 again for ADSL.

We have been made aware of multiple exchanges outages in Sussex affecting 20CN services across multiple providers. At this moment no ETA has been given and we are awaiting further information from our suppliers.

We apologize for any inconvenience caused and will provide further updates as they become available.

UPDATE 01 – 08:34
We have been advised that hardware at a patent exchange has been replaced and services should start to restore. Any client still having issues are advised to power cycle there equipment.

Our network monitoring has alerted us to increased latency and packet loss on some Talk Talk EFM circuits. We have raised a fault with our supplier and are currently awaiting there response.

UPDATE01 – 10:55
Our network monitoring is showing a reduction in packet loss and latency, however we have not had an official update.

UPDATE02 – 11:50
Our network monitoring is again showing latency increasing along with packet loss. We have chased for an update.

UPDATE03 – 13:13
Our supplier has advised this is being caused by an overloaded transit link due to windows updates. We have escalated this response to there management as this level of service should not be contended on a wholesale level. We are awaiting there response.

UPDATE04 – 15:00
Latency has returned back to normal and packet loss has gone. We are still awaiting an official clear and response to our escalation.

We are aware of a issue affecting connectivity to multiple networks within the UK. Initial testing had shown this to be a problem with Level3, however they are advising this to be part of a wider problem within the UK.

We are currently monitoring and will look at re-routing traffic if this continues, however this would involve shutting down several of our peers and affecting traffic via unaffected links. This would be a last resort option.

Once we know more we will advise.

UPDATE 01 – 11:11

Level3 have updated us to advise they are now tracking a global routing issue. This now seems to be isolated to Level3, however will be affecting any network that peer with Level3. Re-routing traffic is still be an option, however may have no impact to networks that rely on Level3.

UPDATE 02 – 11:36

Our network monitoring has alerted us to routing failures to common websites such as the BBC, Facebook & Google. We have taken the decision to shutdown our Level3 peering with Adapt. While this has had some positive impact, 3rd party networks that are using Level3 will still be having issues.

UPDATE 03 – 11:55

Level3 have advised that services are starting to be restored. We have re-enabled our Level3 peer via Adapt and have noticed they also took the decision to showdown there link to Level3. Traffic are currently routing via Cogent via Adapt.

UPDATE 04 – 20:00

Outage report from a 3rd party monitoring network: http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/

At 10:01 our network monitoring detected a loss of IP transit to our primary carrier Adapt / Level3. Engineers have shutdown this link and traffic is routing via alternative carriers. We will continue to monitor and update as more information becomes available.

We apologize for any inconvenience caused.

UPDATE 01 – 11:01
We are continuing to try and contact Adapt for an update as there are no issues with the Level3 side of the link. We believe from external testing there is a routing failure within the Adapt network.

Traffic is continuing to route via other links, however due to the volume of traffic being observed latency may be higher than normal. Should this continue then we will look to re-route traffic via our 3rd interconnect on CORE01 that has a higher capacity link.

UPDATE 02 – 11:18
We have been advised that the issue has been resolved, however we are seeking further confirmation before we bring the peer back in to service.

UPDATE 03 – 11:18
We have spoken to the senior network architect at Adapt who has confirmed service has been restored and we have re-enabled the session. We are observing normal traffic flow, however will continue to monitor the link. Once an RFO has been published by Adapt we will provide this.