Month: July 2016

We reported this morning we where seeing issues with our DSL circuits, this has now escalated to issues within BT wholesale themselves affecting multiple services to providers across the UK.

We believe the same London datacenter that experienced issue yesterday is once again experiencing issues today however is now affecting other providers.

Services affected include DSL circuits dropping and not re-establishing, along with poor quality calls if they can be made / received.

Once again this is affecting providers across the UK and is not just limited to our selves, we are working with BT and other providers to assist BT in restoring services.

This is also affecting our phones and we advise clients to email support.

Once we know more we will provide an update. We apologise for any inconvenience caused.

UPDATE01 – 09:55
We are seeing call quality return to normal, however this has been officially confirmed as resolved. We are still seeing a number of DSL circuits down however.

UPDATE02 – 10:10
As advised in the previous post, we can confirm this is due to another power issue is London that has further affected BT wholesale.

UPDATE03- 10:15
Voice issues have returned and we are pushing BT wholesale for an update.

UPDATE04- 10:21
We have been advised the data centre affected is Telehouse North. We do not have any equipment of our own at Telehouse North, however key service providers do including BT which is why this issue is affecting so many providers and services. Telehouse North is a critical part of the UK infrastructure.

UPDATE05- 10:51
We have started to see all services restore, however no official conformation yet.

UPDATE06- 11:51
Services still remain stable and no further reports have been seen, however the incident has still not been closed off officially.

UPDATE07 – 12:45
This is the latest information we have received from BT about their issue: Due to a power incident at Telehouse North (London) IP Exchange Direct Access has been severely impacted since 07.45 this morning (21/07/2016).Service has been restored for all but a small subset of customers who are connected through a hardware card that requires replacing after the restoration of power. Local spares are being sought. There are still wider power restoration activities on going on site which is still expected to take until approx. 1pm BST. We will update with more information on the next update.

UPDATE08 – 13:07
No further issues have been seen and we are closing this incident for now. We will continue to monitor for the next 24 hours.

We have been made aware of issues on BT wholesale Network whereby all customers connecting though London may experience drops in connectivity but then reconnect to our network via other gateways.

If you are experiencing issues reconnecting, please power down your equipment for a few minutes to clear any stuck sessions.

BT are investigating the issue and updates will be provided when we have them.

We apologies any inconvenience this may cause.

UPDATE01 – 09:29

We have discovered the data centre that experienced issues yesterday for BT in respect to power may also be suffering issues again this morning.

Our network monitoring has alerted us to a loss of peering with one of our primary wholesale SIP carriers. Traffic has re-routed as expected and calls are flowing.

Our engineers are investigating.

UPDATE01 – 13:00
Our engineers have confirmed the fault is not within our network and have logged a priority 2 fault with our fibre provider as well as the affected wholesale SIP carrier.

Once we know more we will provide an update. At the moment this is not service affecting as traffic is automatically re-routing via alternative paths.

UPDATE02 – 14.05
Our network monitoring has alerted to further failures within this wholesale SIP carrier. The failures now extend to there SIP gateways resulting in us loosing all connectivity with them.

Further redundancy our side has automatically kicked in to re-route outbound calls around that carrier, however numbers provisioned via this carrier are offline. We have escalated this to a priority 1 fault with them.

UPDATE03 – 14.20
We are seeing endpoints at our supplier bounce up and down, We can therefor only conclude that there is a serious issue with our supplier.

UPDATE04 – 14.33
We have been provided a notice via our suppliers internal website that they have suffered “multiple failures at one of our data centers (THN)” There engineers are onsite trying to resolve the issues.

We will keep pressing for updates and update as we know more. We apologise for any inconvenience this may cause.

UPDATE05 – 15.18
Our network monitoring has advised that our end points have now come back in to service, however our private interconnect is still down and traffic is routing via the public internet. We are will awaiting official notification.

UPDATE06 – 15.24
We spoke to soon.. endpoints have stopped responding once again.

UPDATE07 – 16:51 We have observed a period of stability to our endpoints along with our interconnect also coming back in to service. Our management team have also been involved due to the lack of support from our supplier and we will be engaging with them Monday morning. We have still not had an official conformation and continue to chase.