Author: Adam Clark (Network Operations)

Broadband – 27/01/2021 – London 020

We are aware of a repeat Openreach fault from yesterday affecting broadband services within the 020 area code. We have raised this with our suppliers and awaiting an update.

UPDATE01 – 13:55

We have been advised the root cause has been found and a fix is being implemented.

UPDATE02 – 16:00

We have asked for an update from our supplier.

UPDATE03 – 17:10

We have been advised an update is not due before 19:00 We have gone back to advise this is unacceptable. Our service account manager at wholesale has stepped in to push Openreach for further details and a resolution time.

We apologize for the inconvenience this is causing.

UPDATE04 – 20:20

We have had updates provided from our wholesale supplier to advise while Openreach have not provided any firm ETA and raw fault detail. They believe the outage is being caused by an aggregated fibre node serving what they refer to as the Parent exchange and the secondary exchanges.

We are continuing to push for updates and proactively getting updates from wholesale now.

UPDATE05 – 02:30

We have been advised the fault has been resolved. We are awaiting an RFO and will publish once provided.

We apologise for the inconvenience.

04/01/2021 22:40 – Broadband Network

Our network monitoring has alerted us to one of our broadband LNS gateways reloading. This has resulted in broadband services connected via this device to disconnect and failover to alternative gateways.

The device has reloaded and returned to service. We are currently investigating the cause internally.

We apologize for any inconvenience this may have caused but do not expect a recurrence.

Network Maintenance 19/10/2020

We will be making some changes to our network on the 19/10/2020 starting from 12:00 noon. These changes are in relation to our LINX peering and no impact is expected to be seen as traffic will be gracefully re-routed via alternative routes prior to work starting.

We will also be taking this opportunity to add additional hardware for future broadband capacity. This will also be non service affecting.

UPDATE 01 – FINAL – 16:50

This work is complete and LINX peering has been restored. No impact was seen to customer traffic.

14/10/2020 12:43– Leased Lines

We are aware of an issue affecting leased lines services delivered via Telehouse North. We are investigating as a matter of urgency. We sorry for any inconvenience being caused.

Services with backup DSL will have been automaticity re-routed.

UPDATE 01 – 12:50

Further investigations have shown this is affecting services delivered from our redundant fibre routes as well. We are continuing to work with our suppliers as the root cause has been identified outside of our network.

UPDATE 02 – 13:02

We are starting to see services recover but services should still be considered at risk.

UPDATE 03 – 14:00

We have had further feedback from our suppliers to advise this has been resolved and they believe this to have been down to a configuration issue there side. We have raised this as concern as this took down multiple fail over links for redundancy.

FINAL – 10:39

We have been advise this issue was cause by human error within our wholesale provider and the failure to follow strict guidelines when undertaking work. Due to the impact this had on ourselves and the loss of our redundant links with this provider, We are undertaking an internal review to ensure we mitigate against this in the future.

14/09/2020 22:00 – Broadband Network

Due to a unexpected reload of a broadband gateway on our network this afternoon, we are seeing a traffic imbalance on part of our broadband network.

We will be taking steps to disconnect sessions gratefully and re balance the affected gateways.

End users will see a PPP reconnect taking approximately 5-20 seconds. In the rare event your connection does not restore. You will need to power OFF and ON your router.

UPDATE01 – 22:10

This work is now complete.

London Data Center Fire – 07:25 – 18/08/2020

We have seen a number of leased lines and peering sessions go down over night. This has been caused by a possible fire (fire alarms are going off) in at least 1 London data center affecting a number of providers. We are working to obtain further information.

UPDATE 01 – 8:00

We have been advised the London harbor building (Equinix LD8)remains evacuated.

UPDATE 02 – 09:15

Equinix have advised that a fire alarm was triggered by the failure of output static switch from their Galaxy UPS system. This has resulted in a loss of power for multiple customers and Equinix IBX Engineers are working to resolve the issue and restore power. At this moment in time we do not believe there to have been a fire.

UPDATE 03 – 10:15

Equinix IBX Site Staff report that the root cause of the fire evacuation was due to the failure of a Galaxy UPS that triggered the fire alarm. The fire system has been reinstated and IBX Staff have been allowed back in to the building. We are now awaiting updates on restoring services.

UPDATE 04 – 11:15

Equinix Engineers have advised that their IBX team have begun restoring power to affected devices. Unfortunately, at present there remains no estimate resolution time.

UPDATE 05 – 12:15

Equinix have advised that services are started to be restored with equipment being migrated over to other newly installed infrastructure. We have yet to see any of our affected connections restore but keep checking for updates.

UPDATE 06 – 13:15

Equinix IBX Site Staff reports that services have been further restored to more customers and IBX Engineers continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure. Equinix advised access will be granted and prioritized to the IBX should any customers need it to work on their equipment.

UPDATE 07 – 14:20

IBX Site Staff reports that services have been further restored to more customers and increasing numbers of those affected are now operational along with the majority of Equinix Network Services. IBX Engineers continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure

UPDATE 08 – 15:15

We are please to advise we have just seen all affected services restore. Circuits remain at risk due to the ongoing power issues on site, however we do not expect them to go down again.

05/08/2020 14:23 – Broadband Disruption

We are aware a number of broadband services have been dropping PPP sessions over the past several hours. Initial diagnostics show nothing wrong our side and we have raised this with our suppliers for further investigation.

UPDATE 01 – 14:27

We have received an update to advise there is an issue further upstream and emergency maintenance work is required. Due to the nature of the work, we have been told this will start at 14:30 today. The impact of this will be further session drops while core devices are potentially reloaded carrier side.

We are sorry for the short notice and impact this will have and have requested a RFO already for the incident.

UPDATE 02 – 15:54

We have been advised the work is complete. We are awaiting this to be 100% confirmed.

28/07/2020 – 20:00 SMTP relay

We are aware our unauthenticated SMTP relay cluster has been subject to relay abuse by a compromised client. Currently SMTP services are suspended on the cluster.

UPDATE 01 – 22:30

SMTP services on the cluster remain suspended while we review. Further updates will be provided on the 29/07/2020

UPDATE 02 – 09:15 – 29/07/2020

After a full review. Due to the age of the platform, End of life support on the OS, extremely low usage levels (less than 0.1%) and the lack of support for enhanced security such as DKIM and DMARC. We have decided to withdraw the platform form service.

For customers who where using the service, We would advise migrating to authenticated SMTP provided via your web hosting provider or signing up with a free relay such as https://www.smtp2go.com/

We understand change is unwelcome, but after review we feel this is in the interest of all who still use the platform to protect your domain and others.