We are observing high levels off optical light loss on one of our backhaul links between our Horsham and London Cores. We have a window scheduled with Openreach between 12:00 & 13:00 on the 20th to clean (and replace) if required optics and cables at the exchange.
No outages are expected as traffic will flow across redundant paths but the works require an “at risk” notification of facility redundancy levels due to the work being undertaken.
We have been informed by one of our wholesale providers that they will be undertaking capacity upgrades on there network that affect several interconnects we provide broadband services via. We expect our redundant links to take over when the works commence but services should be considered “at risk” and may see PPP reconnections / disconnections.
We apologise for any disruption this may cause.
We are observing odd behaviour on LNS04. The device is operational, however we are observing latency on some network operations. We have spoken to our vendor who has advised we apply a software patch. This will result in the device reloading. Circuits connected via this device will experience a PPP drop but will re-route to other gateways.
UPDATE01 – 13:15
We have taken this gateway out of service for new / reconnections connections. If you are having issues then power cycling your router will result in a connection to another non affected gateway.
UPDATE02 – FINAL
The gateway is now back in operation.
We have been made aware by our suppler that they will be making changes to our 10GB interconnects at Telehouse North and East at 02:00 on the 14/04/2021 to resolve the low level packet loss being seen on some broadband services and non critical traffic types. This work should not be service affecting however one of the access switches we are connected to may reload as a result due to an unrelated issue there side. Should this occur, services will fail over to redundant paths but users will experience a PPP reconnect.
An update will be posted tomorrow morning to confirm closure.
An engineer is on route to Telehouse to complete a reboot of our 2 London core devices. Works are due to start from 07:30am today. This will be done in succession pending a successful reboot of the first device and to allow services where possible to re-route across the network. Leased line circuits without backup will see disruption when the associated core reloads.
We do apologise for the short notice.
UPDATE01 – 07:35
This work is about to start
UPDATE02 – 08:30
This work is complete.
We are seeing a number of broadband circuit’s dropping PPP and some intermittent packet loss. We are currently investigating.
UPDATE01 – 15:00
We have raised a case with our wholesale suppler and have moved traffic from Telehouse North to Telehouse East to see if this improves things.
UPDATE02 – 16:00
PPP reconnections have reduced but we are still seeing packet loss spikes to a large number of connections.
UPDATE03 – 18:00
We are still working with our wholesale supplier to understand the root cause of the issue as we have eliminated our network.
UPDATE04 – 22:00
We have chased our wholesale supplier for an update. Traffic remains via Telehouse East. The packet loss we are seeing does not seem to be service affecting at this stage but of course it should not be there.
UPDATE05 – 12:00 (26/03/2021)
We have dispatched an engineer to swap out optics on our side of the link.
UPDATE06 – 13:00
We have escalated this within wholesale as the problem remains. We do apologise for the inconvenience this is causing. We are still pending an optics swap as well.
UPDATE07 – 16:30
We have had a response from our wholesale suppler to advise additional resources have been added to our case. We are also still pending an optic swap from Telehouse.
UPDATE – 17:00
We have provided example cases of circuits where we are not seeing the same issue. We also believe this is only affecting a specific type of traffic.
UPDATE09 – 17:30
We have reached out to our hardware vendor to see if there is additional diagnostic tools can be provided. We apologise for the continued delay in getting this resolved.
UPDATE10 – 20:15
Optics have been changed at Telehouse North. Unfortunately the Telehouse engineer was not given an update from ourselves prior to starting the work which sadly resulted in traffic being dropped. We are continuing to monitor the interface.
UPDATE11 – 20:35 (29/03/2021)
We have observed some stability restore to the network over the past 72 hours. However we are still concerned there may be an issue on both London core devices caused by a Memory Leak and are working towards a maintenance window to eliminate this. Further details will be posted of the maintenance and times / impact as new “events / post” So they are clearly seen.
We are aware of a peering issue within LINX that is affecting all sessions and traffic via LON1. Traffic has re-routed via alternative routes.
Our network monitoring has alerted us that one of our broadband gateways has unexpectedly reloaded. The device has recovered and all connections have been automatically re-routed to other gateways. We are currently reviewing logs to find the root cause.
We are aware of a repeat Openreach fault from yesterday affecting broadband services within the 020 area code. We have raised this with our suppliers and awaiting an update.
UPDATE01 – 13:55
We have been advised the root cause has been found and a fix is being implemented.
UPDATE02 – 16:00
We have asked for an update from our supplier.
UPDATE03 – 17:10
We have been advised an update is not due before 19:00 We have gone back to advise this is unacceptable. Our service account manager at wholesale has stepped in to push Openreach for further details and a resolution time.
We apologize for the inconvenience this is causing.
UPDATE04 – 20:20
We have had updates provided from our wholesale supplier to advise while Openreach have not provided any firm ETA and raw fault detail. They believe the outage is being caused by an aggregated fibre node serving what they refer to as the Parent exchange and the secondary exchanges.
We are continuing to push for updates and proactively getting updates from wholesale now.
UPDATE05 – 02:30
We have been advised the fault has been resolved. We are awaiting an RFO and will publish once provided.
We apologise for the inconvenience.
Our network monitoring has alerted us to one of our broadband LNS gateways reloading. This has resulted in broadband services connected via this device to disconnect and failover to alternative gateways.
The device has reloaded and returned to service. We are currently investigating the cause internally.
We apologize for any inconvenience this may have caused but do not expect a recurrence.