Incident: Horsham DC CORE – 18/04/2024 – 12:15

We have identified an issue within our Horsham Data Center that has impacted several services. Service stability has been restored, and we will provide a further update shortly.

We apologize for any inconvenience caused.

UPDATE01 –

At 11:30 AM, our network monitoring system detected a service disruption affecting multiple platforms situated outside our network.

Preliminary diagnostics indicate that the issue was confined to our Horsham data center.

Subsequent investigations identified the root cause as an MTU (Maximum Transmission Unit) anomaly, implicating either the core infrastructure in Horsham or an interconnect linking our two locations.

Our engineering team pinpointed the issue to a specific interconnect and promptly isolated it from service. This action restored full network operations.

Scheduled Network Maintenance – Broadband (15/04/2024)

We are delighted to inform you that all LNS upgrades have been successfully implemented. As part of this upgrade, we have reconfigured the RADIUS steering away from the legacy devices that remain, facilitating a gradual cessation of sessions on these devices until they are decommissioned during Phase 2 of our network upgrades.

Furthermore, we have removed restrictions on all limited 950Mbps FTTP packages following the completion of the LNS works, thereby enabling the full 950Mbps download throughput for our valued customers.

Scheduled Network Maintenance – Broadband (12/04/2024)

Following the installation of our new broadband gateways, they are now ready to be deployed into active service. Our engineers will be decommissioning the existing gateways, ensuring a smooth migration of sessions. Once a gateway is cleared of sessions, it will be taken offline and replaced with the new device.

LNS01 will be removed from active service for new sessions as of today (12/04/2024). PPP sessions will be gracefully terminated, prompting your router to establish a connection with an alternate gateway. This process should be completed within a few seconds.

UPDATE01 – 12/04/2024

LNS01 has been swapped out with the new device and is now accepting new connections. We will monitor the device and then make the remaining gateway changes.

Scheduled Network Maintenance – Broadband (10/04/2024)

Following the installation of our new broadband gateways, they are now ready to be deployed into active service. Our engineers will be decommissioning the existing gateways, ensuring a smooth migration of sessions. Once a gateway is cleared of sessions, it will be taken offline and replaced with the new device.

LNS02 will be removed from active service for new sessions as of today (10/04/2024). PPP sessions will be gracefully terminated, prompting your router to establish a connection with an alternate gateway. This process should be completed within a few seconds.

UPDATE01 – 11/04/2024

LNS02 has been swapped out with the new device and is now accepting new connections. We will monitor the device and then swap out the remaining gateways in the same way shortly.

Scheduled Core Network Maintenance – THN (10/04/2024)

Our engineering team will be conducting operations within our Telehouse North rack to implement our new LNS units as a component of our scheduled Q2 upgrade initiatives. While no disruptions are anticipated, it is recommended that the site be regarded as potentially vulnerable during this period.

UPDATE01 – Final

This work is complete and the units have been installed. Further notices will be given on bringing them into service.

Scheduled Core Network Enhancements for Q1/Q2 2024

In the coming months, we’re excited to announce a comprehensive upgrade to our core network infrastructure at Telehouse in London. This enhancement involves the integration of cutting-edge hardware for improved performance and reliability. Detailed information about the schedule and impacted services will be provided as we approach each phase of the network maintenance.

Broadband Drops – 20/01/2024 – 22:00

We are aware of several PPP drops this evening that has caused loss of service for several minutes at a time. This has been traced back to our DSL upstream provider.

We apologize for the time taken to raise this notice however we where engaged in finding the root cause.

We have seen service stabilise however this is still at risk but we are closely monitoring.

UPDATE 01 – 20/01/2024 – 23:00

We have observed another PPP drop.

UPDATE 02 – 20/01/2024 – 23:15

We have reached out to our wholesale provider for an update but we are continuing to monitor. Our next update will be 21/01/2024.

UPDATE 03 – 21/01/2024 – 07.54

We have seen services restored over night. We are awaiting an RFO that will provided on request one made avaliable.

Anyone without service this morning please power down your router for 20 minutes.

We do apploigse for the service disruption.

Wholesale DSL (Broadband) maintiance 28/03/2023 – 30/03/2023

We have been advised by our wholesale provider they are migrating our layer 2 broadband traffic to new hardware on the dates below:

Change 1: 28/03 23.00 – 29/03 06.00
Change 2: 29/03 23.00 – 30/03 06.00
Change 3: 30/03 23.00 – 31/03 06.00

The plan is to steer all traffic to the new BNGs as priority, migrate 1 of the current BNGs a night (in turn dropping the subscribers/l2tp tunnels for that device, making sure they’re terminate elsewhere) and keeping the other current BNG in steering also as a lower priority for additional redundancy. This will be staged over the 3-night period.

We have checked our network and the new changes being proposed are already covered by existing configuration but we do expect circuits to drop and reconnect during this maintiance window.

If there are any issues, we expect changes to be reverted.

UPDATE01 – 29/03/2023

Change 1 has bee completed without issue and we are seeing traffic via the new BNGs and TEPs

UPDATE02 – 30/03/2023

Change 2 has bee completed without issue and we are seeing traffic via the new BNGs and TEPs

Wholesale DSL maintiance 26/01/2023

Following on from the broadband issues occurring on the 20/02/2023 (report still pending) We have been advised from our wholesale supplier that they are replacing the equipment at fault there side tonight (26/01/2023) from 23:00 – 06:00 hours under an emergency maintenance window which would drop DSL connections for a few minutes.

We as a pre-emptive measure have already moved our traffic away from the device they are working on in Telehouse North over to Telehouse East to avoid the disruption. This has been confirmed by both network teams.

Once the work is complete, we will move traffic back.

We appreciate the hesitation this notice will bring given the recent issues, However this should not cause disruption given the circumstances are very different.

UPDATE01 21:30

We have been advised this work will likely be suspended due to an pre existing network issue within our wholesale providers network. We have seen our Telehouse North link we removed traffic from this afternoon go out of service which we have been indirectly advised of will be down to the pre existing network issue. Telehouse East remains online but should be considered “at risk”

UPDATE02 22:46

We have been indirectly advised that network issues within our wholesale provider are starting to settle down. Our primary Telehouse North link remains offline but is actively trying to reconnect and restore redundancy. Services remain “at risk”

UPDATE03 23:29

We have seen our primary Telehouse North link restore and remain stable for over 25 minutes. We will leave Telehouse East (Backup) as “preferred” until after the advised wholesale maintenance window and review Friday morning. Until this point the “at risk” notice will remain.

UPDATE04 – 27/01/2023 – 10:25

We are continuing to monitor.