Our network monitoring has flagged high CPU usage on LNS01 that is starting to affect its operation. We are undertaking an emergency reboot to prevent it crashing completely. This will drop active sessions and force them on to other gateways.
UPDATE01 – 21:35
The reboot is complete. Services transferred to redundant gateways as expected. LNS01 is now back in operation. If you do not have service please power down your router for 20 minutes.
We apologise for any inconvenience caused.
Our network monitoring has alerted us to a small number of metro based layer 2 Vodafone circuits that have no service. This has also had an impact on broadband as well as services in to our Horsham based facility.
This was raised with the relevant teams last night and fibre engineers are already on site working on suspected damaged fibre to our selves and a number of other providers.
We have seen some services restore. Services with backup if affected will be operating on backup and wont be affected.
We will post further updates as they become available.
UPDATE01 – 11:02
Engineers have located the break and this will now progress to splicing.
UPDATE02 – 12:45
Engineers are continuing to work on splicing the damaged cable with spares.
UPDATE03 – 13:00
We are starting to see some services restore.
UPDATE04 – 14:25
As part of ongoing investigation into this issue, Vodafone are sending out additional engineering resources to their data centre.
Vodafone believe the break to be located 43 meters into the fibre handoff, which they are working on to resume full service.
UPDATE05 – 15:30
Engineers at the data centre are continuing to work with Vodafone fibre teams and resolve the second identified fibre break.
UPDATE05 – 16:45
Vodafone fibre engineers continue to examine the length of the fibre for further issues impacting service.
UPDATE06 – 17:55
Vodafone engineers remain on site tracing the underlying infrastructure so that a permanent fix can be put in place for all affected services. We continue to work with the carrier to expedite the resolution with regular communication.
Currently the agreed next stage is for engineers to work to determining the exact location for the fibre fault.
UPDATE07 – 18:45
All services have restored and we are awaiting a RFO. This will be provided on request.