Author: Adam Clark (Network Operations)

Broadband – Maintenance (26/11/2021 – 22:00)

We will be upgrading firmware across some of our LNS gateways tonight to correct a RADIUS issue on the broadband platform.

The work will require a reboot of affected LNS once the upgrade is complete. This will take around 30 seconds per LNS. Connected circuits will see a PPP restart while this takes place and may connect back to another gateway.

UPDATE 01 – 22:05

This work is complete.

Broadband – 11:55 – 13/10/2021

We are aware of a large drop on broadband services across the UK. We are currently investigating as a matter of urgency.

UPDATE01 – 12:05

Internal investigations have concluded the fault is not within our network and we are working with our wholesale providers.

UPDATE02 – 13:05

We are aware some users are seeing a BT Wholesale landing page / getting private WAN IPs. This is where connections are not routing to our network. We are continuing to work with our suppliers to find the root cause but we have see connections restore. Anyone without a connection, please reboot your router by removing power for 5 minutes.

UPDATE03 – 13:38

We have had an update from wholesale to advise the issue appears to be with their RADIUS PROXY servers that relay authentication credentials to our network. This would account for the BT Wholesale holding page as requests are not getting to us.

We have asked for an ETA. But would ask end users reboot there router to see if there connection restores.

UPDATE 04 – 14:30

We have been advised the fix has been applied and we are seeing a large number of circuits reconnect. We are awaiting for an RFO and will publish when this is made available. We apologise for any inconvenience caused.

Any users without connection are advised to power down there hardware for 20 minutes.

Voice Calls – 13/09/2021 – 15:45

We are aware some outbound calls are failing and people are hearing a pre-recorded message advising of a service suspension. This message is not being generated from our network and has been traced further upstream to a 3rd party carrier. We are working with our carriers to identify the root cause and updates will be posted shortly.

UPDATE01 – 16:25

This has been resolved. We apologise for any inconvenience caused.

Broadband – Emergency Maintenance (31/08/2021 – 22:00)

We have been advised by one of our wholesale providers that they will be undertaking emergency maintiance on the following exchanges that may result in an outage of up to 30 minutes for DSL circuits.

If you have still have no service after the work, Please power down your hardware for 20 minutes and power back on.

We apologise for any inconvenience caused.

Exchange areas affected (including ADSL2+, FTTC, SoGEA, FTTP, G.Fast)

Upper Holloway

Stratford On Avon

Great Yarmouth Town

Norwich City

Arkwright

Leicester Central

Nottingham Longbow

Newark

Rushden

Cupar Crossgates

Penrith

Todmorden

Wigan

Poplar

Deptford

Ewell

Greenwich

Orpington

Redhill

Skelmersdale

Ealing

Golders Green

Hayes North

Hendon

Twickenham

Didsbury

Stockport

Gravesend

Crawley

Hove

Letchworth

Bedminster

Downend

Frome Keyford

Gloucester

North

South

West

Bridgend

Merthyr Tydfil

Whitchurch

Godalming

Haslemere

Henley-on-thames

Newbury

Reading South

Lords

Maida Vale

Bromsgrove

St Peters

Chester/south

Ross On Wye

South Kensington

Western

Barnstaple

Exeter

Falmouth

Newquay

Redruth

St Austell

Taunton

Truro

Planned Maintenance – Horsham DC – 30/07/2021 – 12:00 – 13:00

We are observing high levels off optical light loss on one of our backhaul links between our Horsham and London Cores. We have a window scheduled with Openreach between 12:00 & 13:00 on the 20th to clean (and replace) if required optics and cables at the exchange.

No outages are expected as traffic will flow across redundant paths but the works require an “at risk” notification of facility redundancy levels due to the work being undertaken.

Broadband Planned Maintenance – 26/07/2021 – 29/07/2021 (23:00 – 06:00)

We have been informed by one of our wholesale providers that they will be undertaking capacity upgrades on there network that affect several interconnects we provide broadband services via. We expect our redundant links to take over when the works commence but services should be considered “at risk” and may see PPP reconnections / disconnections.

We apologise for any disruption this may cause.

Emergency Works – Broadband – 12/05/2021 – 22:00

We are observing odd behaviour on LNS04. The device is operational, however we are observing latency on some network operations. We have spoken to our vendor who has advised we apply a software patch. This will result in the device reloading. Circuits connected via this device will experience a PPP drop but will re-route to other gateways.

UPDATE01 – 13:15

We have taken this gateway out of service for new / reconnections connections. If you are having issues then power cycling your router will result in a connection to another non affected gateway.

UPDATE02 – FINAL

The gateway is now back in operation.

Emergency Works 14/04/2021 – 02:00 (Broadband)

We have been made aware by our suppler that they will be making changes to our 10GB interconnects at Telehouse North and East at 02:00 on the 14/04/2021 to resolve the low level packet loss being seen on some broadband services and non critical traffic types. This work should not be service affecting however one of the access switches we are connected to may reload as a result due to an unrelated issue there side. Should this occur, services will fail over to redundant paths but users will experience a PPP reconnect.

An update will be posted tomorrow morning to confirm closure.

Emergency Works 30/03/2021

An engineer is on route to Telehouse to complete a reboot of our 2 London core devices. Works are due to start from 07:30am today. This will be done in succession pending a successful reboot of the first device and to allow services where possible to re-route across the network. Leased line circuits without backup will see disruption when the associated core reloads.

We do apologise for the short notice.

UPDATE01 – 07:35

This work is about to start

UPDATE02 – 08:30

This work is complete.