Category: Emergency Work

Incident: sip03.easyipt.co.uk – 29/07/2024 – 12:20

We are aware SIP03 is suffering from issues following an emergency reboot. We are currently working on this as a matter of urgency.

UPDATE01 – 12:42

We are currently awaiting the database to recover. As a precaution, we are also restoring a backup of the platform to another host.

UPDATE02 – 13:10

A restore of the platform has resolved the issue. We are sorry for the disruption.

UPDATE03 – 13:15

After review we will be migrating this server to a new version of the software.

UPDATE04 – 17:37

A new software build has been deployed and accounts migrated over. A further window will be raised for the migration works.

UPDATE05 – 02/08/2024 – 21:00

SIP03 accounts have been migrated to the new platform. Test calls have been completed successfully.

Horsham DC – Power Maintenance – 21/06/2024 – 14:00 – 18:00

UK Power Networks have advised us they will be undertaking work on our main incoming power feed for our Horsham Data Center between 14:00 and 18:00 on Friday the 21st June. We will be running on backup power for the duration of the works.

Both UKPN and Structured engineers will be on site for the duration. No impact is expected but the site will be classed as “at risk”

Load tests will be completed prior to the works to ensure a smooth transfer.

UPDATE 01 – 14:29

The site is now on backup power.

UPDATE 02 – 16:00

This work is now complete and the facility is now back on grid power with both A and B feeds being back in service. No service disruption was experienced.

Broadband – 27/10/2022 – IPv6

We are aware of an IPv6 issue on the network following on from a firmware upgrade within our core last night.

We have been working with the hardware vendor to resolve the issue but while this is ongoing , we have been reverting back to a previous version of firmware and moving connections between gateways.

Some users will experience a graceful PPP drop of around 5 seconds while there connection re-authenticates.

We do apologise for any inconvenicance

UPDATE01

LNS02, LNS03, LNS04 have been reverted and IPv6 connectivity has been restored.

LNS01 has been isolated for testing. Anyone experiencing slow DNS lookups or applications now loading are advised to reboot there router which will be routed to one of the other gateways with the fix applied.

UPDATE02

Further issues where identified with IPv6 within our core network (Broadband facing) Work to resolve this has now completed and IPv6 should now be fully operational again across all gateways.

Broadband – Emergency Maintenance (31/08/2021 – 22:00)

We have been advised by one of our wholesale providers that they will be undertaking emergency maintiance on the following exchanges that may result in an outage of up to 30 minutes for DSL circuits.

If you have still have no service after the work, Please power down your hardware for 20 minutes and power back on.

We apologise for any inconvenience caused.

Exchange areas affected (including ADSL2+, FTTC, SoGEA, FTTP, G.Fast)

Upper Holloway

Stratford On Avon

Great Yarmouth Town

Norwich City

Arkwright

Leicester Central

Nottingham Longbow

Newark

Rushden

Cupar Crossgates

Penrith

Todmorden

Wigan

Poplar

Deptford

Ewell

Greenwich

Orpington

Redhill

Skelmersdale

Ealing

Golders Green

Hayes North

Hendon

Twickenham

Didsbury

Stockport

Gravesend

Crawley

Hove

Letchworth

Bedminster

Downend

Frome Keyford

Gloucester

North

South

West

Bridgend

Merthyr Tydfil

Whitchurch

Godalming

Haslemere

Henley-on-thames

Newbury

Reading South

Lords

Maida Vale

Bromsgrove

St Peters

Chester/south

Ross On Wye

South Kensington

Western

Barnstaple

Exeter

Falmouth

Newquay

Redruth

St Austell

Taunton

Truro

Emergency Works – Broadband – 12/05/2021 – 22:00

We are observing odd behaviour on LNS04. The device is operational, however we are observing latency on some network operations. We have spoken to our vendor who has advised we apply a software patch. This will result in the device reloading. Circuits connected via this device will experience a PPP drop but will re-route to other gateways.

UPDATE01 – 13:15

We have taken this gateway out of service for new / reconnections connections. If you are having issues then power cycling your router will result in a connection to another non affected gateway.

UPDATE02 – FINAL

The gateway is now back in operation.

Emergency Works 14/04/2021 – 02:00 (Broadband)

We have been made aware by our suppler that they will be making changes to our 10GB interconnects at Telehouse North and East at 02:00 on the 14/04/2021 to resolve the low level packet loss being seen on some broadband services and non critical traffic types. This work should not be service affecting however one of the access switches we are connected to may reload as a result due to an unrelated issue there side. Should this occur, services will fail over to redundant paths but users will experience a PPP reconnect.

An update will be posted tomorrow morning to confirm closure.

Emergency Works 30/03/2021

An engineer is on route to Telehouse to complete a reboot of our 2 London core devices. Works are due to start from 07:30am today. This will be done in succession pending a successful reboot of the first device and to allow services where possible to re-route across the network. Leased line circuits without backup will see disruption when the associated core reloads.

We do apologise for the short notice.

UPDATE01 – 07:35

This work is about to start

UPDATE02 – 08:30

This work is complete.

14/09/2020 22:00 – Broadband Network

Due to a unexpected reload of a broadband gateway on our network this afternoon, we are seeing a traffic imbalance on part of our broadband network.

We will be taking steps to disconnect sessions gratefully and re balance the affected gateways.

End users will see a PPP reconnect taking approximately 5-20 seconds. In the rare event your connection does not restore. You will need to power OFF and ON your router.

UPDATE01 – 22:10

This work is now complete.

16/04/2020 23:00 – Broadband Maintenance

We will be making some changes to our broadband network tonight in order to isolate 2 upstream gateways we suspect of causing additional latency to circuits routed via them.

This will cause existing connection via these gateways to drop and reconnect. Due to the nature of the change this can take up-to 20 minutes.

UPDATE 01 – 23:03

This work is about to start.

UPDATE 02 – 23:06

Tunnels have been terminated and traffic is starting to move across to other gateways.

UPDATE 03 – 23:28

We have seen an issue with the L2TP control messages not being accepted by the upstream gateways and releasing circuits to other gateways. We have therefore had to revert part of the configuration. Further works will be required at a later date.

Emergency Work – 23/11/2018 – 21:30 – LNS01 Broadband Gateway

Our network monitoring has flagged high CPU usage on LNS01 that is starting to affect its operation. We are undertaking an emergency reboot to prevent it crashing completely. This will drop active sessions and force them on to other gateways. UPDATE01 – 21:35 The reboot is complete. Services transferred to redundant gateways as expected. LNS01 … Continue reading “Emergency Work – 23/11/2018 – 21:30 – LNS01 Broadband Gateway”

Our network monitoring has flagged high CPU usage on LNS01 that is starting to affect its operation. We are undertaking an emergency reboot to prevent it crashing completely. This will drop active sessions and force them on to other gateways.

UPDATE01 – 21:35
The reboot is complete. Services transferred to redundant gateways as expected. LNS01 is now back in operation. If you do not have service please power down your router for 20 minutes.

We apologise for any inconvenience caused.