EasyHTTP – 09/09/2022 – 12:30

We have made some changes to our hosted email platform in respect to DNSBL validation for SPAM and listings to the spam services running on the server.

This is in response to some office365 delivery issues that took place last month.

We don’t expect any mail delivery issues as a result of this and changes have been tested with known problematic mailboxes but any NDRs (none delivery reports) should be immediately reported to us via support.

Ethernet EAD – 22/07/2022 – 11:30

We are aware there are a small number of layer 2 ethernet services currently down at the moment. Internal investigations show this to be a upstream supplier issue and we are currently engaging with them to locate the fault.

UPDATE01: 11:40

We are seeing services down from BT Wholesale, Openreach EAD Direct, TTB so we suspect this may be a common POP failure between layer2 providers landing circuits in London. Customers with backup will have automatically kicked in and re-routed.

UPDATE02: 12:00

We are still awaiting for an official update from our layer2 provider as to the root cause of this issue but they have advised they are seeing circuits down with a large number of calls on hold to the service desks.

We have already escalated to our management contacts to push for information so we can provide detailed updates.

UPDATE03: 12:25

Our Layer2 provider has now declared a “Major Incident”

They have advised this appears to be related to a “DNS issue” but we have disputed that. We are continuing to chase for updates. We apologise to affected customers and the inconvenience this is causing.

UPDATE04: 12:41

Our layer2 provider has now advised of a major internal core network problem affecting more than just layer2 services. This is currently affecting less than 10% of our overall EAD circuits with this layer2 provider and services delivered via other layer2 partners are unaffected.

We have been advised internal teams are working to identify the root cause and issue a fix. We again apologise to affected customers and the inconvenience this is causing.

UPDATE05: 12:55

Following further pressure to our account manager we have been advised this is affecting layer2 services being delivered to us via a core device located in there network at Interxion London with multiple service tunnels flapping.

We have been advised there will be a further update by 13:30

UPDATE06: 13:40

A further update has been provided to advise they are still working on why network tunnels on this device are “flapping” They have advised a further update by 15:30 but we will keep pushing for information and a ETA on service resolution.

UPDATE07: 15:05

Our network monitoring has shown our end points are reachable again on the affected circuit’s. We have no been provided an official clear yet and services should still be classed as at risk.

UPDATE08: 17:00

Our layer2 provider has advised they have re-routed around the affected device and are currently working with the hardware vendor to establish why the device has failed in the way it has. We suspect there may be a short outage at a later date once services are re-routed back via this device but we will advise at the time.

HOR-DC – *at risk*

Our network monitoring has alerted us to multiple circuit failures within our Horsham facility. Initial diagnostics seem to show fibre breaks and we suspect this may be the result of civil contractors. Traffic is flowing across redundant paths in to the building with no loss of primary peering or transit, but should be considered “at risk” due to operating on redundant links.

Ethernet services that terminate in to our Horsham facility will have automatically failed over to backup if purchased.

Faults have been logged with Openreach and we will keep updating this page as we know more.

UPDATE 01 – 12:01

We have seen all our “primary” fibre links recover and service has been restored, however no official update has been provided. We are still awaiting recovery of the other affected fibre links.

UPDATE 02 – 12:10

Openreach engineering teams are on route to our facility.

UPDATE 03 – 14:50

Openreach are on site

UPDATE 04 – 15:00 *FINAL*

All fiber links have been restored. Contractors working on the Openreach network had trapped one of our fibre tubes running that route and caused bends on the groups of affected fibre to the point light was unable to pass.

Tubing and fibres have been re run in the AG Node by Openreach and service has been restored.

Broadband – Maintenance (26/11/2021 – 22:00)

We will be upgrading firmware across some of our LNS gateways tonight to correct a RADIUS issue on the broadband platform.

The work will require a reboot of affected LNS once the upgrade is complete. This will take around 30 seconds per LNS. Connected circuits will see a PPP restart while this takes place and may connect back to another gateway.

UPDATE 01 – 22:05

This work is complete.

Broadband – 11:55 – 13/10/2021

We are aware of a large drop on broadband services across the UK. We are currently investigating as a matter of urgency.

UPDATE01 – 12:05

Internal investigations have concluded the fault is not within our network and we are working with our wholesale providers.

UPDATE02 – 13:05

We are aware some users are seeing a BT Wholesale landing page / getting private WAN IPs. This is where connections are not routing to our network. We are continuing to work with our suppliers to find the root cause but we have see connections restore. Anyone without a connection, please reboot your router by removing power for 5 minutes.

UPDATE03 – 13:38

We have had an update from wholesale to advise the issue appears to be with their RADIUS PROXY servers that relay authentication credentials to our network. This would account for the BT Wholesale holding page as requests are not getting to us.

We have asked for an ETA. But would ask end users reboot there router to see if there connection restores.

UPDATE 04 – 14:30

We have been advised the fix has been applied and we are seeing a large number of circuits reconnect. We are awaiting for an RFO and will publish when this is made available. We apologise for any inconvenience caused.

Any users without connection are advised to power down there hardware for 20 minutes.

Voice Calls – 13/09/2021 – 15:45

We are aware some outbound calls are failing and people are hearing a pre-recorded message advising of a service suspension. This message is not being generated from our network and has been traced further upstream to a 3rd party carrier. We are working with our carriers to identify the root cause and updates will be posted shortly.

UPDATE01 – 16:25

This has been resolved. We apologise for any inconvenience caused.

Broadband – Emergency Maintenance (31/08/2021 – 22:00)

We have been advised by one of our wholesale providers that they will be undertaking emergency maintiance on the following exchanges that may result in an outage of up to 30 minutes for DSL circuits.

If you have still have no service after the work, Please power down your hardware for 20 minutes and power back on.

We apologise for any inconvenience caused.

Exchange areas affected (including ADSL2+, FTTC, SoGEA, FTTP, G.Fast)

Upper Holloway

Stratford On Avon

Great Yarmouth Town

Norwich City

Arkwright

Leicester Central

Nottingham Longbow

Newark

Rushden

Cupar Crossgates

Penrith

Todmorden

Wigan

Poplar

Deptford

Ewell

Greenwich

Orpington

Redhill

Skelmersdale

Ealing

Golders Green

Hayes North

Hendon

Twickenham

Didsbury

Stockport

Gravesend

Crawley

Hove

Letchworth

Bedminster

Downend

Frome Keyford

Gloucester

North

South

West

Bridgend

Merthyr Tydfil

Whitchurch

Godalming

Haslemere

Henley-on-thames

Newbury

Reading South

Lords

Maida Vale

Bromsgrove

St Peters

Chester/south

Ross On Wye

South Kensington

Western

Barnstaple

Exeter

Falmouth

Newquay

Redruth

St Austell

Taunton

Truro