Over the coming 3 months we will be retiring our legacy sip gateways known as SIP01, SIP02 and SIP03. Customers who take our managed services will be automatically migrated to our new infrastructure. Customers who manage there own PBXs will need to migrate manually.
You will be contacted in due course if this applies to you.
Our network monitoring has alerted us to multiple circuit failures within our Horsham facility. Initial diagnostics seem to show fibre breaks and we suspect this may be the result of civil contractors. Traffic is flowing across redundant paths in to the building with no loss of primary peering or transit, but should be considered “at risk” due to operating on redundant links.
Ethernet services that terminate in to our Horsham facility will have automatically failed over to backup if purchased.
Faults have been logged with Openreach and we will keep updating this page as we know more.
UPDATE 01 – 12:01
We have seen all our “primary” fibre links recover and service has been restored, however no official update has been provided. We are still awaiting recovery of the other affected fibre links.
UPDATE 02 – 12:10
Openreach engineering teams are on route to our facility.
UPDATE 03 – 14:50
Openreach are on site
UPDATE 04 – 15:00 *FINAL*
All fiber links have been restored. Contractors working on the Openreach network had trapped one of our fibre tubes running that route and caused bends on the groups of affected fibre to the point light was unable to pass.
Tubing and fibres have been re run in the AG Node by Openreach and service has been restored.
We are aware some outbound calls are failing and people are hearing a pre-recorded message advising of a service suspension. This message is not being generated from our network and has been traced further upstream to a 3rd party carrier. We are working with our carriers to identify the root cause and updates will be posted shortly.
UPDATE01 – 16:25
This has been resolved. We apologise for any inconvenience caused.
We are observing odd behaviour on LNS04. The device is operational, however we are observing latency on some network operations. We have spoken to our vendor who has advised we apply a software patch. This will result in the device reloading. Circuits connected via this device will experience a PPP drop but will re-route to other gateways.
UPDATE01 – 13:15
We have taken this gateway out of service for new / reconnections connections. If you are having issues then power cycling your router will result in a connection to another non affected gateway.
UPDATE02 – FINAL
The gateway is now back in operation.
We are aware of a peering issue within LINX that is affecting all sessions and traffic via LON1. Traffic has re-routed via alternative routes.
Our network monitoring has alerted us to one of our broadband LNS gateways reloading. This has resulted in broadband services connected via this device to disconnect and failover to alternative gateways.
The device has reloaded and returned to service. We are currently investigating the cause internally.
We apologize for any inconvenience this may have caused but do not expect a recurrence.
We will be making some changes to our network on the 19/10/2020 starting from 12:00 noon. These changes are in relation to our LINX peering and no impact is expected to be seen as traffic will be gracefully re-routed via alternative routes prior to work starting.
We will also be taking this opportunity to add additional hardware for future broadband capacity. This will also be non service affecting.
UPDATE 01 – FINAL – 16:50
This work is complete and LINX peering has been restored. No impact was seen to customer traffic.
We are aware of an issue affecting leased lines services delivered via Telehouse North. We are investigating as a matter of urgency. We sorry for any inconvenience being caused.
Services with backup DSL will have been automaticity re-routed.
UPDATE 01 – 12:50
Further investigations have shown this is affecting services delivered from our redundant fibre routes as well. We are continuing to work with our suppliers as the root cause has been identified outside of our network.
UPDATE 02 – 13:02
We are starting to see services recover but services should still be considered at risk.
UPDATE 03 – 14:00
We have had further feedback from our suppliers to advise this has been resolved and they believe this to have been down to a configuration issue there side. We have raised this as concern as this took down multiple fail over links for redundancy.
FINAL – 10:39
We have been advise this issue was cause by human error within our wholesale provider and the failure to follow strict guidelines when undertaking work. Due to the impact this had on ourselves and the loss of our redundant links with this provider, We are undertaking an internal review to ensure we mitigate against this in the future.
We are aware a number of broadband services have been dropping PPP sessions over the past several hours. Initial diagnostics show nothing wrong our side and we have raised this with our suppliers for further investigation.
UPDATE 01 – 14:27
We have received an update to advise there is an issue further upstream and emergency maintenance work is required. Due to the nature of the work, we have been told this will start at 14:30 today. The impact of this will be further session drops while core devices are potentially reloaded carrier side.
We are sorry for the short notice and impact this will have and have requested a RFO already for the incident.
UPDATE 02 – 15:54
We have been advised the work is complete. We are awaiting this to be 100% confirmed.
We are aware our unauthenticated SMTP relay cluster has been subject to relay abuse by a compromised client. Currently SMTP services are suspended on the cluster.
UPDATE 01 – 22:30
SMTP services on the cluster remain suspended while we review. Further updates will be provided on the 29/07/2020
UPDATE 02 – 09:15 – 29/07/2020
After a full review. Due to the age of the platform, End of life support on the OS, extremely low usage levels (less than 0.1%) and the lack of support for enhanced security such as DKIM and DMARC. We have decided to withdraw the platform form service.
For customers who where using the service, We would advise migrating to authenticated SMTP provided via your web hosting provider or signing up with a free relay such as https://www.smtp2go.com/
We understand change is unwelcome, but after review we feel this is in the interest of all who still use the platform to protect your domain and others.