Our network monitoring has alerted us to one of our broadband LNS gateways reloading. This has resulted in broadband services connected via this device to disconnect and failover to alternative gateways.
The device has reloaded and returned to service. We are currently investigating the cause internally.
We apologize for any inconvenience this may have caused but do not expect a recurrence.
We will be making some changes to our network on the 19/10/2020 starting from 12:00 noon. These changes are in relation to our LINX peering and no impact is expected to be seen as traffic will be gracefully re-routed via alternative routes prior to work starting.
We will also be taking this opportunity to add additional hardware for future broadband capacity. This will also be non service affecting.
UPDATE 01 – FINAL – 16:50
This work is complete and LINX peering has been restored. No impact was seen to customer traffic.
We are aware of an issue affecting leased lines services delivered via Telehouse North. We are investigating as a matter of urgency. We sorry for any inconvenience being caused.
Services with backup DSL will have been automaticity re-routed.
UPDATE 01 – 12:50
Further investigations have shown this is affecting services delivered from our redundant fibre routes as well. We are continuing to work with our suppliers as the root cause has been identified outside of our network.
UPDATE 02 – 13:02
We are starting to see services recover but services should still be considered at risk.
UPDATE 03 – 14:00
We have had further feedback from our suppliers to advise this has been resolved and they believe this to have been down to a configuration issue there side. We have raised this as concern as this took down multiple fail over links for redundancy.
FINAL – 10:39
We have been advise this issue was cause by human error within our wholesale provider and the failure to follow strict guidelines when undertaking work. Due to the impact this had on ourselves and the loss of our redundant links with this provider, We are undertaking an internal review to ensure we mitigate against this in the future.
We are aware a number of broadband services have been dropping PPP sessions over the past several hours. Initial diagnostics show nothing wrong our side and we have raised this with our suppliers for further investigation.
UPDATE 01 – 14:27
We have received an update to advise there is an issue further upstream and emergency maintenance work is required. Due to the nature of the work, we have been told this will start at 14:30 today. The impact of this will be further session drops while core devices are potentially reloaded carrier side.
We are sorry for the short notice and impact this will have and have requested a RFO already for the incident.
UPDATE 02 – 15:54
We have been advised the work is complete. We are awaiting this to be 100% confirmed.
We are aware our unauthenticated SMTP relay cluster has been subject to relay abuse by a compromised client. Currently SMTP services are suspended on the cluster.
UPDATE 01 – 22:30
SMTP services on the cluster remain suspended while we review. Further updates will be provided on the 29/07/2020
UPDATE 02 – 09:15 – 29/07/2020
After a full review. Due to the age of the platform, End of life support on the OS, extremely low usage levels (less than 0.1%) and the lack of support for enhanced security such as DKIM and DMARC. We have decided to withdraw the platform form service.
For customers who where using the service, We would advise migrating to authenticated SMTP provided via your web hosting provider or signing up with a free relay such as https://www.smtp2go.com/
We understand change is unwelcome, but after review we feel this is in the interest of all who still use the platform to protect your domain and others.
At 08:15 GMT this morning, we were alerted to a number of DSL broadband sessions disconnecting. Initial diagnostics showed there was no fault within our network and this was escalated to our wholesale supplier.
Our wholesale supplier responded to advise a DSL gateway “cr2.th-lon” at 08:15 GMT had dropped a number of sessions however had started to recover at 08:23 GMT. At this time the root cause of the outage is unknown but investigations are continuing. Services should be considered at risk until we ascertain the cause.
UPDATE 01 – 10:50
We have seen a further drop in sessions where sessions have had to re-authenticate. We have requested an update from our supplier to enquire of this is related to the issues seen this morning.
At the end of 2019, Windows 2008 server will become EOL (End Of Life) and cease to be supported by Microsoft. In response to this, we will be retiring the platform which we have had in operation since 2011 and migrating accounts to a Debian based operating system well before this. Unfortunately due to the … Continue reading “03/07/2019 – Web hosting / email retirement and upgrade”
At the end of 2019, Windows 2008 server will become EOL (End Of Life) and cease to be supported by Microsoft. In response to this, we will be retiring the platform which we have had in operation since 2011 and migrating accounts to a Debian based operating system well before this.
Unfortunately due to the age of the operating system and the cross vendor platforms there is no direct migration path between systems and will requires changes on both our side and customer applications. (such as emails programs)
The change does bring around a vast number of new features and security options such as SSL (Secure) email which is now becoming industry standard.
All user accounts will need to be migrated by the 30th Aug 2019 when the platform will be turned off. We are currently migrating accounts on a domain by domain basis due to the manual requirement to transfer account data.
Users that have existing accounts setup as IMAP will simply need to delete there accounts when told to and re-create there them with the new server details provided. While we will make every effort to help and backup data, we are limited in what remote assistance we can provide due to unmanaged nature of the product and would strongly suggest users backup there email to local storage prior to making any changes.
Changes our side to DNS will require time to populate around the internet and some disruption is expected on your migration day. Sadly this cant be avoided but we will do all we can to minimize this.
Affected customers will be contacted directly and you are welcome to contact us to expedite your move.
UPDATE 01 – 04/07/2019
In order to ensure data consistency, we have moved to suspend logins to the old platform and would advise anyone who still has access to no make changes until after you have migrated your account as data may be lost.
UPDATE 02 – 10/07/2019
End users will start to see messages about new accounts being created. This is expected as they are being cloned on to the new platform. It also ensures we have up to date contact information for the migration email to follow
UPDATE 03 – 15/07/2019
It has come to our attention that the formatting on the mass email sent out last night has broken. We have uploaded the original email below.
Dear sirs,
Further to our recent advisory on the retirement and renewal of the existing webhosting & email platform. (Details can be found at https://noc.structuredcommunications.co.uk/?p=1387)
Please find below further details and plans on the account migrations that will affect you.
Due to the more modern file structure and use of secure connections on the new platform. Users are advised against simply updating existing email accounts by changing the server details. For ease, users are advised to create a new separate account on there respective email clients and devices using the new server details at the bottom of this email.
Initially, once you create the duplicate account, the contents of the inbox will be out of date. Upon creation of the new account and once you are ready to migrate please visit http://noc.structuredcommunications.co.uk/EasyHTTP/ and complete the form.
Within 24-48 hours the contents of your mailbox will be synchronised to your new account and will begin downloading on your devices. (This may take time if you have a large inbox) Your old account will then stop working and can be deleted from your email client and devices.
Structured Communications will take steps to preserve your email data however is in no way responsible should data loss occur and advise clients to always keep their own data backups.
If you have your email client configured for POP3 based email, then we would advise you to consult your IT personal as the above will result in data loss. You are welcome to contact us if you have questions.
Due to the unmanaged nature of this product and data protection, Structured Communications will be limited in the assistants it can offer end users in setting up and deleting accounts. We can however refer users to one of our IT partners who can offer remote assistance at a charge.
NEW SERVER DETAILS:
USERNAME: Your full email address
PASSWORD: Your existing password
INCOMING SERVER: plesk.easyhttp.co.uk
PORT: 993
ENCRYPTION METHOD: SSL or TLS
OUTGOING SERVER: plesk.easyhttp.co.uk
PORT: 465
ENCRYPTION METHOD: SSL or TLS
OUTBOUND AUTHENTICATION: YES (use same username and password)
Users with a website and custom DNS entries have been mirrored where possible
We understand these changes can be quite confusing and will do all we can do assist and answer any questions however they are necessary.
We will send out reminders for accounts that’s have not migrated before the August 2019 deadline. Should you take no action then your email accounts will stop working and we may lose the ability to sync your inbox data from the date it was initially cloned.
You can contact us on support@structuredcommunications.co.uk or call 0203 301 4000 (even if you want to confirm this email has been sent by ourselves)
UPDATE 04 – 19/07/2019 – User Guides
UPDATE 05 – 01/08/2019 -Helping Users Migrate
We have been working internally to looks at options of migrating current DNS records. This is possible however without the use of SSL due to the IP miss-match between old and new.
In order to assist users in the short term we are moving records requiring little changes if any to email clients. This will allow users to move over before the cut over date automatically, however changes will be required long term as non-SSL connections are turned off.
Domains where no contact has been made will commence from the 05/08/2019. Users should see no interruption.
UPDATE 06 – 30/08/2019 -Extended Deadline
We have, at our own discretion extended the “switch off” date until the 03/09/2019.
We have observed a small number of broadband services suffering from intermittent connection problems between 19:46 and 20:46 this evening. This issue has been tracked down to one of our wholesale suppliers who suffered a network outage that has since recovered. This has resulted in the affected connections not being able to reach our RADIUS … Continue reading “09/05/2019 20:50 – Broadband Disruption”
We have observed a small number of broadband services suffering from intermittent connection problems between 19:46 and 20:46 this evening.
This issue has been tracked down to one of our wholesale suppliers who suffered a network outage that has since recovered. This has resulted in the affected connections not being able to reach our RADIUS servers for authentication and where simply being terminated on the local BTW RAS with a non route able IP address.
Users who do not have a connection are advised to reboot or power off there router for 20 minutes to recover any stuck sessions.
We apologies for the inconvenience and are awaiting an RFO.
We are aware a small percentage of outbound calls being made on our VoIP network are taking longer than normal to connect. We are aware of this issue and investigating . UPDATE01 – 09:40 Calls have been re-routed over another carrier while we work to understand what is going on with calls on BTIPX UPDATE02 … Continue reading “05/04/2019 – SIP VoIP – Outbound Calls”
We are aware a small percentage of outbound calls being made on our VoIP network are taking longer than normal to connect. We are aware of this issue and investigating .
Calls have been re-routed over another carrier while we work to understand what is going on with calls on BTIPX
Calls are being rejected due to changes made at BT in respect to number formatting. We are in the process of making changes to how we present numbers.
New number scrips have been designed and put in to operation. Outbound calls are now routing correctly once again and this has been marked as closed.
No changes are required on existing customer systems however we will be changing how we configure new systems
We are aware of a unexpected reload of LNS02 which has occurred again. We are currently dealing with our vendor as to the cause and resolution. UPDATE 01 – 22:44 Our vendor has responded providing some technical information from the resulting crash and we have some additional logging capability in place. We are unable to advise any further … Continue reading “01/1/2019 – 22:20 – LNS01 Broadband Gateway”
We are aware of a unexpected reload of LNS02 which has occurred again. We are currently dealing with our vendor as to the cause and resolution.
Our vendor has responded providing some technical information from the resulting crash and we have some additional logging capability in place. We are unable to advise any further at this stage due to ongoing investigations.