We are aware of an issue with outbound SMTP connections being refused and requesting an SSL connection. As we don’t support SSL these are also being refused. We are taking the service offline at 17:30 to perform a software upgrade to the SMTP service. *UPDATE 01 @ 17:53** The upgrade is now complete which we … Continue reading “SMTP – Structured Communications Server Array 1”
We are aware of an issue with outbound SMTP connections being refused and requesting an SSL connection. As we don’t support SSL these are also being refused.
We are taking the service offline at 17:30 to perform a software upgrade to the SMTP service.
*UPDATE 01 @ 17:53**
The upgrade is now complete which we hope will resolve the above issue, However we are also taking this Opportunity to update a few other components.
*UPDATE 01 @ 19:50**
This work is now complete.
We are aware of an issue with one of our wholesale interconnects with Virgin Media, which is causing packet loss for a small number of circuits. An engineer is on route to investigate this link further. We are expecting to have further updates at approximately 13:00. *UPDATE 01 @ 13:38** Service has now been restored … Continue reading “Packet Loss on Virgin Media Circuits”
We are aware of an issue with one of our wholesale interconnects with Virgin Media, which is causing packet loss for a small number of circuits.
An engineer is on route to investigate this link further. We are expecting to have further updates at approximately 13:00.
*UPDATE 01 @ 13:38**
Service has now been restored following the fiber link being cleaned and re-seated.
We will be upgrading our bonded platform from version 2.19 to 2.32. During this time your router may disconnect a few times, and there may be a period of instability. We expect no more than 30-60 seconds of downtime during this window. UPDATE 01 @ 23:30 This upgrade is now complete and we are monitoring … Continue reading “NOC: Planned Maintenance – 27/07/2013 – 22:00 till 23:30 – EasyBOND”
We will be upgrading our bonded platform from version 2.19 to 2.32. During this time your router may disconnect a few times, and there may be a period of instability. We expect no more than 30-60 seconds of downtime during this window.
UPDATE 01 @ 23:30
This upgrade is now complete and we are monitoring the service.
In line with our previous upgrade work to LON01 we are now ready to “bring up” our BGP sessions to our new and existingt transit providers. During this time there may be a period of routing instability, however no service outages are expected. **UPDATE 01 @ 18:44 – 24/07/2013** Engineers are now onsite and will … Continue reading “NOC: Planned Maintenance – 24/07/2013 – 22:00 till 25/07/2013 – 02:00 – LON01 Goswell Road”
In line with our previous upgrade work to LON01 we are now ready to “bring up” our BGP sessions to our new and existingt transit providers. During this time there may be a period of routing instability, however no service outages are expected.
**UPDATE 01 @ 18:44 – 24/07/2013**
Engineers are now onsite and will start to prepair for this work.
**UPDATE 02 @ 21:54 – 24/07/2013**
Engineers are now ready to start this work.
**UPDATE 03 @ 00:54 – 25/07/2013**
Engineers have now completed this work and our BGP sessions to our new peers are live, we are now solely announcing our IP space from our ASN (AS199283) Fail-over between links has been tested and completed successfully. Some routing instability maybe seen over the next 12-48 hours while BGP tables update to reflect our ASN, however should be minimal.
No outages where experienced durning this window, other than testing which lasted 15 seconds.
UPDATE 04 @ 16:33 – 25/07/2013 FINAL
90% of the world IPv4 registers now see our ASN and we are happy that the remaining will update over the next 12 hours and be reflected via RIPE.
We are aware of an issue with the SMTP service on Array1 and are looking in to the problem. A temporary service suspension has been placed on inbound and outbound SMTP and IMAP services. Inbound Emails are being queued. We will update as we know more. **UPDATE 01 @ 13:33** Upon investigation, we found the … Continue reading “SMTP – Structured Communications Server Array 1”
We are aware of an issue with the SMTP service on Array1 and are looking in to the problem.
A temporary service suspension has been placed on inbound and outbound SMTP and IMAP services.
Inbound Emails are being queued. We will update as we know more.
**UPDATE 01 @ 13:33**
Upon investigation, we found the SMTP connectors to be at 30% load processing over 2,000 emails a second. We have disabled the SMTP connector to enable us to flush the queue. We are monitoring the service however it is now back online.
**UPDATE 02 @ 14:01**
We have observed further high usage on the SMTP connectors and are now looking in to the originating traffic.
**UPDATE 03 @ 14:40**
Traffic analysis and a review of the SMTP logs flagged that an email account had been compromised. This account has been disabled. Email flow and services have returned to normal. We will however contunie to monitor.
**UPDATE 04 @ 16:51**
The platform has remained stable, however we have noted a few temporary blacklisting’s of the servers external facing IPs. We have taken action to clear these blocks from mainstream listings, however there are still some temporary bans in place that we are unable to clear and will automatically revoke within 24 hours. This may affect the sending of emails to some systems who reference to these databases.
**UPDATE 05 @ 23:41**
The platform has continued to remain stable and will be closing this incident down in the next 24 hours. In light of the issues seen today we have implemented further security checks upon connecting to help prevent this happerning again.
Adational monitoring will also be out in place to alert on higher procress / service demand. This will alow us to respond faster to any potential future problems.
**UPDATE 06 FINAL @ 16:53 – 27/05**
The SMTP platform has been removed from all visible blacklists. This incident is now closed.
We have been advised by our upstream carrier for VoIP calls they will be conducting maintenance during the above window. This will affect call flow for a short duration. Below is the official notification sent to us: Will be migrating some infrastructure configuration from a core router to a higher capacity device on Saturday 25 … Continue reading “UPSTREAM – Maintenance Window Saturday 25 May 2013 0400-0500 UTC”
We have been advised by our upstream carrier for VoIP calls they will be conducting maintenance during the above window. This will affect call flow for a short duration. Below is the official notification sent to us:
Will be migrating some infrastructure configuration from a core router to a higher capacity device on Saturday 25 May 2013. The outage window will be between 0400 and 0500 UTC. The outage duration is not expected to be more than 5 minutes. During this time calls in progress may be cut off. This activity is part of our ongoing capacity improvement programme. We apologise for any inconvenience that may be caused.
We have been advised that the un-planned outage on Friday the 17/05/2013 between 14:00 – 15:00 lasting approximately 5 minutes was caused by a power loss on BTs transmission network. Report from 02 Wholesale: We had a problem with a link between GS2 and one of our switch sites which took out some sessions going … Continue reading “02 Wholesale – Friday 17/05/2013 – Unplanned Outage”
We have been advised that the un-planned outage on Friday the 17/05/2013 between 14:00 – 15:00 lasting approximately 5 minutes was caused by a power loss on BTs transmission network.
Report from 02 Wholesale:
We had a problem with a link between GS2 and one of our switch sites which took out some sessions going through that site. The sessions were immediately re-established on another link, but you would have seen a drop and then re-establishment. It was caused by a power outage on BT’s transmission. The link is being closely monitored and we haven’t seen any further problems.
Our software vendor for MailEnable (who provide our SMTP server software) has alerted us to a vulnerability in several releases of their software. We will take immediate action and install these patches. As a result this will stop email flow for around 5 minutes while we take the service off line to perform the upgrade. … Continue reading “Emergency Upgrade – SMTP (Array01 EasyHTTP)”
Our software vendor for MailEnable (who provide our SMTP server software) has alerted us to a vulnerability in several releases of their software. We will take immediate action and install these patches.
As a result this will stop email flow for around 5 minutes while we take the service off line to perform the upgrade.
This is for your own protection and we will advise once complete. We apologise for any inconvenience this may cause.
***UPDATE***
This work is now complete and the platform version has been updated to 7.07.
***UPDATE*** 20/05/2013 This work has been cancelled and will be rescheduled for a later date. ————————– Please be advised there is Planned Maintenance work commencing on Friday 24th of May 2013 at our Goswell Road (LON01) datacentre. (Please see attached notification for complete details) The work being undertaken is a required upgrade to our Cisco … Continue reading “Planned Maintenance- 24/05/2013 – LON01 (Goswell Road)”
***UPDATE*** 20/05/2013
This work has been cancelled and will be rescheduled for a later date.
————————–
Please be advised there is Planned Maintenance work commencing on Friday 24th of May 2013 at our Goswell Road (LON01) datacentre. (Please see attached notification for complete details)
The work being undertaken is a required upgrade to our Cisco core allowing us to implement BGP (Border Gateway Patrol) resulting in Structured Communications being able to announce our own IP space via our own AS number (autonomous systems) This upgrade will allow us to directly peer with a range of new upstream providers, greatly improving our network resilience and allowing us to create a geographically diverse network.
If you have any further questions or queries regarding this activity then please e-mail support@structuredcommunications.co.uk or alternatively please call 0203 301 4000 quoting reference #PM006.
Attachment – Planned Maintenance Advisory 24-05-2013
We are currently investigating reports of packet loss across the LLU Network. O2 have provided us with the following update: We are currently investigating a possible problem within our core network, which may cause packet loss and intermittent sessions for your customers. We will let you know as soon as we have any further information. … Continue reading “Possible Packet Loss: LLU Network – 07/05/2013”
We are currently investigating reports of packet loss across the LLU Network. O2 have provided us with the following update:
We are currently investigating a possible problem within our core network, which may cause packet loss and intermittent sessions for your customers. We will let you know as soon as we have any further information.
The above issue may cause issue with the quality of VoIP calls, we will provide more updates as they become available.
Apologies for any inconvenience caused.
———————————————
UPDATE 2200 07/05/2013
We have received the below update from O2.
The core links are back in service. However, we are still testing the connections and increased latency might be affecting the broadband service in Sheffield and Northampton areas.
We believe the packet loss issue to now be resolved, we will continue to monitor the situation.
Again please accept our apologies for any inconvenience caused today.