Category: Incidents

We are aware of an issue with one of our carriers where SIP registrations are being refused resulting in a loss of service. We feel this is a repeat of a recent issue experienced with this Carrier and are in discussions for clarification.

We will provide updates as they become available.

UPDATE01 @ 15:19

We are seeing sessions starting to reconnect and service is being restored. We apologise for any inconvenience and will be taking this up with the Carrier and obtaining an outage report which will be provided once available

Between 22:07 22:55 we saw a loss of connectivity to EasyHTTP. An MOR will be available shortly.

UPDATE 01 @ 23:44
MOR is now avaliable. Major Incident Report – EasyHTTP Outage 08-10-13

Between 19:20 & 20:45 we saw a DNS failure on ns01.r00.lon01.structuredcommunications.co.uk and ns02.r00.lon01.structuredcommunications.co.uk. This DNS failure lead to the failure of the SMTP service on EasyHTTP in that is was unable to lookup domains for email delivery. Emails where queued on the outbound SMTP connector and delivered once the issue was resolved at 20:41.

Looking at the logs prior to the above, its possible there may have been small period where this was intermittant resulting in Delivery delays.

The 2 listed name servers remain offline and DNS traffic has been diverted to ns01.r01.lon01.structuredcommunications.co.uk & ns02.r01.lon01.structuredcommunications.co.uk

We will advise when both the servers located in R00 are back in service.

We are aware of an outage on our Hosted EasyIPT services. Engineers are currently assigned remotely and we have dispatched engineers to the data centre.

Update01**Final Resolved** @ 21:37

Engineers attended LON01 and arrived on site at 17:01 where the server was power cycled restoring service at 17:05. Handsets then started to re-register with individual sites coming back online.

The service has remained stable.

An outage report will be made available shortly.

We are aware of an issue with the SMTP service on Array1 and are looking in to the problem.

A temporary service suspension has been placed on inbound and outbound SMTP and IMAP services.

Inbound Emails are being queued. We will update as we know more.

**UPDATE 01 @ 13:33**

Upon investigation, we found the SMTP connectors to be at 30% load processing over 2,000 emails a second. We have disabled the SMTP connector to enable us to flush the queue. We are monitoring the service however it is now back online.

**UPDATE 02 @ 14:01**

We have observed further high usage on the SMTP connectors and are now looking in to the originating traffic.

**UPDATE 03 @ 14:40**

Traffic analysis and a review of the SMTP logs flagged that an email account had been compromised. This account has been disabled. Email flow and services have returned to normal. We will however contunie to monitor.

**UPDATE 04 @ 16:51**

The platform has remained stable, however we have noted a few temporary blacklisting’s of the servers external facing IPs. We have taken action to clear these blocks from mainstream listings, however there are still some temporary bans in place that we are unable to clear and will automatically revoke within 24 hours. This may affect the sending of emails to some systems who reference to these databases.

**UPDATE 05 @ 23:41**

The platform has continued to remain stable and will be closing this incident down in the next 24 hours. In light of the issues seen today we have implemented further security checks upon connecting to help prevent this happerning again.

Adational monitoring will also be out in place to alert on higher procress / service demand. This will alow us to respond faster to any potential future problems.

**UPDATE 06 FINAL @ 16:53 – 27/05**

The SMTP platform has been removed from all visible blacklists. This incident is now closed.

We have been advised that the un-planned outage on Friday the 17/05/2013 between 14:00 – 15:00 lasting approximately 5 minutes was caused by a power loss on BTs transmission network.

Report from 02 Wholesale:

We had a problem with a link between GS2 and one of our switch sites which took out some sessions going through that site. The sessions were immediately re-established on another link, but you would have seen a drop and then re-establishment. It was caused by a power outage on BT’s transmission. The link is being closely monitored and we haven’t seen any further problems.