NOC: Planned Maintenance – 25/09/2013 – 21:00 till 22:00 – LON01 (EasyHTTP Service) *COMPLETE*

We will be reviewing our out-of-band management on this service and associated
servers. If adjustments are required then we will be conducting a controlled
reboot in order to adjust the iLO BIOS on each device. Several small outages may
be seen for HTTP traffic, FTP traffic and Email flow (both inbound and outbound). Email will not be lost durning this window as it should be queued on the originating server.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 22:00 – 07/10/2013

This work is now complete

NOC: Planned Maintenance – 25/09/2013 – 20:00 till 21:00 – LON01 (Easybond Service) *COMPLETE*

We will be reviewing our out-of-band management on this service and associated servers. If adjustments are required then we will be conducting a controlled reboot in order to adjust the iLO BIOS on each device. Several small outages may be seen while connections automatically fail-over to redundant hardware and ultimately back to the primary hardware.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

UPDATE @ 09:30 – 30/09/2013

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 21:00 – 07/10/2013

This work is now complete

Hosted EasyIPT

We are aware of an outage on our Hosted EasyIPT services. Engineers are currently assigned remotely and we have dispatched engineers to the data centre.

Update01**Final Resolved** @ 21:37

Engineers attended LON01 and arrived on site at 17:01 where the server was power cycled restoring service at 17:05. Handsets then started to re-register with individual sites coming back online.

The service has remained stable.

An outage report will be made available shortly.

LON01 IPv6 Support

Providers continue to deplete there IPv4 allocations and Structured Communications are no different. In preparation for our own depletion we will soon be enabling IPv6 support with our upstream providers. Further information will follow shortly.

**UPDATE 01 @ 01:00 – 08/09/2013**
We have experienced a delay in getting our current BGP sessions IPv6 enabled with our primary upstream peers. For this reason we have entered in to a peering agreement with Hurricane Electric to tunnel our IPv6 subnet over IPv4 for testing. We are announcing this range via our ASN (199283) to Hurricane Electric ASN (6939). The range is visible globally.

**UPDATE 02 @ 22:07 – 13/10/2013**
We are pleased to advise all our existing IPV4 BGP peering has been reconfigured to include IPV6 BGP sessions. At this moment in time we are announcing our entire /29 subnet (2a04:2140::/29) while we continue to test. We have several IPv6 DualStack servers operating within our range without incident and will be looking to fully enable our network for IPV6 in the next coming months. IPv6 will be available to customers once this is complete and will start off with co-location packages.

SSL Certificate – easyhttp.co.uk

We have just installed our new SSL Certificate for the Plesk control panel at https://easyhttp.co.uk:8443

The certificate expires October 13, 2014 (404 days from today)
Certificate Name matches easyhttp.co.uk
Subject easyhttp.co.uk
Valid from 29/Aug/2013 to 13/Oct/2014
Issuer AlphaSSL CA – G2

Please note, anyone who gets an error after the 13/09/2013 advising the Certificate is invalid. please flush your browser cache.

SMTP – Structured Communications Server Array 1

We are aware of an issue with outbound SMTP connections being refused and requesting an SSL connection. As we don’t support SSL these are also being refused.

We are taking the service offline at 17:30 to perform a software upgrade to the SMTP service.

*UPDATE 01 @ 17:53**

The upgrade is now complete which we hope will resolve the above issue, However we are also taking this Opportunity to update a few other components.

*UPDATE 01 @ 19:50**

This work is now complete.

Packet Loss on Virgin Media Circuits

We are aware of an issue with one of our wholesale interconnects with Virgin Media, which is causing packet loss for a small number of circuits.

An engineer is on route to investigate this link further. We are expecting to have further updates at approximately 13:00.

*UPDATE 01 @ 13:38**

Service has now been restored following the fiber link being cleaned and re-seated.

NOC: Planned Maintenance – 27/07/2013 – 22:00 till 23:30 – EasyBOND

We will be upgrading our bonded platform from version 2.19 to 2.32. During this time your router may disconnect a few times, and there may be a period of instability. We expect no more than 30-60 seconds of downtime during this window.

UPDATE 01 @ 23:30
This upgrade is now complete and we are monitoring the service.

NOC: Planned Maintenance – 24/07/2013 – 22:00 till 25/07/2013 – 02:00 – LON01 Goswell Road

In line with our previous upgrade work to LON01 we are now ready to “bring up” our BGP sessions to our new and existingt transit providers. During this time there may be a period of routing instability, however no service outages are expected.

**UPDATE 01 @ 18:44 – 24/07/2013**
Engineers are now onsite and will start to prepair for this work.

**UPDATE 02 @ 21:54 – 24/07/2013**
Engineers are now ready to start this work.

**UPDATE 03 @ 00:54 – 25/07/2013**
Engineers have now completed this work and our BGP sessions to our new peers are live, we are now solely announcing our IP space from our ASN (AS199283) Fail-over between links has been tested and completed successfully. Some routing instability maybe seen over the next 12-48 hours while BGP tables update to reflect our ASN, however should be minimal.

No outages where experienced durning this window, other than testing which lasted 15 seconds.

UPDATE 04 @ 16:33 – 25/07/2013 FINAL
90% of the world IPv4 registers now see our ASN and we are happy that the remaining will update over the next 12 hours and be reflected via RIPE.

    This work is now complete

SMTP – Structured Communications Server Array 1

We are aware of an issue with the SMTP service on Array1 and are looking in to the problem.

A temporary service suspension has been placed on inbound and outbound SMTP and IMAP services.

Inbound Emails are being queued. We will update as we know more.

**UPDATE 01 @ 13:33**

Upon investigation, we found the SMTP connectors to be at 30% load processing over 2,000 emails a second. We have disabled the SMTP connector to enable us to flush the queue. We are monitoring the service however it is now back online.

**UPDATE 02 @ 14:01**

We have observed further high usage on the SMTP connectors and are now looking in to the originating traffic.

**UPDATE 03 @ 14:40**

Traffic analysis and a review of the SMTP logs flagged that an email account had been compromised. This account has been disabled. Email flow and services have returned to normal. We will however contunie to monitor.

**UPDATE 04 @ 16:51**

The platform has remained stable, however we have noted a few temporary blacklisting’s of the servers external facing IPs. We have taken action to clear these blocks from mainstream listings, however there are still some temporary bans in place that we are unable to clear and will automatically revoke within 24 hours. This may affect the sending of emails to some systems who reference to these databases.

**UPDATE 05 @ 23:41**

The platform has continued to remain stable and will be closing this incident down in the next 24 hours. In light of the issues seen today we have implemented further security checks upon connecting to help prevent this happerning again.

Adational monitoring will also be out in place to alert on higher procress / service demand. This will alow us to respond faster to any potential future problems.

**UPDATE 06 FINAL @ 16:53 – 27/05**

The SMTP platform has been removed from all visible blacklists. This incident is now closed.