Author: Adam Clark (Network Operations)

In line with our current service level agreements and previously advised (where applicable) we are pleased to announce we will be upgrading our current webhosting service (EasyHTTP) to our new hardware platform (HP DL580) very shortly. Our planned migration date for this is the 20th December 2013, however this will likely change depending on the migration method used and should be considered as TBC (to be confirmed)

There are no planned changes to IP addressing at this moment in time, and is the preferred migration method. However this will result in some downtime in order to ensure no new data is upload to the old server during the migration window.

Inbound and outbound SMTP will be refused at times during this window, unless you are using our backup MX service. In which case inbound mail will be held and delivered once the new server is online.

The new platform consists of:

Quad power supplies 6000 Watts (one per CPU)
4 x Intel quad core 2.93 GHz processers
32GB of RAM
8 x 500GB SAS drives in a RAID 5 configuration (3.2TB of useable space)
Dual 1Gbps uplinks

The upgrade is a very welcome project and more details will follow as the migration plan is finalised.

UPDATE01 – 17:04 – 09/12/2013
During an audit of existing scripting languages we have discovered PHP 4.4.9 is still installed. As this release has gone EOL (End of Life) it will not be installed on to the new platform. All existing sites using 4.4.9 will be re-graded to 5.2.17

We will also be upgrading MS SQL to 2012 which will result in a Instance Name change from “default” to “MSSQLSERVER2012” (excluding the “”) PLease ensure you update any connection strings.

UPDATE02 – 21:30 – 19/12/2013
We have started a FULL manual backup of the platform which will be used as the base / current image on the new array. Should the restore go as planned and once we are happy then we will manually copy over any new email received on the old system between now and midnight tomorrow. Once this is complete we will be taking the old system offline. Any new website data will not be migrated due to the size and time it would take resulting is a further loss of data. We would advise keeping any new uploads / changes to existing website data to a minimum.

We will update this once the backup and restore has completed advising of the next steps.

UPDATE03 – 23:00 – 20/12/2013
Migration is running behind due to an issue that has occurred with the use of the same IP addressing and the way Plesk maps this to the physical network card when not physically enabled. A new VLAN was installed to avoid this however we have had to process the backup / restore another way due to the extreme size of the backup and processing limitations on various protocol.

We are migrating data to a temporary IP range that we will revert once complete.

UPDATE04 – 01:15 – 21/12/2013
Migration work is still on-going and back on track, However due to the additional changes required we have extended this maintenance window for a further 24 hours.

This window will now close Sunday 6am.

UPDATE05 – 15:15 – 21/12/2013
The migration work completed last night however several warnings where encountered as MySQL stopped accepting connections from “localhost” and prevented a small number of domains having their mail boxes created. This issue has been resolved and we are now re-restoring the effected domains.

UPDATE06 – 18:15 – 21/12/2013
We have finished restoring the effected domains and the primary migration is now complete. We have now started the test migration of any email received since the initial copy. We have again extended this maintenance window for a further 24 hours.

This window will now close Monday 6am.

UPDATE07 – 19:00 – 22/12/2013
We have now started the final migration of any email received since the initial copy. Concerns over the size means we will be unable to copy any further email received on the old system from this time (19:00) We advise everyone to make a local copy of email received from this point to their local mail platform which can be copied back after

UPDATE08 – 23:30 – 22/12/2013

We have encountered an error that is preventing us from changing the primary default ip address. The error we are getting is in relation to an IIS module, however we suspect this is a corruption in an MS sql database.
We have been forced to log this with the software vendor as an escalation. As we have no influence on timing we are unlikely to have this resolved within our maintenance window and have therefore suspended this change. We are hopeful however this should be resolved by midday tomorrow so have extended this maintenance window until 23:00 tomorrow (Monday 23rd) This will allow us to migrate email data (again) however we are unable to restore any new website content due to time implications.

UPDATE09 – 12:15 – 23/12/2013

We have been advised by our software vendor that they have resolved the issue effecting the reconfiguration of IP addressing and the issue was related to corruption in the Master Plesk database and missing SSL certificate data following on from the migration. As the migration was completed via the built in migration tool we have raised a BUG with the vendor and ask them to investigate, however this may be related to the way we were forced to complete the migration.

The changes have now been made and the server is ready to come online and take over following the migration of any new email. As the new server now has the same IP addressing as the old system we are unable to complete a direct copy and will be required to copy the data to 3rd party storage device within our network. This additional step will require more time to complete the transfer

We therefore have confirmed the following times:

Copy email: 21:00
Down old server: 22:00
Up new server: 23:30

This will result in a planned 1 hour outage.

If any problems are experanced with the server then we will revert.

UPDATE10 – 20:15 – 23/12/2013

The work above has started. Times have been amended due to delays with weather and engineering.

UPDATE11 – 22:50 – 23/12/2013

We have completed the copy of any new email delivered to the old platform since 21:00 and are currently downloading this to the new system. The additional migration to an external storage source has added delay as expected. We have added an extra hour to the maintenance window.

UPDATE12 – 23:59 – 23/12/2013

We have completed the migration of email and the VLANS have been swapped on our core. This upgrade is now complete

We will be conducting an upgrade of our bonded platform on the above date to Patch 9 Version 2.32

The update will cause a brief interruption to service

UPDATE01 *FINAL* @ 21:25 – 29/11/2013

This work is now complete.

We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware.

In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically re-routed via our other upstream providers, we will be taking our BGP / transit down several hours before the works are due to take place.

No long term service impact is anticipated, however people routing to us via our Enta transit will see between 30-60 second of routing instability while routes re-converge via BGP.

Peering shutdown time: 22:00 (25/11/2013)
Peering start-up time: 08:00 (26/11/2013)

UPDATE01 @ 19:10 – 25/11/2013

We have been advised that there is a problem with the newly run fibre. Due to this we have suspended the above planned works and will re-schedule.

UPDATE02 @ 08:00 – 26/11/2013

Our Upstream provider Entanet still relocated one of their POPs as planned. We will migrate of the existing core once the fibre fault has been resolved.

UPDATE03 @ 09:09 – 26/11/2013

I have just been advised that Level3 re-ran our Fibre cross-connects late last night following our escalation and both tested OK. Our upstream provider continued with the move on our behalf and we can advise that our transit links have now been migrated to the new core. During this move we did lose our BGP session as it was not taken down as originally planned, However other upstream providers took the traffic as our network is designed to with minimum downtime.

We will be reviewing our out-of-band management on this service and associated
servers. If adjustments are required then we will be conducting a controlled
reboot in order to adjust the iLO BIOS on each device. An outage is expected during this window as handsets will be unable to register to the platform and will result in a timeout.

This outage will due to its nature and the individual settings of each handset result in a possible stuck NAT session with a customers router, Where a customer has all our required managed hardware and where Structured Communications have a current and valid maintenance contract with the site, we will reboot the required hardware remotely. Where this is not the case, customers are requested to power-cycle there hardware if they find there handsets have not re-registered following this work.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 22:00 – 07/10/2013

This work is now complete