Category: Planned Works

We will shortly be taking ns02.easybond.co.uk offline to copy the virtual image over to another ESXi host for future upgrade work. No service impact is anticipated

UPDATE01 – 22:05 *FINAL*
This work is now complete.

Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification

This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator banks.

Our core network and customer facing services (EasyHTTP, EasyIPT & EasyBOND) are fully power redundant and no service disruption is expected, however they will be classed as “at risk” for the 24 hours while the work is undertaken starting at 22:00 hours on the 17th Feb until 22:00 hours on the 18th Feb.

We will be attending site for the duration and taking the B feed out of service prior to the works in a controlled manor. Any customer services with only a single fed supply currently on B will be transferred to A prior to the work and returned once complete.

UPDATE 01 – 20:30 – 17/02/2014
Engineers have been onsite all day and prep work is now underway. Traffic is going to be transfered from AG-Primary to AG-Backup, this will cause a 30 second outage while devices transfer.

UPDATE 02 – 22:00 – 17/02/2014
An unexpected outage was seen for EasyBond circuits where an old version of the NOC came online due to ESXi01 not saving it’s startup configuration on shutdown. This has resulted in some stale sessions. Please power down your hardware for 20 minutes where power routers are in PPPOE mode or STATIC IP.

In line with our current service level agreements and previously advised (where applicable) we are pleased to announce we will be upgrading our current webhosting service (EasyHTTP) to our new hardware platform (HP DL580) very shortly. Our planned migration date for this is the 20th December 2013, however this will likely change depending on the migration method used and should be considered as TBC (to be confirmed)

There are no planned changes to IP addressing at this moment in time, and is the preferred migration method. However this will result in some downtime in order to ensure no new data is upload to the old server during the migration window.

Inbound and outbound SMTP will be refused at times during this window, unless you are using our backup MX service. In which case inbound mail will be held and delivered once the new server is online.

The new platform consists of:

Quad power supplies 6000 Watts (one per CPU)
4 x Intel quad core 2.93 GHz processers
32GB of RAM
8 x 500GB SAS drives in a RAID 5 configuration (3.2TB of useable space)
Dual 1Gbps uplinks

The upgrade is a very welcome project and more details will follow as the migration plan is finalised.

UPDATE01 – 17:04 – 09/12/2013
During an audit of existing scripting languages we have discovered PHP 4.4.9 is still installed. As this release has gone EOL (End of Life) it will not be installed on to the new platform. All existing sites using 4.4.9 will be re-graded to 5.2.17

We will also be upgrading MS SQL to 2012 which will result in a Instance Name change from “default” to “MSSQLSERVER2012” (excluding the “”) PLease ensure you update any connection strings.

UPDATE02 – 21:30 – 19/12/2013
We have started a FULL manual backup of the platform which will be used as the base / current image on the new array. Should the restore go as planned and once we are happy then we will manually copy over any new email received on the old system between now and midnight tomorrow. Once this is complete we will be taking the old system offline. Any new website data will not be migrated due to the size and time it would take resulting is a further loss of data. We would advise keeping any new uploads / changes to existing website data to a minimum.

We will update this once the backup and restore has completed advising of the next steps.

UPDATE03 – 23:00 – 20/12/2013
Migration is running behind due to an issue that has occurred with the use of the same IP addressing and the way Plesk maps this to the physical network card when not physically enabled. A new VLAN was installed to avoid this however we have had to process the backup / restore another way due to the extreme size of the backup and processing limitations on various protocol.

We are migrating data to a temporary IP range that we will revert once complete.

UPDATE04 – 01:15 – 21/12/2013
Migration work is still on-going and back on track, However due to the additional changes required we have extended this maintenance window for a further 24 hours.

This window will now close Sunday 6am.

UPDATE05 – 15:15 – 21/12/2013
The migration work completed last night however several warnings where encountered as MySQL stopped accepting connections from “localhost” and prevented a small number of domains having their mail boxes created. This issue has been resolved and we are now re-restoring the effected domains.

UPDATE06 – 18:15 – 21/12/2013
We have finished restoring the effected domains and the primary migration is now complete. We have now started the test migration of any email received since the initial copy. We have again extended this maintenance window for a further 24 hours.

This window will now close Monday 6am.

UPDATE07 – 19:00 – 22/12/2013
We have now started the final migration of any email received since the initial copy. Concerns over the size means we will be unable to copy any further email received on the old system from this time (19:00) We advise everyone to make a local copy of email received from this point to their local mail platform which can be copied back after

UPDATE08 – 23:30 – 22/12/2013

We have encountered an error that is preventing us from changing the primary default ip address. The error we are getting is in relation to an IIS module, however we suspect this is a corruption in an MS sql database.
We have been forced to log this with the software vendor as an escalation. As we have no influence on timing we are unlikely to have this resolved within our maintenance window and have therefore suspended this change. We are hopeful however this should be resolved by midday tomorrow so have extended this maintenance window until 23:00 tomorrow (Monday 23rd) This will allow us to migrate email data (again) however we are unable to restore any new website content due to time implications.

UPDATE09 – 12:15 – 23/12/2013

We have been advised by our software vendor that they have resolved the issue effecting the reconfiguration of IP addressing and the issue was related to corruption in the Master Plesk database and missing SSL certificate data following on from the migration. As the migration was completed via the built in migration tool we have raised a BUG with the vendor and ask them to investigate, however this may be related to the way we were forced to complete the migration.

The changes have now been made and the server is ready to come online and take over following the migration of any new email. As the new server now has the same IP addressing as the old system we are unable to complete a direct copy and will be required to copy the data to 3rd party storage device within our network. This additional step will require more time to complete the transfer

We therefore have confirmed the following times:

Copy email: 21:00
Down old server: 22:00
Up new server: 23:30

This will result in a planned 1 hour outage.

If any problems are experanced with the server then we will revert.

UPDATE10 – 20:15 – 23/12/2013

The work above has started. Times have been amended due to delays with weather and engineering.

UPDATE11 – 22:50 – 23/12/2013

We have completed the copy of any new email delivered to the old platform since 21:00 and are currently downloading this to the new system. The additional migration to an external storage source has added delay as expected. We have added an extra hour to the maintenance window.

UPDATE12 – 23:59 – 23/12/2013

We have completed the migration of email and the VLANS have been swapped on our core. This upgrade is now complete

We will be conducting an upgrade of our bonded platform on the above date to Patch 9 Version 2.32

The update will cause a brief interruption to service

UPDATE01 *FINAL* @ 21:25 – 29/11/2013

This work is now complete.

We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware.

In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically re-routed via our other upstream providers, we will be taking our BGP / transit down several hours before the works are due to take place.

No long term service impact is anticipated, however people routing to us via our Enta transit will see between 30-60 second of routing instability while routes re-converge via BGP.

Peering shutdown time: 22:00 (25/11/2013)
Peering start-up time: 08:00 (26/11/2013)

UPDATE01 @ 19:10 – 25/11/2013

We have been advised that there is a problem with the newly run fibre. Due to this we have suspended the above planned works and will re-schedule.

UPDATE02 @ 08:00 – 26/11/2013

Our Upstream provider Entanet still relocated one of their POPs as planned. We will migrate of the existing core once the fibre fault has been resolved.

UPDATE03 @ 09:09 – 26/11/2013

I have just been advised that Level3 re-ran our Fibre cross-connects late last night following our escalation and both tested OK. Our upstream provider continued with the move on our behalf and we can advise that our transit links have now been migrated to the new core. During this move we did lose our BGP session as it was not taken down as originally planned, However other upstream providers took the traffic as our network is designed to with minimum downtime.

We will be reviewing our out-of-band management on this service and associated
servers. If adjustments are required then we will be conducting a controlled
reboot in order to adjust the iLO BIOS on each device. An outage is expected during this window as handsets will be unable to register to the platform and will result in a timeout.

This outage will due to its nature and the individual settings of each handset result in a possible stuck NAT session with a customers router, Where a customer has all our required managed hardware and where Structured Communications have a current and valid maintenance contract with the site, we will reboot the required hardware remotely. Where this is not the case, customers are requested to power-cycle there hardware if they find there handsets have not re-registered following this work.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 22:00 – 07/10/2013

This work is now complete

We will be reviewing our out-of-band management on this service and associated
servers. If adjustments are required then we will be conducting a controlled
reboot in order to adjust the iLO BIOS on each device. Several small outages may
be seen for HTTP traffic, FTP traffic and Email flow (both inbound and outbound). Email will not be lost durning this window as it should be queued on the originating server.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 22:00 – 07/10/2013

This work is now complete

We will be reviewing our out-of-band management on this service and associated servers. If adjustments are required then we will be conducting a controlled reboot in order to adjust the iLO BIOS on each device. Several small outages may be seen while connections automatically fail-over to redundant hardware and ultimately back to the primary hardware.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

UPDATE @ 09:30 – 30/09/2013

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 21:00 – 07/10/2013

This work is now complete

Providers continue to deplete there IPv4 allocations and Structured Communications are no different. In preparation for our own depletion we will soon be enabling IPv6 support with our upstream providers. Further information will follow shortly.

**UPDATE 01 @ 01:00 – 08/09/2013**
We have experienced a delay in getting our current BGP sessions IPv6 enabled with our primary upstream peers. For this reason we have entered in to a peering agreement with Hurricane Electric to tunnel our IPv6 subnet over IPv4 for testing. We are announcing this range via our ASN (199283) to Hurricane Electric ASN (6939). The range is visible globally.

**UPDATE 02 @ 22:07 – 13/10/2013**
We are pleased to advise all our existing IPV4 BGP peering has been reconfigured to include IPV6 BGP sessions. At this moment in time we are announcing our entire /29 subnet (2a04:2140::/29) while we continue to test. We have several IPv6 DualStack servers operating within our range without incident and will be looking to fully enable our network for IPV6 in the next coming months. IPv6 will be available to customers once this is complete and will start off with co-location packages.