LON01 – Zen BGP Session – 01:00- 04:00 – 12/06/2014 *Planned Maintenance* *COMPLETE*

We have been advised by Zen that they will be upgrading supervisor software on part of their core network where our cross-connect terminates. During this time we expect our BGP session to drop and be unavailable. Minimal service disruption is expected as traffic will re-coverage over other upstream providers while the link is down.

We have been advised by Zen that they will be upgrading supervisor software on part of their core network where our cross-connect terminates. During this time we expect our BGP session to drop and be unavailable.

Minimal service disruption is expected as traffic will re-coverage over other upstream providers while the link is down.

LON01 – BGP Update – 22:30- 00:30 – 04/06/2014 *COMPLETE*

BT Wholesale have today completed the fit and test on our optical bearer to Zen for private peering and transit. We will be bringing the BGP session online tonight in the above maintenance window. There will be service disruption for any customer who route to us via Zen on a single xDSL circuit, bonded customers … Continue reading “LON01 – BGP Update – 22:30- 00:30 – 04/06/2014 *COMPLETE*”

BT Wholesale have today completed the fit and test on our optical bearer to Zen for private peering and transit.

We will be bringing the BGP session online tonight in the above maintenance window.

There will be service disruption for any customer who route to us via Zen on a single xDSL circuit, bonded customers who have a Zen circuit in the cluster will see this go indirect and a reduction in speed and redundancy will be seen for the duration.

We expect the disruption to be minimal and only during BGP convergence (this includes testing)

UPDATE01 – 21:00
maintenance window moved to 22:30 – 00:30

UPDATE02 – 23:45
Work has started

UPDATE03 – 00:10
The BGP session is now up and traffic is flowing as expected.

UPDATE04 – 02:47
maintenance window closed

EasyHTTP – 16/04/2014 – 20:00 – Maintenance *COMPLETE*

Following on from today’s CPU issues, We have been advised there is a major upgrade for Mail Enable. This will be installed tonight to ensure we are running the latest release.# UPDATE 01 – 20:15 This work is now complete. Any users having problems with account syncing are advised to remove and re-add there account … Continue reading “EasyHTTP – 16/04/2014 – 20:00 – Maintenance *COMPLETE*”

Following on from today’s CPU issues, We have been advised there is a major upgrade for Mail Enable. This will be installed tonight to ensure we are running the latest release.#

UPDATE 01 – 20:15
This work is now complete. Any users having problems with account syncing are advised to remove and re-add there account to their mail client. (Remember to back up your messages first)

EasyBOND NS02 – 10/02/2014 – 22:00 – Planned Maintenance *COMPLETE*

We will shortly be taking ns02.easybond.co.uk offline to copy the virtual image over to another ESXi host for future upgrade work. No service impact is anticipated UPDATE01 – 22:05 *FINAL* This work is now complete.

We will shortly be taking ns02.easybond.co.uk offline to copy the virtual image over to another ESXi host for future upgrade work. No service impact is anticipated

UPDATE01 – 22:05 *FINAL*
This work is now complete.

LON01 Data Centre – 17/02/2014 – 22:00 – Planned Power Maintenance (B Feed)

Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator … Continue reading “LON01 Data Centre – 17/02/2014 – 22:00 – Planned Power Maintenance (B Feed)”

Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification

This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator banks.

Our core network and customer facing services (EasyHTTP, EasyIPT & EasyBOND) are fully power redundant and no service disruption is expected, however they will be classed as “at risk” for the 24 hours while the work is undertaken starting at 22:00 hours on the 17th Feb until 22:00 hours on the 18th Feb.

We will be attending site for the duration and taking the B feed out of service prior to the works in a controlled manor. Any customer services with only a single fed supply currently on B will be transferred to A prior to the work and returned once complete.

UPDATE 01 – 20:30 – 17/02/2014
Engineers have been onsite all day and prep work is now underway. Traffic is going to be transfered from AG-Primary to AG-Backup, this will cause a 30 second outage while devices transfer.

UPDATE 02 – 22:00 – 17/02/2014
An unexpected outage was seen for EasyBond circuits where an old version of the NOC came online due to ESXi01 not saving it’s startup configuration on shutdown. This has resulted in some stale sessions. Please power down your hardware for 20 minutes where power routers are in PPPOE mode or STATIC IP.

Withdrawal of 02/BE* Wholesale LLU DSL services

The 02 Wholesale network which is used to provide some of our ADSL services is being switched off on the 27th February 2014 following last year’s acquisition of the 02 and BE* retail customer base by Sky. This news is unwelcome within the industry and requires a change in service we provide to you if … Continue reading “Withdrawal of 02/BE* Wholesale LLU DSL services”

The 02 Wholesale network which is used to provide some of our ADSL services is being switched off on the 27th February 2014 following last year’s acquisition of the 02 and BE* retail customer base by Sky.

This news is unwelcome within the industry and requires a change in service we provide to you if you are on an LLU package provided by Structured Communications. BT Wholesale has advised they are accepting migrations from the 02 Wholesale network in light of the recent industry news, This will automatically be arranged by Structured Communications for effected circuits prior to the 27th February 2014.

The current pricing structure and packages will be migrated like for like as will login credentials and IP address settings.

We are starting migrations on the 14th of January 2014 whereby you will experience a brief loss of service while the physical exchange work is completed. (30 minutes average maximum) We hope to have all affected circuits migrated by the 1st February 2014

If you have any questions, please don’t hesitate to contact us on 0203 301 4000

NOC: Planned Maintenance – 20/12/2013 20:00 till 23/12/2013 23:59 – LON01 (EasyHTTP Service) *COMPLETE*

In line with our current service level agreements and previously advised (where applicable) we are pleased to announce we will be upgrading our current webhosting service (EasyHTTP) to our new hardware platform (HP DL580) very shortly. Our planned migration date for this is the 20th December 2013, however this will likely change depending on the … Continue reading “NOC: Planned Maintenance – 20/12/2013 20:00 till 23/12/2013 23:59 – LON01 (EasyHTTP Service) *COMPLETE*”

In line with our current service level agreements and previously advised (where applicable) we are pleased to announce we will be upgrading our current webhosting service (EasyHTTP) to our new hardware platform (HP DL580) very shortly. Our planned migration date for this is the 20th December 2013, however this will likely change depending on the migration method used and should be considered as TBC (to be confirmed)

There are no planned changes to IP addressing at this moment in time, and is the preferred migration method. However this will result in some downtime in order to ensure no new data is upload to the old server during the migration window.

Inbound and outbound SMTP will be refused at times during this window, unless you are using our backup MX service. In which case inbound mail will be held and delivered once the new server is online.

The new platform consists of:

Quad power supplies 6000 Watts (one per CPU)
4 x Intel quad core 2.93 GHz processers
32GB of RAM
8 x 500GB SAS drives in a RAID 5 configuration (3.2TB of useable space)
Dual 1Gbps uplinks

The upgrade is a very welcome project and more details will follow as the migration plan is finalised.

UPDATE01 – 17:04 – 09/12/2013
During an audit of existing scripting languages we have discovered PHP 4.4.9 is still installed. As this release has gone EOL (End of Life) it will not be installed on to the new platform. All existing sites using 4.4.9 will be re-graded to 5.2.17

We will also be upgrading MS SQL to 2012 which will result in a Instance Name change from “default” to “MSSQLSERVER2012” (excluding the “”) PLease ensure you update any connection strings.

UPDATE02 – 21:30 – 19/12/2013
We have started a FULL manual backup of the platform which will be used as the base / current image on the new array. Should the restore go as planned and once we are happy then we will manually copy over any new email received on the old system between now and midnight tomorrow. Once this is complete we will be taking the old system offline. Any new website data will not be migrated due to the size and time it would take resulting is a further loss of data. We would advise keeping any new uploads / changes to existing website data to a minimum.

We will update this once the backup and restore has completed advising of the next steps.

UPDATE03 – 23:00 – 20/12/2013
Migration is running behind due to an issue that has occurred with the use of the same IP addressing and the way Plesk maps this to the physical network card when not physically enabled. A new VLAN was installed to avoid this however we have had to process the backup / restore another way due to the extreme size of the backup and processing limitations on various protocol.

We are migrating data to a temporary IP range that we will revert once complete.

UPDATE04 – 01:15 – 21/12/2013
Migration work is still on-going and back on track, However due to the additional changes required we have extended this maintenance window for a further 24 hours.

This window will now close Sunday 6am.

UPDATE05 – 15:15 – 21/12/2013
The migration work completed last night however several warnings where encountered as MySQL stopped accepting connections from “localhost” and prevented a small number of domains having their mail boxes created. This issue has been resolved and we are now re-restoring the effected domains.

UPDATE06 – 18:15 – 21/12/2013
We have finished restoring the effected domains and the primary migration is now complete. We have now started the test migration of any email received since the initial copy. We have again extended this maintenance window for a further 24 hours.

This window will now close Monday 6am.

UPDATE07 – 19:00 – 22/12/2013
We have now started the final migration of any email received since the initial copy. Concerns over the size means we will be unable to copy any further email received on the old system from this time (19:00) We advise everyone to make a local copy of email received from this point to their local mail platform which can be copied back after

UPDATE08 – 23:30 – 22/12/2013

We have encountered an error that is preventing us from changing the primary default ip address. The error we are getting is in relation to an IIS module, however we suspect this is a corruption in an MS sql database.
We have been forced to log this with the software vendor as an escalation. As we have no influence on timing we are unlikely to have this resolved within our maintenance window and have therefore suspended this change. We are hopeful however this should be resolved by midday tomorrow so have extended this maintenance window until 23:00 tomorrow (Monday 23rd) This will allow us to migrate email data (again) however we are unable to restore any new website content due to time implications.

UPDATE09 – 12:15 – 23/12/2013

We have been advised by our software vendor that they have resolved the issue effecting the reconfiguration of IP addressing and the issue was related to corruption in the Master Plesk database and missing SSL certificate data following on from the migration. As the migration was completed via the built in migration tool we have raised a BUG with the vendor and ask them to investigate, however this may be related to the way we were forced to complete the migration.

The changes have now been made and the server is ready to come online and take over following the migration of any new email. As the new server now has the same IP addressing as the old system we are unable to complete a direct copy and will be required to copy the data to 3rd party storage device within our network. This additional step will require more time to complete the transfer

We therefore have confirmed the following times:

Copy email: 21:00
Down old server: 22:00
Up new server: 23:30

This will result in a planned 1 hour outage.

If any problems are experanced with the server then we will revert.

UPDATE10 – 20:15 – 23/12/2013

The work above has started. Times have been amended due to delays with weather and engineering.

UPDATE11 – 22:50 – 23/12/2013

We have completed the copy of any new email delivered to the old platform since 21:00 and are currently downloading this to the new system. The additional migration to an external storage source has added delay as expected. We have added an extra hour to the maintenance window.

UPDATE12 – 23:59 – 23/12/2013

We have completed the migration of email and the VLANS have been swapped on our core. This upgrade is now complete

NOC: Planned Maintenance – 29/11/2013 – 21:00 till 22:00 – LON01 (EasyBond) *COMPLETE*

We will be conducting an upgrade of our bonded platform on the above date to Patch 9 Version 2.32 The update will cause a brief interruption to service UPDATE01 *FINAL* @ 21:25 – 29/11/2013 This work is now complete.

We will be conducting an upgrade of our bonded platform on the above date to Patch 9 Version 2.32

The update will cause a brief interruption to service

UPDATE01 *FINAL* @ 21:25 – 29/11/2013

This work is now complete.

Entanet BGP Peering – Planned outage – 26/11/2013 – 00:01 till 06:00 *COMPLETE*

We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware. In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically … Continue reading “Entanet BGP Peering – Planned outage – 26/11/2013 – 00:01 till 06:00 *COMPLETE*”

We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware.

In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically re-routed via our other upstream providers, we will be taking our BGP / transit down several hours before the works are due to take place.

No long term service impact is anticipated, however people routing to us via our Enta transit will see between 30-60 second of routing instability while routes re-converge via BGP.

Peering shutdown time: 22:00 (25/11/2013)
Peering start-up time: 08:00 (26/11/2013)

UPDATE01 @ 19:10 – 25/11/2013

We have been advised that there is a problem with the newly run fibre. Due to this we have suspended the above planned works and will re-schedule.

UPDATE02 @ 08:00 – 26/11/2013

Our Upstream provider Entanet still relocated one of their POPs as planned. We will migrate of the existing core once the fibre fault has been resolved.

UPDATE03 @ 09:09 – 26/11/2013

I have just been advised that Level3 re-ran our Fibre cross-connects late last night following our escalation and both tested OK. Our upstream provider continued with the move on our behalf and we can advise that our transit links have now been migrated to the new core. During this move we did lose our BGP session as it was not taken down as originally planned, However other upstream providers took the traffic as our network is designed to with minimum downtime.

NOC: Planned Maintenance – 25/09/2013 – 21:00 till 22:00 – LON01 (Hosted EasyIPT) *COMPLETE*

We will be reviewing our out-of-band management on this service and associated servers. If adjustments are required then we will be conducting a controlled reboot in order to adjust the iLO BIOS on each device. An outage is expected during this window as handsets will be unable to register to the platform and will result … Continue reading “NOC: Planned Maintenance – 25/09/2013 – 21:00 till 22:00 – LON01 (Hosted EasyIPT) *COMPLETE*”

We will be reviewing our out-of-band management on this service and associated
servers. If adjustments are required then we will be conducting a controlled
reboot in order to adjust the iLO BIOS on each device. An outage is expected during this window as handsets will be unable to register to the platform and will result in a timeout.

This outage will due to its nature and the individual settings of each handset result in a possible stuck NAT session with a customers router, Where a customer has all our required managed hardware and where Structured Communications have a current and valid maintenance contract with the site, we will reboot the required hardware remotely. Where this is not the case, customers are requested to power-cycle there hardware if they find there handsets have not re-registered following this work.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 22:00 – 07/10/2013

This work is now complete