LON01 – 4th Floor suite / network migration – 01/06/2016 – 03/06/2016 – 20:00 – 03:00

Over the past 4-5 years we have continued to invest in our network with the continued aim of providing our own services and simply not reselling as many communication providers do and as we ourselves once did!

Growth over the years has consumed a large amount of physical rack space within our own private suite at Level3, Goswell Road. We have taken steps over the past 6 weeks to secure a larger suite within Goswell Road and have already started the complex task of moving our network to this new area.

Over the past 2 weeks we have been adding additional servers to key strategic network locations in order to replace legacy setup to provide continued service for the upcoming works which will be detailed below.

Our core network is fully redundant as are many of our services we provide as standard and on more complex products this can be provided at additional costs, however as we are physically moving core devices and hardware there will unfortunately be some disruption to services. Works are being done outside of hours to minimise this.

Due to various 3rd party contracts being involved, we have been delaying in getting this notice out as various orders have had to fall in to place, such as new bulk fibre runs, power and infrastructure to name but a few. However as detailed above works have been ongoing to insure we keep services operational where possible during the planned works.

Below is a overview of the disruption to services with a full plan being published on the 30th (tomorrow) We would like to stress that due to the works already done, the way in which the works are being carried out along with the time in which they are being conducted will mean that 99% of customers wont see an impact.

Service Type Impact
Connectivity Copper Broadband Minor Impact Expected
Connectivity Fibre Broadband Minor Impact Expected
Connectivity Bonded Broadband Limited Service
Connectivity Leased Line No Impact Expected
Connectivity EFM No Impact Expected
Hosting SMTP, POP, IMAP, HTTP Limited Service
Hosting VPS Limited Service
Hosting Co-Location Limited Service
Voice Managed “Shared” VoiP Limited Service
Voice Managed “Dedicated” VoiP Limited Service
Voice SIP Trunking Limited Service

Again a full detailed report will be available tomorrow and a time line of planned events. (The above looks worse than it really is)

Upon arrival on the 01/06/2016, We will conduct a risk assessment to ensure the new area has been correctly handed over and is operational to our standards. Should this fail or impact our migration to a level that puts our network and thus clients at risk, then we will propose the planned works.

UPDATE 01 – 30/05/2016

As previously advised, please see below for a full breakdown of events and what impact is to be expected.

Connectivity – Copper & Fibre Broadband

Prior to the works commencing we will be updating our network routing to utilise our backup link in to Telehose East for L2TP DSL. This link is provided via another part of the network (CORE01) that is unaffected by the planned works. We will also be updating our RADIUS session steering to terminate all new and existing DSL sessions on LNS02 via RADIUS02 which are both also connected to Core01.

Unfortunately our primary SQL server while redundant is located within our suite and connected to CORE02 which is affected by the planned works. This will mean that DSL users who drop there PPP session(s) while the works are under way will be unable to re-authenticate until this server is back on-line.

DSL sessions will be manually moved tonight @ 22:00 as advised above to ensure the fail over works as expected. DSL customers will see a small outage while there routers re authenticate following the switch over.

Connectivity – Bonded Broadband

While we do provide redundant servers for our bonded platform they are both located within the suite and connected to CORE03 which is affected by the works. Unfortunately our software vendor has not provided a recent software image that compromises of all there recent updates (1 years worth). While we had hoped to “simply” be able to move the redundant server to its new network location as we had done so with other platforms, we where advised this was not possible without a re-image and reconfiguration of the software.

This we where happy to undertake, however due to the volume of updates that then need to be applied following this and the possible outages this would cause due to the requirement of taking the controller offline we are unable to complete the works without greater disruption being caused than simply moving the platform. We are still talking with the vendor to see if other options are available.

At this stage, during the move Bonded Broadband will be unavailable. We have scheduled this platform move to cause as little downtime as possible subject to no unforeseen problems. We apologise for any inconvenience this may cause.

Connectivity – EFM and Fibre Leased Lines

No impact is expected on these services as they are provided from CORE01 which is unaffected by the works

Hosting – All Services

Due to the physical relocation of these services, they will need to be powered down during the maintiance window. Services will be unavailable during this time. Any email should be queued at remote servers. We apologise for any inconvenience this may cause.

Voice – All Managed Services

Due to the physical relocation of the underlying servers they will need to be powered down during the maintiance window. Voice services will be unavailable during this time. We advise clients to have alternative methods of communications for an emergency such as a mobile and apologise for any inconvenience this may cause.

Voice – SIP Trunking

Due to the physical relocation of our media servers and softswitch they will need to be powered down during the maintiance window. SIP trunks will be unavailable during this time. We advise clients to have alternative methods of communications for an emergency such as a mobile and apologise for any inconvenience this may cause.

Our migration plan factors in 3 days worth of works, with the 1st day mostly comprising of physically moving cables and re-routing parts of the network ready for the physical move the following day(s) During day 1 the physical movement of cables will cause our network to suffer re-convergence of traffic , links and transit providers.

UPDATE 02 – 03/06/2016 – 06:00

Works went ahead as planned and all services have been restored however we encountered problems with our new bulk fibre and where forced to utilise existing spares, as a result of this various network peering are down and traffic is being re-routed. Remedial works are being undertaken over the next week to restore the network back to its redundant state.

UPDATE 03 – 03/06/2016 – 09:30

We have observed that one of our private primary voice peering links is flapping due to another fibre problem. This has been taken out of service and traffic is being re-routed.

UPDATE 04 – 03/06/2016 – 17:30

The network has remained stable on its redundant links. Work orders are being raised with Level3 for additional fibre.

UPDATE 05 – 06/06/2016 – 13:00

We have restored our primary voice peering link along with other internal redundant links, along with adding a tempory work around for other down peers. Work is still ongoing with Level3 for additional fibre

LON01 – EasyIPT – 16/05/2016 – 21:40 till 22:45 *Emergency Maintenance* *Complete*

We have taken action to reload our primary soft switch to clear down some stuck SIP sessions. We are working with our software vendor to try an automate this process without the need for a complete reload of the singling platform in future.

We have been advised that an update will help limit the need to do this, with a further one planned to resolve this completely.

Calls are routing correctly

LON01 – SMTP Decommission- 95.87.104.26 – 04/06/2016

On the 04/06/2016 we will be decommissioning SMTP relay 95.87.104.26 as it has been relocated within the network to provide greater redundancy in line with our network migration & upgrade plans during the 1st, 2nd and 3rd of June 2016. (Further updates to follow on that)

We have commissioned 95.87.111.24 in place of the above and is actively taking requests as of today. SMTP users who use one of our “round robin” DNS records will automatically see the updated address within the next 24-48 hours.

Customers relaying directly to the various mail server IPs we provide including the above, are advised to updated there configuration to a round robin address as below or the new IP address.

smtp.dsl.structuredcommunications.co.uk
smtp.easybond.co.uk

Failure to make this update may result in you loosing SMTP service and or redundancy.

If you have any doubt please contact us.

UPDATE 01 – 11/06/2016 – 11:30 – FINAL

This SMTP relay has now been decommissioned. Users are advised to update there secondary SMTP to 95.87.111.25 to avoid any loss of service. We have updated our round robin DNS so users making use of:

smtp.dsl.structuredcommunications.co.uk
smtp.easybond.co.uk

should be unaffected.

LON01 – DNS Decommission- 95.87.104.25 – 04/06/2016

On the 04/06/2016 we will be decommissioning DNS resolver 95.87.104.25 as it has been relocated within the network to provide greater redundancy in line with our network migration & upgrade plans during the 1st, 2nd and 3rd of June 2016. (Further updates to follow on that)

We have commissioned 95.87.111.25 in place of the above and is actively taking requests as of today. DSL users who rely on PPP to auto-negotiate these servers will automatically see the updated address within the next 24 hours. Customers with a Structured Communications managed CPE, firewall and or system will also automatically see these changes prior to the 4th.

Customers with there own provided hardware using a static configuration (not related to static IPs) may need to make this update manually.

Failure to make this update may result in you loosing DNS service and or redundancy.

If you have any doubt please contact us.

UPDATE 01 – 11/06/2016 – 11:30 – FINAL

This DNS resolver has now been decommissioned. Users are advised to update there secondary DNS to 95.87.111.25 to avoid any loss of service. Managed CPEs whereby we still have access have been updated.

LON01 – EasyIPT – 06/05/2016 – 15:50 *Resolved*

We are aware some customers may have received an automated message advising they where unable to dial out due to there account be suspended. This is not a message originating from our network and we have re-routed calls while investigations are ongoing with that carrier.

We applogise for any inconvenience caused.

UPDATE01 – 16:10 – FINAL

We have received an update from our carrier to advise a billing script was run against wholesale interconnects (such as ours) that had an unexpected effect on call barring restrictions which prevented all but emergency calls to progress. As the calls where routing correctly to our carrier with no normal failure message at a “network level” our systems did not exclude that carrier on the 1st attempt. Callers that waited until the end of the message would have seen the call connect.

Once again we applogise for any inconvenience caused and have been advised this script wont be run again.