Over the past 4-5 years we have continued to invest in our network with the continued aim of providing our own services and simply not reselling as many communication providers do and as we ourselves once did! Growth over the years has consumed a large amount of physical rack space within our own private suite … Continue reading “LON01 – 4th Floor suite / network migration – 01/06/2016 – 03/06/2016 – 20:00 – 03:00”
Over the past 4-5 years we have continued to invest in our network with the continued aim of providing our own services and simply not reselling as many communication providers do and as we ourselves once did!
Growth over the years has consumed a large amount of physical rack space within our own private suite at Level3, Goswell Road. We have taken steps over the past 6 weeks to secure a larger suite within Goswell Road and have already started the complex task of moving our network to this new area.
Over the past 2 weeks we have been adding additional servers to key strategic network locations in order to replace legacy setup to provide continued service for the upcoming works which will be detailed below.
Our core network is fully redundant as are many of our services we provide as standard and on more complex products this can be provided at additional costs, however as we are physically moving core devices and hardware there will unfortunately be some disruption to services. Works are being done outside of hours to minimise this.
Due to various 3rd party contracts being involved, we have been delaying in getting this notice out as various orders have had to fall in to place, such as new bulk fibre runs, power and infrastructure to name but a few. However as detailed above works have been ongoing to insure we keep services operational where possible during the planned works.
Below is a overview of the disruption to services with a full plan being published on the 30th (tomorrow) We would like to stress that due to the works already done, the way in which the works are being carried out along with the time in which they are being conducted will mean that 99% of customers wont see an impact.
| Service |
Type |
Impact |
| Connectivity |
Copper Broadband |
Minor Impact Expected |
| Connectivity |
Fibre Broadband |
Minor Impact Expected |
| Connectivity |
Bonded Broadband |
Limited Service |
| Connectivity |
Leased Line |
No Impact Expected |
| Connectivity |
EFM |
No Impact Expected |
| Hosting |
SMTP, POP, IMAP, HTTP |
Limited Service |
| Hosting |
VPS |
Limited Service |
| Hosting |
Co-Location |
Limited Service |
| Voice |
Managed “Shared” VoiP |
Limited Service |
| Voice |
Managed “Dedicated” VoiP |
Limited Service |
| Voice |
SIP Trunking |
Limited Service |
Again a full detailed report will be available tomorrow and a time line of planned events. (The above looks worse than it really is)
Upon arrival on the 01/06/2016, We will conduct a risk assessment to ensure the new area has been correctly handed over and is operational to our standards. Should this fail or impact our migration to a level that puts our network and thus clients at risk, then we will propose the planned works.
UPDATE 01 – 30/05/2016
As previously advised, please see below for a full breakdown of events and what impact is to be expected.
|
Connectivity – Copper & Fibre Broadband |
|
Prior to the works commencing we will be updating our network routing to utilise our backup link in to Telehose East for L2TP DSL. This link is provided via another part of the network (CORE01) that is unaffected by the planned works. We will also be updating our RADIUS session steering to terminate all new and existing DSL sessions on LNS02 via RADIUS02 which are both also connected to Core01.
Unfortunately our primary SQL server while redundant is located within our suite and connected to CORE02 which is affected by the planned works. This will mean that DSL users who drop there PPP session(s) while the works are under way will be unable to re-authenticate until this server is back on-line.
DSL sessions will be manually moved tonight @ 22:00 as advised above to ensure the fail over works as expected. DSL customers will see a small outage while there routers re authenticate following the switch over.
|
|
Connectivity – Bonded Broadband |
|
While we do provide redundant servers for our bonded platform they are both located within the suite and connected to CORE03 which is affected by the works. Unfortunately our software vendor has not provided a recent software image that compromises of all there recent updates (1 years worth). While we had hoped to “simply” be able to move the redundant server to its new network location as we had done so with other platforms, we where advised this was not possible without a re-image and reconfiguration of the software.
This we where happy to undertake, however due to the volume of updates that then need to be applied following this and the possible outages this would cause due to the requirement of taking the controller offline we are unable to complete the works without greater disruption being caused than simply moving the platform. We are still talking with the vendor to see if other options are available.
At this stage, during the move Bonded Broadband will be unavailable. We have scheduled this platform move to cause as little downtime as possible subject to no unforeseen problems. We apologise for any inconvenience this may cause.
|
|
Connectivity – EFM and Fibre Leased Lines |
|
No impact is expected on these services as they are provided from CORE01 which is unaffected by the works
|
|
Hosting – All Services |
|
Due to the physical relocation of these services, they will need to be powered down during the maintiance window. Services will be unavailable during this time. Any email should be queued at remote servers. We apologise for any inconvenience this may cause.
|
|
Voice – All Managed Services |
|
Due to the physical relocation of the underlying servers they will need to be powered down during the maintiance window. Voice services will be unavailable during this time. We advise clients to have alternative methods of communications for an emergency such as a mobile and apologise for any inconvenience this may cause.
|
|
Voice – SIP Trunking |
|
Due to the physical relocation of our media servers and softswitch they will need to be powered down during the maintiance window. SIP trunks will be unavailable during this time. We advise clients to have alternative methods of communications for an emergency such as a mobile and apologise for any inconvenience this may cause.
|
Our migration plan factors in 3 days worth of works, with the 1st day mostly comprising of physically moving cables and re-routing parts of the network ready for the physical move the following day(s) During day 1 the physical movement of cables will cause our network to suffer re-convergence of traffic , links and transit providers.
UPDATE 02 – 03/06/2016 – 06:00
Works went ahead as planned and all services have been restored however we encountered problems with our new bulk fibre and where forced to utilise existing spares, as a result of this various network peering are down and traffic is being re-routed. Remedial works are being undertaken over the next week to restore the network back to its redundant state.
UPDATE 03 – 03/06/2016 – 09:30
We have observed that one of our private primary voice peering links is flapping due to another fibre problem. This has been taken out of service and traffic is being re-routed.
UPDATE 04 – 03/06/2016 – 17:30
The network has remained stable on its redundant links. Work orders are being raised with Level3 for additional fibre.
UPDATE 05 – 06/06/2016 – 13:00
We have restored our primary voice peering link along with other internal redundant links, along with adding a tempory work around for other down peers. Work is still ongoing with Level3 for additional fibre