This work forms the final phase of network upgrades that have been undertaken recently to increase core capacity and allow for new L2TP services.
During this window we will be moving the final interconnects to there new respective cores as well as undertaking a complete rebuild of the legacy core in order to transition this to an Access device. Configuration for the interconnect migrations will be in place prior to the physical moves, however there will be disruption expected on the following peers due to routing table updates following on from the disconnections / re-connections which is expected to be around 30-60 seconds per event.
Adapt / Level3
MDMX / Easynet
There will also be some changes made to our internal BGP sessions and filters due to the nature of the above works which will also result in disruption while tables are updated and BGP sessions established.
Once the above interconnects have been moved and tested, we will then be undertaking the rebuild to Access01. Unfortunately this work will result in several reboots of the device as well as some major configuration changes. This will require a longer downtime, however will be towards the end of the maintenance window.
The above works will affect all direct services (EasyIPT, EasyHTTP, EasyBond, EasyEFM, EasyLL) We apologize for any inconvenience this may cause.
UPDATE 01 – 23:30
Works have been completed with all the interconnects moved from the now “ACCESS01” device to there expected core devices. During the migration and build of our internal BGP sessions we did see an extended outage due to configuration. This was resolved within 45 minutes. We apologize for the unexpected extended disruption.
Due to limited remaining time and further planned disruption expected with the full rebuild of Access01 the decision was made to suspend that part of work and the device would be reviewed remotely to see if these changes could be done another way.
A full review will also be under taken over the next few days to confirm the changes made and any amendments required due to the unexpected configuration issues seen.
We are aware of a permission issue on the Plesk control panel for EasyHTTP which is generating “Access Denied” errors when changing various account settings.
A patch has been released and installed, however this does require a full reload of the server which will take place tonight at 20:00. Due to the size of the system this reload may take up to 20 minutes to complete. During this time all services will be unavailable.
This maintenance window consists of the works originally scheduled for the “26-03-2015 – 20:00 – 20:05” That where suspended.
Original notice from the 26th:
We will be shutting down our Frontier interconnect in preparation of moving the circuit from our current legacy access device to CORE01 as part of our ongoing network upgrade works. The circuit will be physically moved on the 07/04/2015 while and engineer is on site and bought back in to service that evening outside of hours.
Call traffic will be routed away from this interconnect several hours prior to the works, However any calls still active on this link will drop once disconnected. Routing to Frontier will then be via one of our indirect peers after the disconnection. No further impact is expected after this.
Further to these works we will also now be moving our Zen interconnect from our current legacy access device to CORE02 in-line with the same ongoing network upgrade works. Any traffic that comes in to us via Zen will see a short 30 second outage while our routing tables update to reflect the removal. Traffic will then flow via one of our indirect peers after the disconnection. This link will be bought back in to service along side our Frontier link.
We apologize for any inconvenience caused.
UPDATE 01 – 20:07
Engineers have shut down both peers and traffic has re-converged without further incident.
We have discovered that one of our internal BGP filters is incorrectly configured. We are currently reviewing all filters to ensure this is only limited to 1 instance. Once the review is complete we will be taking steps to correct the configuration issues found. This will result in routing table changes and disruption to various services.
We apologies for any inconvenience this may cause.
UPDATE 01 – 20:20
We have discovered only 1 miss-configured BGP filter on core01.lon01.structuredcommunications.co.uk. The BGP filter was only client facing and not carrier as originally suspected. No changes have been made to our carrier peering and no loss of service was seen.