Author: Adam Clark (Network Operations)

In the above window we will be making changes to ACCESS01.lON01 converting this to our 3rd core device (CORE03.LON01) We will then be expanding BGP to the device to provide a full IPv4 routing table. No outage is expected, however as routing tables update within our network users may see routes changes around our network.

UPDATE 01
This work was part prosponed due to enginner avalibity and will take place on Friday along with the IPv6 works.

In the above maintenance window we will be adding IPv6 support back to ACCESS01.LON01 after its removal during our previous network upgrades. We will also be enabling IPv6 on CORE02.LON01 to support IPv6 on our upcoming DSL network. IPv6 is already available on CORE01.LON01 for directly connected services and leased line circuits.

No disruption is expected to IPv4 traffic or the network general.

UPDATE 01 – 19:00
Engineers have started the works.

UPDATE 02 – 19:50
IPv6 BGP sessions to our primary peer “Adapt” have been established along with our tertiary peer “Zen” Prefixes are being received on CORE01.lON01. A BGP session will now be created between CORE01.LON & CORE02.LON01

UPDATE 03 – 20:30
A BGP session between CORE01.LON & CORE02.LON01 has been created and is now in testing.

UPDATE 04 – 20:45
Testing has completed OK, however there is a problem with routing traffic to both Adapt & Zen. We are unable to pass IPv6 traffic to Adapt and we are not receiving a default route from Zen. Tickets have been raised with each carrier. At this moment in time we are not passing IPv6 traffic due to this fault. If a quick resolve is not provided then we will drop the effected peers.

UPDATE 05 – 21:20
While waiting for responses from upstream peers, we continued to investigate and discovered the ranges we where testing with where not announced within our internal V6 BGP session due to the use of a GRE tunnel. Network statements have been added and traffic is flowing via Adapt without issue from our test locations. No traffic loss would have been seen other than from our test locations and retract the previous statement about not passing traffic. We are still awaiting a response from Zen.

UPDATE 06 – 21:45
Due to time lost we have suspended the integration of ACCESS01.LON01. While technically there is less work to complete on this and could be completed within the window. We feel this should be rescheduled to avoid human error given this late hour and the concentration required when its comes to BGP!

We will be installing 2 new internal outbound BGP route maps on CORE01.LON01 & CORE02.LON01 tonight between 20:00 – 20:15.

This work is to further improve BGP security internally between cores.

The new rules should not cause any BGP session resets or any loss of existing known routes from upstream peers, however we may reset the sessions to ensure they are working correctly. This will cause traffic to re-route for a few moments before returning to normal.

UPDATE 01 – 20:20
This window has been extended until 21:00 due to a late start by engineers.

UPDATE 02 – 20:27
CORE01.LON01 has been updated and the filters tested without the need for a full session reset.

UPDATE 03 – 20:35
CORE02.LON01 has been updated and the filters tested without the need for a full session reset.

Network monitoring shows all routing tables have returned to normal however engineers will monitor for the next few hours to ensure all routing tables remain consistent between Cores.

UPDATE 04 – 21:30
The routing tables on both CORE01.LON01 and CORE02.LON01 have remained constant for the past hour and we are happy to fully close this maintenance window.

This work forms the final phase of network upgrades that have been undertaken recently to increase core capacity and allow for new L2TP services.

During this window we will be moving the final interconnects to there new respective cores as well as undertaking a complete rebuild of the legacy core in order to transition this to an Access device. Configuration for the interconnect migrations will be in place prior to the physical moves, however there will be disruption expected on the following peers due to routing table updates following on from the disconnections / re-connections which is expected to be around 30-60 seconds per event.

Adapt / Level3
Zen Internet
MDMX / Easynet

There will also be some changes made to our internal BGP sessions and filters due to the nature of the above works which will also result in disruption while tables are updated and BGP sessions established.

Once the above interconnects have been moved and tested, we will then be undertaking the rebuild to Access01. Unfortunately this work will result in several reboots of the device as well as some major configuration changes. This will require a longer downtime, however will be towards the end of the maintenance window.

The above works will affect all direct services (EasyIPT, EasyHTTP, EasyBond, EasyEFM, EasyLL) We apologize for any inconvenience this may cause.

UPDATE 01 – 23:30
Works have been completed with all the interconnects moved from the now “ACCESS01” device to there expected core devices. During the migration and build of our internal BGP sessions we did see an extended outage due to configuration. This was resolved within 45 minutes. We apologize for the unexpected extended disruption.

Due to limited remaining time and further planned disruption expected with the full rebuild of Access01 the decision was made to suspend that part of work and the device would be reviewed remotely to see if these changes could be done another way.

A full review will also be under taken over the next few days to confirm the changes made and any amendments required due to the unexpected configuration issues seen.

This maintenance window consists of the works originally scheduled for the “26-03-2015 – 20:00 – 20:05” That where suspended.

Original notice from the 26th:

We will be shutting down our Frontier interconnect in preparation of moving the circuit from our current legacy access device to CORE01 as part of our ongoing network upgrade works. The circuit will be physically moved on the 07/04/2015 while and engineer is on site and bought back in to service that evening outside of hours.

Call traffic will be routed away from this interconnect several hours prior to the works, However any calls still active on this link will drop once disconnected. Routing to Frontier will then be via one of our indirect peers after the disconnection. No further impact is expected after this.

Further to these works we will also now be moving our Zen interconnect from our current legacy access device to CORE02 in-line with the same ongoing network upgrade works. Any traffic that comes in to us via Zen will see a short 30 second outage while our routing tables update to reflect the removal. Traffic will then flow via one of our indirect peers after the disconnection. This link will be bought back in to service along side our Frontier link.

We apologize for any inconvenience caused.

UPDATE 01 – 20:07
Engineers have shut down both peers and traffic has re-converged without further incident.