Our network monitoring has alerted us to a intermittent fan failure on PSU 1 within CORE01.LON01. This device has redundant PSUs so no impact is expected. Engineers will attend this afternoon to swap out the failing power supply. Again no impact is expected to be seen, however the device will be marked as “at risk” … Continue reading “LON01 – 26/06/2015 – 13:00 – CORE01.LON01 *EMERGENCY WORKS* COMPLETE*”
Our network monitoring has alerted us to a intermittent fan failure on PSU 1 within CORE01.LON01. This device has redundant PSUs so no impact is expected.
Engineers will attend this afternoon to swap out the failing power supply. Again no impact is expected to be seen, however the device will be marked as “at risk” for the duration enginners are working on it.
Updates will be provided once available.
UPDATE01 – 14:00
This work has been completed without any issue.
In the above window we will be re-configuring BGP to provide a full IPv6 routing table to CORE03.LON01. No outage is expected, however as routing tables update within our network users may see routes change around our network.
In the above window we will be re-configuring BGP to provide a full IPv6 routing table to CORE03.LON01. No outage is expected, however as routing tables update within our network users may see routes change around our network.
In the above window we will be making changes to ACCESS01.lON01 converting this to our 3rd core device (CORE03.LON01) We will then be expanding BGP to the device to provide a full IPv4 routing table. No outage is expected, however as routing tables update within our network users may see routes changes around our network. … Continue reading “LON01 – 24-06-2015 – 20:00 – 22:00 – Network Maintenance (IPv4) *COMPLETE*”
In the above window we will be making changes to ACCESS01.lON01 converting this to our 3rd core device (CORE03.LON01) We will then be expanding BGP to the device to provide a full IPv4 routing table. No outage is expected, however as routing tables update within our network users may see routes changes around our network.
UPDATE 01
This work was part prosponed due to enginner avalibity and will take place on Friday along with the IPv6 works.
We are aware of a issue affecting connectivity to multiple networks within the UK. Initial testing had shown this to be a problem with Level3, however they are advising this to be part of a wider problem within the UK. We are currently monitoring and will look at re-routing traffic if this continues, however this … Continue reading “LON01 – 12/06/2015 *UK Routing Issue* *RESOLVED*”
We are aware of a issue affecting connectivity to multiple networks within the UK. Initial testing had shown this to be a problem with Level3, however they are advising this to be part of a wider problem within the UK.
We are currently monitoring and will look at re-routing traffic if this continues, however this would involve shutting down several of our peers and affecting traffic via unaffected links. This would be a last resort option.
Once we know more we will advise.
UPDATE 01 – 11:11
Level3 have updated us to advise they are now tracking a global routing issue. This now seems to be isolated to Level3, however will be affecting any network that peer with Level3. Re-routing traffic is still be an option, however may have no impact to networks that rely on Level3.
UPDATE 02 – 11:36
Our network monitoring has alerted us to routing failures to common websites such as the BBC, Facebook & Google. We have taken the decision to shutdown our Level3 peering with Adapt. While this has had some positive impact, 3rd party networks that are using Level3 will still be having issues.
UPDATE 03 – 11:55
Level3 have advised that services are starting to be restored. We have re-enabled our Level3 peer via Adapt and have noticed they also took the decision to showdown there link to Level3. Traffic are currently routing via Cogent via Adapt.
UPDATE 04 – 20:00
Outage report from a 3rd party monitoring network: http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/
In the above maintenance window we will be adding IPv6 support back to ACCESS01.LON01 after its removal during our previous network upgrades. We will also be enabling IPv6 on CORE02.LON01 to support IPv6 on our upcoming DSL network. IPv6 is already available on CORE01.LON01 for directly connected services and leased line circuits. No disruption is … Continue reading “LON01 – 07-06-2015 – 19:00 – 22:00 – Network Maintenance (IPv6) *COMPLETE*”
In the above maintenance window we will be adding IPv6 support back to ACCESS01.LON01 after its removal during our previous network upgrades. We will also be enabling IPv6 on CORE02.LON01 to support IPv6 on our upcoming DSL network. IPv6 is already available on CORE01.LON01 for directly connected services and leased line circuits.
No disruption is expected to IPv4 traffic or the network general.
UPDATE 01 – 19:00
Engineers have started the works.
UPDATE 02 – 19:50
IPv6 BGP sessions to our primary peer “Adapt” have been established along with our tertiary peer “Zen” Prefixes are being received on CORE01.lON01. A BGP session will now be created between CORE01.LON & CORE02.LON01
UPDATE 03 – 20:30
A BGP session between CORE01.LON & CORE02.LON01 has been created and is now in testing.
UPDATE 04 – 20:45
Testing has completed OK, however there is a problem with routing traffic to both Adapt & Zen. We are unable to pass IPv6 traffic to Adapt and we are not receiving a default route from Zen. Tickets have been raised with each carrier. At this moment in time we are not passing IPv6 traffic due to this fault. If a quick resolve is not provided then we will drop the effected peers.
UPDATE 05 – 21:20
While waiting for responses from upstream peers, we continued to investigate and discovered the ranges we where testing with where not announced within our internal V6 BGP session due to the use of a GRE tunnel. Network statements have been added and traffic is flowing via Adapt without issue from our test locations. No traffic loss would have been seen other than from our test locations and retract the previous statement about not passing traffic. We are still awaiting a response from Zen.
UPDATE 06 – 21:45
Due to time lost we have suspended the integration of ACCESS01.LON01. While technically there is less work to complete on this and could be completed within the window. We feel this should be rescheduled to avoid human error given this late hour and the concentration required when its comes to BGP!
We are currently experiencing issues with some internal systems that is preventing various customer service / support operations. Engineers are on route to the data centre, however local network rail cancellations and disruptions are delaying engineers from attending. Please also note the above is effectiving email delivery so we are unable to process support tickets. … Continue reading “Internal Systems *RESOLVED*”
We are currently experiencing issues with some internal systems that is preventing various customer service / support operations. Engineers are on route to the data centre, however local network rail cancellations and disruptions are delaying engineers from attending.
Please also note the above is effectiving email delivery so we are unable to process support tickets. We advise calling for any urgent issues.
Apologies for any inconvenience this may cause. Once we know more we will update this notice.
UPDATE01 – 12.00
Engineers have arrived on site.
UPDATE 02 – 12.30
Internal systems have been restored
We will be installing 2 new internal outbound BGP route maps on CORE01.LON01 & CORE02.LON01 tonight between 20:00 – 20:15. This work is to further improve BGP security internally between cores. The new rules should not cause any BGP session resets or any loss of existing known routes from upstream peers, however we may reset … Continue reading “LON01 – 26-05-2015 – 20:00 – 20:15 – Network Maintenance (BGP Filters) *COMPLETE*”
We will be installing 2 new internal outbound BGP route maps on CORE01.LON01 & CORE02.LON01 tonight between 20:00 – 20:15.
This work is to further improve BGP security internally between cores.
The new rules should not cause any BGP session resets or any loss of existing known routes from upstream peers, however we may reset the sessions to ensure they are working correctly. This will cause traffic to re-route for a few moments before returning to normal.
UPDATE 01 – 20:20
This window has been extended until 21:00 due to a late start by engineers.
UPDATE 02 – 20:27
CORE01.LON01 has been updated and the filters tested without the need for a full session reset.
UPDATE 03 – 20:35
CORE02.LON01 has been updated and the filters tested without the need for a full session reset.
Network monitoring shows all routing tables have returned to normal however engineers will monitor for the next few hours to ensure all routing tables remain consistent between Cores.
UPDATE 04 – 21:30
The routing tables on both CORE01.LON01 and CORE02.LON01 have remained constant for the past hour and we are happy to fully close this maintenance window.
At 10:01 our network monitoring detected a loss of IP transit to our primary carrier Adapt / Level3. Engineers have shutdown this link and traffic is routing via alternative carriers. We will continue to monitor and update as more information becomes available. We apologize for any inconvenience caused. UPDATE 01 – 11:01 We are continuing … Continue reading “LON01 – 06-05-2015 – Adapt / Level3 Transit Outage *Resolved*”
At 10:01 our network monitoring detected a loss of IP transit to our primary carrier Adapt / Level3. Engineers have shutdown this link and traffic is routing via alternative carriers. We will continue to monitor and update as more information becomes available.
We apologize for any inconvenience caused.
UPDATE 01 – 11:01
We are continuing to try and contact Adapt for an update as there are no issues with the Level3 side of the link. We believe from external testing there is a routing failure within the Adapt network.
Traffic is continuing to route via other links, however due to the volume of traffic being observed latency may be higher than normal. Should this continue then we will look to re-route traffic via our 3rd interconnect on CORE01 that has a higher capacity link.
UPDATE 02 – 11:18
We have been advised that the issue has been resolved, however we are seeking further confirmation before we bring the peer back in to service.
UPDATE 03 – 11:18
We have spoken to the senior network architect at Adapt who has confirmed service has been restored and we have re-enabled the session. We are observing normal traffic flow, however will continue to monitor the link. Once an RFO has been published by Adapt we will provide this.
This work forms the final phase of network upgrades that have been undertaken recently to increase core capacity and allow for new L2TP services. During this window we will be moving the final interconnects to there new respective cores as well as undertaking a complete rebuild of the legacy core in order to transition this … Continue reading “LON01 – 29-04-2015 – 20:00 – 02:00 – Network Maintenance *COMPLETE*”
This work forms the final phase of network upgrades that have been undertaken recently to increase core capacity and allow for new L2TP services.
During this window we will be moving the final interconnects to there new respective cores as well as undertaking a complete rebuild of the legacy core in order to transition this to an Access device. Configuration for the interconnect migrations will be in place prior to the physical moves, however there will be disruption expected on the following peers due to routing table updates following on from the disconnections / re-connections which is expected to be around 30-60 seconds per event.
Adapt / Level3
Zen Internet
MDMX / Easynet
There will also be some changes made to our internal BGP sessions and filters due to the nature of the above works which will also result in disruption while tables are updated and BGP sessions established.
Once the above interconnects have been moved and tested, we will then be undertaking the rebuild to Access01. Unfortunately this work will result in several reboots of the device as well as some major configuration changes. This will require a longer downtime, however will be towards the end of the maintenance window.
The above works will affect all direct services (EasyIPT, EasyHTTP, EasyBond, EasyEFM, EasyLL) We apologize for any inconvenience this may cause.
UPDATE 01 – 23:30
Works have been completed with all the interconnects moved from the now “ACCESS01” device to there expected core devices. During the migration and build of our internal BGP sessions we did see an extended outage due to configuration. This was resolved within 45 minutes. We apologize for the unexpected extended disruption.
Due to limited remaining time and further planned disruption expected with the full rebuild of Access01 the decision was made to suspend that part of work and the device would be reviewed remotely to see if these changes could be done another way.
A full review will also be under taken over the next few days to confirm the changes made and any amendments required due to the unexpected configuration issues seen.
We are aware of a permission issue on the Plesk control panel for EasyHTTP which is generating “Access Denied” errors when changing various account settings. A patch has been released and installed, however this does require a full reload of the server which will take place tonight at 20:00. Due to the size of the … Continue reading “EasyHTTP – 24/04/2015 – 20:00 – 20:30 – Emergency Maintenance *COMPLETE*”
We are aware of a permission issue on the Plesk control panel for EasyHTTP which is generating “Access Denied” errors when changing various account settings.
A patch has been released and installed, however this does require a full reload of the server which will take place tonight at 20:00. Due to the size of the system this reload may take up to 20 minutes to complete. During this time all services will be unavailable.