LON01 – EasyIPT – 13/07/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*

We are aware some users have been seeing stuck SIP sessions today, whereby “BYE” messages have not been completely clearing down resulting in accounts hitting a concurrent call limit. A work around has been put in place for this until this evening when we can reload the core device to drop any sessions still in … Continue reading “LON01 – EasyIPT – 13/07/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*”

We are aware some users have been seeing stuck SIP sessions today, whereby “BYE” messages have not been completely clearing down resulting in accounts hitting a concurrent call limit. A work around has been put in place for this until this evening when we can reload the core device to drop any sessions still in this state.

UPDATE01 – 20:05
This work has started and the server has been requested to reload.

UPDATE02 – 20:15
The server has reloaded and is accepting SIP requests, however is not processing any call routing.

UPDATE03 – 20:35
We have discovered that the Kamailio Proxy service has failed to start which is why calls are not being processed. We are working to restore the service.

UPDATE04 – 20:40
The service has been restored and test calls are now routing OK.

LON01 – 01/07/2015 – 20:00 – 21:00 – EasyBond – Maintenance *COMPLETE*

During this window we will be upgrading our bonded platform to the latest software version as per our software vendors recommendation. Customers on the platform may notice drops in service while this takes place. Any customer using the service with our EasyIPT VOIP solution may notice there handsets fail to register with status lights remaining … Continue reading “LON01 – 01/07/2015 – 20:00 – 21:00 – EasyBond – Maintenance *COMPLETE*”

During this window we will be upgrading our bonded platform to the latest software version as per our software vendors recommendation. Customers on the platform may notice drops in service while this takes place. Any customer using the service with our EasyIPT VOIP solution may notice there handsets fail to register with status lights remaining orange. If this occurs please power down all hardware for 5 minutes and power back up once the maintenance window is marked as complete.

UPDATE01 – 21:06
This work is complete.

LON01 – 26/06/2015 – 13:00 – CORE01.LON01 *EMERGENCY WORKS* COMPLETE*

Our network monitoring has alerted us to a intermittent fan failure on PSU 1 within CORE01.LON01. This device has redundant PSUs so no impact is expected. Engineers will attend this afternoon to swap out the failing power supply. Again no impact is expected to be seen, however the device will be marked as “at risk” … Continue reading “LON01 – 26/06/2015 – 13:00 – CORE01.LON01 *EMERGENCY WORKS* COMPLETE*”

Our network monitoring has alerted us to a intermittent fan failure on PSU 1 within CORE01.LON01. This device has redundant PSUs so no impact is expected.

Engineers will attend this afternoon to swap out the failing power supply. Again no impact is expected to be seen, however the device will be marked as “at risk” for the duration enginners are working on it.

Updates will be provided once available.

UPDATE01 – 14:00

This work has been completed without any issue.

LON01 – 26-06-2015 – 20:00 – 22:00 – Network Maintenance (CORE03.LON01) *COMPLETE*

In the above window we will be re-configuring BGP to provide a full IPv6 routing table to CORE03.LON01. No outage is expected, however as routing tables update within our network users may see routes change around our network.

In the above window we will be re-configuring BGP to provide a full IPv6 routing table to CORE03.LON01. No outage is expected, however as routing tables update within our network users may see routes change around our network.

LON01 – 24-06-2015 – 20:00 – 22:00 – Network Maintenance (IPv4) *COMPLETE*

In the above window we will be making changes to ACCESS01.lON01 converting this to our 3rd core device (CORE03.LON01) We will then be expanding BGP to the device to provide a full IPv4 routing table. No outage is expected, however as routing tables update within our network users may see routes changes around our network. … Continue reading “LON01 – 24-06-2015 – 20:00 – 22:00 – Network Maintenance (IPv4) *COMPLETE*”

In the above window we will be making changes to ACCESS01.lON01 converting this to our 3rd core device (CORE03.LON01) We will then be expanding BGP to the device to provide a full IPv4 routing table. No outage is expected, however as routing tables update within our network users may see routes changes around our network.

UPDATE 01
This work was part prosponed due to enginner avalibity and will take place on Friday along with the IPv6 works.

LON01 – 12/06/2015 *UK Routing Issue* *RESOLVED*

We are aware of a issue affecting connectivity to multiple networks within the UK. Initial testing had shown this to be a problem with Level3, however they are advising this to be part of a wider problem within the UK. We are currently monitoring and will look at re-routing traffic if this continues, however this … Continue reading “LON01 – 12/06/2015 *UK Routing Issue* *RESOLVED*”

We are aware of a issue affecting connectivity to multiple networks within the UK. Initial testing had shown this to be a problem with Level3, however they are advising this to be part of a wider problem within the UK.

We are currently monitoring and will look at re-routing traffic if this continues, however this would involve shutting down several of our peers and affecting traffic via unaffected links. This would be a last resort option.

Once we know more we will advise.

UPDATE 01 – 11:11

Level3 have updated us to advise they are now tracking a global routing issue. This now seems to be isolated to Level3, however will be affecting any network that peer with Level3. Re-routing traffic is still be an option, however may have no impact to networks that rely on Level3.

UPDATE 02 – 11:36

Our network monitoring has alerted us to routing failures to common websites such as the BBC, Facebook & Google. We have taken the decision to shutdown our Level3 peering with Adapt. While this has had some positive impact, 3rd party networks that are using Level3 will still be having issues.

UPDATE 03 – 11:55

Level3 have advised that services are starting to be restored. We have re-enabled our Level3 peer via Adapt and have noticed they also took the decision to showdown there link to Level3. Traffic are currently routing via Cogent via Adapt.

UPDATE 04 – 20:00

Outage report from a 3rd party monitoring network: http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/

LON01 – 07-06-2015 – 19:00 – 22:00 – Network Maintenance (IPv6) *COMPLETE*

In the above maintenance window we will be adding IPv6 support back to ACCESS01.LON01 after its removal during our previous network upgrades. We will also be enabling IPv6 on CORE02.LON01 to support IPv6 on our upcoming DSL network. IPv6 is already available on CORE01.LON01 for directly connected services and leased line circuits. No disruption is … Continue reading “LON01 – 07-06-2015 – 19:00 – 22:00 – Network Maintenance (IPv6) *COMPLETE*”

In the above maintenance window we will be adding IPv6 support back to ACCESS01.LON01 after its removal during our previous network upgrades. We will also be enabling IPv6 on CORE02.LON01 to support IPv6 on our upcoming DSL network. IPv6 is already available on CORE01.LON01 for directly connected services and leased line circuits.

No disruption is expected to IPv4 traffic or the network general.

UPDATE 01 – 19:00
Engineers have started the works.

UPDATE 02 – 19:50
IPv6 BGP sessions to our primary peer “Adapt” have been established along with our tertiary peer “Zen” Prefixes are being received on CORE01.lON01. A BGP session will now be created between CORE01.LON & CORE02.LON01

UPDATE 03 – 20:30
A BGP session between CORE01.LON & CORE02.LON01 has been created and is now in testing.

UPDATE 04 – 20:45
Testing has completed OK, however there is a problem with routing traffic to both Adapt & Zen. We are unable to pass IPv6 traffic to Adapt and we are not receiving a default route from Zen. Tickets have been raised with each carrier. At this moment in time we are not passing IPv6 traffic due to this fault. If a quick resolve is not provided then we will drop the effected peers.

UPDATE 05 – 21:20
While waiting for responses from upstream peers, we continued to investigate and discovered the ranges we where testing with where not announced within our internal V6 BGP session due to the use of a GRE tunnel. Network statements have been added and traffic is flowing via Adapt without issue from our test locations. No traffic loss would have been seen other than from our test locations and retract the previous statement about not passing traffic. We are still awaiting a response from Zen.

UPDATE 06 – 21:45
Due to time lost we have suspended the integration of ACCESS01.LON01. While technically there is less work to complete on this and could be completed within the window. We feel this should be rescheduled to avoid human error given this late hour and the concentration required when its comes to BGP!

Internal Systems *RESOLVED*

We are currently experiencing issues with some internal systems that is preventing various customer service / support operations. Engineers are on route to the data centre, however local network rail cancellations and disruptions are delaying engineers from attending. Please also note the above is effectiving email delivery so we are unable to process support tickets. … Continue reading “Internal Systems *RESOLVED*”

We are currently experiencing issues with some internal systems that is preventing various customer service / support operations. Engineers are on route to the data centre, however local network rail cancellations and disruptions are delaying engineers from attending.

Please also note the above is effectiving email delivery so we are unable to process support tickets. We advise calling for any urgent issues.

Apologies for any inconvenience this may cause. Once we know more we will update this notice.

UPDATE01 – 12.00
Engineers have arrived on site.

UPDATE 02 – 12.30
Internal systems have been restored

LON01 – 26-05-2015 – 20:00 – 20:15 – Network Maintenance (BGP Filters) *COMPLETE*

We will be installing 2 new internal outbound BGP route maps on CORE01.LON01 & CORE02.LON01 tonight between 20:00 – 20:15. This work is to further improve BGP security internally between cores. The new rules should not cause any BGP session resets or any loss of existing known routes from upstream peers, however we may reset … Continue reading “LON01 – 26-05-2015 – 20:00 – 20:15 – Network Maintenance (BGP Filters) *COMPLETE*”

We will be installing 2 new internal outbound BGP route maps on CORE01.LON01 & CORE02.LON01 tonight between 20:00 – 20:15.

This work is to further improve BGP security internally between cores.

The new rules should not cause any BGP session resets or any loss of existing known routes from upstream peers, however we may reset the sessions to ensure they are working correctly. This will cause traffic to re-route for a few moments before returning to normal.

UPDATE 01 – 20:20
This window has been extended until 21:00 due to a late start by engineers.

UPDATE 02 – 20:27
CORE01.LON01 has been updated and the filters tested without the need for a full session reset.

UPDATE 03 – 20:35
CORE02.LON01 has been updated and the filters tested without the need for a full session reset.

Network monitoring shows all routing tables have returned to normal however engineers will monitor for the next few hours to ensure all routing tables remain consistent between Cores.

UPDATE 04 – 21:30
The routing tables on both CORE01.LON01 and CORE02.LON01 have remained constant for the past hour and we are happy to fully close this maintenance window.

LON01 – 06-05-2015 – Adapt / Level3 Transit Outage *Resolved*

At 10:01 our network monitoring detected a loss of IP transit to our primary carrier Adapt / Level3. Engineers have shutdown this link and traffic is routing via alternative carriers. We will continue to monitor and update as more information becomes available. We apologize for any inconvenience caused. UPDATE 01 – 11:01 We are continuing … Continue reading “LON01 – 06-05-2015 – Adapt / Level3 Transit Outage *Resolved*”

At 10:01 our network monitoring detected a loss of IP transit to our primary carrier Adapt / Level3. Engineers have shutdown this link and traffic is routing via alternative carriers. We will continue to monitor and update as more information becomes available.

We apologize for any inconvenience caused.

UPDATE 01 – 11:01
We are continuing to try and contact Adapt for an update as there are no issues with the Level3 side of the link. We believe from external testing there is a routing failure within the Adapt network.

Traffic is continuing to route via other links, however due to the volume of traffic being observed latency may be higher than normal. Should this continue then we will look to re-route traffic via our 3rd interconnect on CORE01 that has a higher capacity link.

UPDATE 02 – 11:18
We have been advised that the issue has been resolved, however we are seeking further confirmation before we bring the peer back in to service.

UPDATE 03 – 11:18
We have spoken to the senior network architect at Adapt who has confirmed service has been restored and we have re-enabled the session. We are observing normal traffic flow, however will continue to monitor the link. Once an RFO has been published by Adapt we will provide this.