Author: Adam Clark (Network Operations)

LON01 – 17/02/2016 – 23:00 till 03:00 – EasyXDSL – Maintenance *COMPLETE*

We have been advised by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 on the 17/02/2016 from 23:00 – 03:00. This upgrade work will affect one of our fibre waves we use for DSL termination back in … Continue reading “LON01 – 17/02/2016 – 23:00 till 03:00 – EasyXDSL – Maintenance *COMPLETE*”

We have been advised by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 on the 17/02/2016 from 23:00 – 03:00. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at Goswell Road. DSL traffic traversing over this link will drop during the maintenance window, however should re-establish over an alternative link via Telehouse East which is unaffected.

The nature of these works include a reload of the Junos hardware at Telehouse North by our wave supplier and in theory should be the only disruption we see. Unfortunately sort disruptions like this will cause multiple DSL PPP session re-terminations as they flip back and forth between Telehouse North and Telehouse East as our network is designed to use the former where possible. Due this concern and avoiding any other unforeseen issues that will cause session drops during the maintenance windows, we will be manually shutting down the Telehouse North link at 22:00 on the night with the view of bringing it backup at 07:00 the following morning. This will limit the session drops to these times and provide a stable service throughout.

UPDATE01 – 17/02/2016 – 22:00
Traffic has been re-routed

UPDATE02 – 18/02/2016 – 09:05
Traffic has been re-routed

BT Outage – DSL Authentication Issue – 02/02/2016 – 16:25

We are aware of the national BT DSL outage that is sweeping social media. In addition to this we are now starting to see wholesale DSL services affected, whereby if a connection looses sync or PPP then it wont recover. Further diagnostics shows that BT are not delivering the session to us correctly and is … Continue reading “BT Outage – DSL Authentication Issue – 02/02/2016 – 16:25”

We are aware of the national BT DSL outage that is sweeping social media. In addition to this we are now starting to see wholesale DSL services affected, whereby if a connection looses sync or PPP then it wont recover. Further diagnostics shows that BT are not delivering the session to us correctly and is also affecting other wholesale ISPs.

LON01 – EasyIPT – 28/01/2016 – 15:55 – Voice Disruption *Resolved*

We are aware of an issue affecting inbound and outbound call routing on our network that started at 15:55 and are investigating as a matter of urgency. Initial diagnostics and reports show this to be an upstream carrier issue. UPDATE01 – 16:10 Due to the nature of the outage, Automatic outbound re-routing to an alternative … Continue reading “LON01 – EasyIPT – 28/01/2016 – 15:55 – Voice Disruption *Resolved*”

We are aware of an issue affecting inbound and outbound call routing on our network that started at 15:55 and are investigating as a matter of urgency. Initial diagnostics and reports show this to be an upstream carrier issue.

UPDATE01 – 16:10 Due to the nature of the outage, Automatic outbound re-routing to an alternative carrier has been intermittent as the carrier suffering the issue has been going on and offline. We have therefore manually re-routed outbound calls and can see them terminating OK. We are continuing to investigate.

UPDATE02 – 16:15 Further investigations are showing our BGP session to the affected upstream carrier as being intermittent. While reviewing the link we have received notice from 1 of our fibre providers that they have suffered a fibre issue in the Docklands area and this is causing there network to re-converge. Outbound calls continue to terminate correctly, however inbound calls via this upstream provider may be intermittent.

UPDATE03 – 16:45 We have seen the link stabilise and inbound calls from the carrier terminating OK. We are continuing to route outbound calls over alternative carriers as a precaution.

LON01 – 29/01/2016 – 21:00 till 21:30 – EasyXDSL – Planned Maintenance *COMPLETE*

On the Friday the 29th of January between 21:00 & 21:30 we will be upgrading our LNS servers to the latest vendor software release. This work will start by upgrading lns02.dsl.structuredcommunications.co.uk first. We will then drop some DSL sessions on lns01.dsl.structuredcommunications.co.uk to ensure they properly re-terminate on LNS02 with traffic flow as expected. We will … Continue reading “LON01 – 29/01/2016 – 21:00 till 21:30 – EasyXDSL – Planned Maintenance *COMPLETE*”

On the Friday the 29th of January between 21:00 & 21:30 we will be upgrading our LNS servers to the latest vendor software release.

This work will start by upgrading lns02.dsl.structuredcommunications.co.uk first. We will then drop some DSL sessions on lns01.dsl.structuredcommunications.co.uk to ensure they properly re-terminate on LNS02 with traffic flow as expected. We will then upgrade LNS01, dropping all sessions once complete. Sessions should re-terminate on LNS01. Any sessions that remain on LNS02 after the upgrade ,will be dropped and moved back to LNS01.

UPDATE01 – 29/01/2016 – 20:05 This work is complete. If you still dont have service then please restart your equipment. Some device may need to be powered down for 20 minutes to clear any stale or stuck sessions.

LON01- Planned Maintenance: Telehouse East Virgin Media Circuits – Tuesday 19th Jan 2015 between 0:01 – 06:00 *COMPLETE*

We have been advised that our wholesale provider will be moving some of our Virgin Media circus to a new POP as part of planned capacity upgrade works at Telehouse East on Tuesday 19th Jan 2015 between 0:01 – 06:00. Circuits with backup will continue service operation with the only interruption seen when the main … Continue reading “LON01- Planned Maintenance: Telehouse East Virgin Media Circuits – Tuesday 19th Jan 2015 between 0:01 – 06:00 *COMPLETE*”

We have been advised that our wholesale provider will be moving some of our Virgin Media circus to a new POP as part of planned capacity upgrade works at Telehouse East on Tuesday 19th Jan 2015 between 0:01 – 06:00.

Circuits with backup will continue service operation with the only interruption seen when the main service goes offline.

We apologies for any inconvenience this may cause.

LON01 – 04/01/2016 – 11:50 – EasyXDSL – PPP Sessions *Resolved*

We are aware that some DSL PPP sessions have dropped and are not re-establishing. We are working with our wholesale supplier to try and resolve the situation. UPDATE01 – 12:20 We have have tracked down the fault to circuits that are provided to us as “off-net” circuits from Openreach. We have raised an escalation with … Continue reading “LON01 – 04/01/2016 – 11:50 – EasyXDSL – PPP Sessions *Resolved*”

We are aware that some DSL PPP sessions have dropped and are not re-establishing. We are working with our wholesale supplier to try and resolve the situation.

UPDATE01 – 12:20

We have have tracked down the fault to circuits that are provided to us as “off-net” circuits from Openreach. We have raised an escalation with our supplier and and working to resolve this as a priority.

UPDATE02 – 12:25

We are stating to see off-net circuits re-establish. We have not yet officially received a clear from our wholesale provider, however are pushing for a update.

UPDATE03 – 12:38

We have spoken to our wholesale NOC engineers who have advised that they added a new DSL gateway in to there network pool following a failure of an existing device. Unfortunately this device was not announced to our network by our supplier within our DSL BGP session, Therefore we where unable to route PPP responses back to this device. The device has been removed from our suppliers device pool and services are terminating back to us via known L2TP gateways. Service has now returned to normal. We apologize for any inconvenience caused.

2015 Christmas Network Freeze Information

During the Christmas holiday period several of our suppliers will implement network freezes during which non-essential work will be restricted. They are as follows: BT: No formal freeze, but staffing levels will be reduced over the Christmas period. Vodafone: From 00:01 on Friday 11th December until 00:01 on Monday 4th January. TalkTalk Business: From Wednesday … Continue reading “2015 Christmas Network Freeze Information”

During the Christmas holiday period several of our suppliers will implement network freezes during which non-essential work will be restricted. They are as follows:

BT:

No formal freeze, but staffing levels will be reduced over the Christmas period.

Vodafone:

From 00:01 on Friday 11th December until 00:01 on Monday 4th January.

TalkTalk Business:

From Wednesday 16th December until Tuesday 5th January. From Wednesday 16th December to Friday 18th December 2015 no Medium & High Risk work will be allowed to proceed. Saturday 19th December 2015 to Monday 4th January 2016 only emergency fault fixes will be allowed.

Virgin Media:

From 17:00 on Friday 11th December until 09:00 on Monday 4th January.

Level 3:

Freeze starts 00:01 on Saturday 12th December and ends 23:59 on Sunday 3rd January 2016. During this period only essential repairs and maintenance will be permitted.

Colt:

Freeze starts at 18:00 on Friday 18th December until 23:59 on Sunday 3rd January 2016. During this period only emergency repairs and maintenance will be permitted.

Structured:

Provisioning / Support will be business as usual until 17:00 on Wednesday 23rd December. From that time until 09:00 on Monday 4th January no new installs, routine maintenance or internal network changes will be performed. However, any emergency maintenance required during this time will be performed as normal and advised of through the usual channels.

LON01 – 07/12/2015 – 23:00 till 03:00 – EasyXDSL – Maintenance

We have been advised by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 on the 07/12/2015 from 23:00 – 03:00. This upgrade work will affect one of our fibre waves we use for DSL termination back in … Continue reading “LON01 – 07/12/2015 – 23:00 till 03:00 – EasyXDSL – Maintenance”

We have been advised by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 on the 07/12/2015 from 23:00 – 03:00. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at Goswell Road. DSL traffic traversing over this link will drop during the maintenance window, however should re-establish over an alternative link via Telehouse East which is unaffected.

The nature of these works include a reload of the Junos hardware at Telehouse North by our wave supplier and in theory should be the only disruption we see. Unfortunately sort disruptions like this will cause multiple DSL PPP session re-terminations as they flip back and forth between Telehouse North and Telehouse East as our network is designed to use the former where possible. Due this concern and avoiding any other unforeseen issues that will cause session drops during the maintenance windows, we will be manually shutting down the Telehouse North link at 22:00 on the night with the view of bringing it backup at 07:00 the following morning. This will limit the session drops to these times and provide a stable service throughout.

We apologize for any inconvenience this may cause.

UPDATE 01 – 21:50 We have removed LNS02 from the available pool. This should aid in services re-terminating quickly on LNS01.

UPDATE 02 – 22:00 We have shutdown our Telehouse North link, with traffic re-routing over to Telehouse East as expected

LON01 – 17/11/2015 – 13:50 – Leased Lines – *Vodafone incident* *Resolved*

We are currently aware of an issue affecting a number of on-net and off-net Vodafone based circuits. We will continue to work with our suppliers and publish updates as they become available. UPDATE 01 – 12:10 We have seen sessions restore, however are still pending an update from Vodafone. UPDATE 02 – 12:40 Telecity have … Continue reading “LON01 – 17/11/2015 – 13:50 – Leased Lines – *Vodafone incident* *Resolved*”

We are currently aware of an issue affecting a number of on-net and off-net Vodafone based circuits.

We will continue to work with our suppliers and publish updates as they become available.

UPDATE 01 – 12:10
We have seen sessions restore, however are still pending an update from Vodafone.

UPDATE 02 – 12:40
Telecity have confirmed that this incident was caused by an isolated power issue. Telecity engineers are working to ensure the underlying issue is fully resolved. Until we receive confirmation from Telecity that all work has completed, previously affected services should be considered as at risk.

Further updates will be provided as they are made available from our suppliers.

UPDATE 03 – 14:57
We have been advised via another supplier of the following:

We are aware of an issue at Telecity Sovereign House. The issue has been confirmed to be a wide-spread power outage affecting the ground floor up to, and including the 4th floor. The power appears to be currently intermittently available and engineers are on site investigating the source of the problem. At this time, the DC are unable to advise the cause or why resilient systems failed to take over.

LON01 – 11/11/2015 – 23:00 till 03:00 – EasyXDSL – Emergency Maintenance *COMPLETE*

We have been advised this afternoon by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 tonight. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at … Continue reading “LON01 – 11/11/2015 – 23:00 till 03:00 – EasyXDSL – Emergency Maintenance *COMPLETE*”

We have been advised this afternoon by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 tonight. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at Goswell Road. DSL traffic traversing over this link will drop during the maintenance window, however should re-establish over an alternative link via Telehouse East which is unaffected.

The nature of these works include a reload of the Junos hardware at Telehouse North by our wave supplier and in theory should be the only disruption we see. Unfortunately sort disruptions like this will cause multiple DSL PPP session re-terminations as they flip back and forth between Telehouse North & Telehouse East as our network is designed to use the former where possible. Due this concern and avoiding any other unforeseen issues that will cause session drops during the maintenance windows, we will be manually shutting down the Telehouse North link at 22:50 tonight with the view of bringing it backup at 07:00 tomorrow morning. This will limit the session drops to these times and provide a stable service throughout.

We apologise for any inconvenience this may cause.

UPDATE 01 – 18:45

We have reviewed the original manual switch over time and concluded it does not allow any contingency should this be required. We have therefor updated this to 22:00
UPDATE 02 – 07:10 – 12/11/2015

This work is complete.