Adam Clark (Network Operations) – Page 15 – Structured Communications NOC

LON01 – 17/11/2015 – 13:50 – Leased Lines – Vodafone incident Resolved

We are currently aware of an issue affecting a number of on-net and off-net Vodafone based circuits. We will continue to work with our suppliers and publish updates as they become available. UPDATE 01 – 12:10 We have seen sessions restore, however are still pending an update from Vodafone. UPDATE 02 – 12:40 Telecity have … Continue reading “LON01 – 17/11/2015 – 13:50 – Leased Lines – *Vodafone incident* *Resolved*”

We are currently aware of an issue affecting a number of on-net and off-net Vodafone based circuits.

We will continue to work with our suppliers and publish updates as they become available.

UPDATE 01 – 12:10
We have seen sessions restore, however are still pending an update from Vodafone.

UPDATE 02 – 12:40
Telecity have confirmed that this incident was caused by an isolated power issue. Telecity engineers are working to ensure the underlying issue is fully resolved. Until we receive confirmation from Telecity that all work has completed, previously affected services should be considered as at risk.

Further updates will be provided as they are made available from our suppliers.

UPDATE 03 – 14:57
We have been advised via another supplier of the following:

We are aware of an issue at Telecity Sovereign House. The issue has been confirmed to be a wide-spread power outage affecting the ground floor up to, and including the 4th floor. The power appears to be currently intermittently available and engineers are on site investigating the source of the problem. At this time, the DC are unable to advise the cause or why resilient systems failed to take over.

LON01 – 11/11/2015 – 23:00 till 03:00 – EasyXDSL – Emergency Maintenance COMPLETE

We have been advised this afternoon by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 tonight. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at … Continue reading “LON01 – 11/11/2015 – 23:00 till 03:00 – EasyXDSL – Emergency Maintenance *COMPLETE*”

We have been advised this afternoon by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 tonight. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at Goswell Road. DSL traffic traversing over this link will drop during the maintenance window, however should re-establish over an alternative link via Telehouse East which is unaffected.

The nature of these works include a reload of the Junos hardware at Telehouse North by our wave supplier and in theory should be the only disruption we see. Unfortunately sort disruptions like this will cause multiple DSL PPP session re-terminations as they flip back and forth between Telehouse North & Telehouse East as our network is designed to use the former where possible. Due this concern and avoiding any other unforeseen issues that will cause session drops during the maintenance windows, we will be manually shutting down the Telehouse North link at 22:50 tonight with the view of bringing it backup at 07:00 tomorrow morning. This will limit the session drops to these times and provide a stable service throughout.

We apologise for any inconvenience this may cause.

UPDATE 01 – 18:45

We have reviewed the original manual switch over time and concluded it does not allow any contingency should this be required. We have therefor updated this to 22:00
UPDATE 02 – 07:10 – 12/11/2015

This work is complete.

LON01 – EasyXDSL – 22/10/2015 – 13:39 – MTU Issue Resolved

We are aware of a MTU issue on our DSL network affecting lns01.dsl.structuredcommunications.co.uk. This is preventing JUMBO FRAMES from getting to this device and forcing DSL circuits that terminate on our network to obtain a lower MTU (1460-1462) Where end users CPEs and networks are not configured to handle fragmentation correctly this will manifest itself … Continue reading “LON01 – EasyXDSL – 22/10/2015 – 13:39 – MTU Issue *Resolved*”

We are currently working to resolve the problem.

We apologise for any inconvenience this may cause.

UPDATE 01 – 13:56We have isolated the problem to a interconnecting VLAN between LNS01 & CORE02. The VLAN will need to be reconfigured, however this will drop all active DSL sessions that terminate on that device. We are looking at other work around at this time.

UPDATE 02 – 14:20 We have reconfigured the VLAN and interface which has resolved the issue with JUMBO frames and our carriers. Existing L2TP tunnels to our carriers LACs are still negotiated at a lower MTU. Any existing sessions and new session that join that tunnel will still have a lower than normal MTU.

UPDATE 03 – 16:00 Tunnels have been reset and sessions restored. MTUs are now being seen at 1500 again for ADSL.

LON01 – 14/10/2015 – 10:00 – CORE01 & CORE02 – Maintenance COMPLETE

At the above time today, we will be replacing 1 line card in each of the above cores with a newer revision to facilitate an update on LNS01 & LNS02. Any traffic on these cards will be re-routed to there backup counterpart . No outage is expected however a drop in call traffic may be … Continue reading “LON01 – 14/10/2015 – 10:00 – CORE01 & CORE02 – Maintenance *COMPLETE*”

At the above time today, we will be replacing 1 line card in each of the above cores with a newer revision to facilitate an update on LNS01 & LNS02.

Any traffic on these cards will be re-routed to there backup counterpart . No outage is expected however a drop in call traffic may be observed.This devices should be considered at risk until complete.

UPDATE01 – 09:55
Work has been undertaken to re-route traffic on CORE02. As expected we have seen a small drop in call traffic. We apologize for any inconvenience caused. Traffic will need to be re-routed, however will be done at a time when call levels are lower.

UPDATE01 – 11:30
Work has been completed on both cores and any primary services reconnected.

LON01 – 15/10/2015 – 20:00 – All Services – Planned Maintenance. Complete

We have been advised of planned network maintenance by one of our primary transit providers (Adapt) This work is due to retirement of access41.lon2 and the need to move the network equipment into new core cabinet within Goswell Road. We have been advised works will start at 20:00 and continue on until 04:00 on the … Continue reading “LON01 – 15/10/2015 – 20:00 – All Services – Planned Maintenance. *Complete*”

We have been advised works will start at 20:00 and continue on until 04:00 on the 16th. We take multiple feeds from adapt for transit and have been advised of service disruption while these are physically relocated to the new device of up to 20 minutes. Although we take multiple redundant feeds, we have been advised of disruption due to network re-convergence on there side.

Our network is comprised of multiple redundant routes to different carriers. Should we loose our Adapt link for any reason then traffic will automatically re-route with minimal disruption.

This work wont affect phone services as these route in to our network via interdependent private interconnects unless you connect in to our network via a 3rd party connection. If this is the case you may need to restart your handsets after the maintenance works has been completed.

UPDATE01 – 16/10/2015
This work completed without issue from our supplier. Our network reacted as expected and re-routed traffic where required.

Easy xDSL – Multiple Exchange Outages RESOLVED

We have been made aware of multiple exchanges outages in Sussex affecting 20CN services across multiple providers. At this moment no ETA has been given and we are awaiting further information from our suppliers. We apologize for any inconvenience caused and will provide further updates as they become available. UPDATE 01 – 08:34 We have … Continue reading “Easy xDSL – Multiple Exchange Outages *RESOLVED*”

We apologize for any inconvenience caused and will provide further updates as they become available.

UPDATE 01 – 08:34
We have been advised that hardware at a patent exchange has been replaced and services should start to restore. Any client still having issues are advised to power cycle there equipment.

LON01 – EasyHTTP – 14/09/2015 – 20:00 – 20:30 – Maintenance COMPLETE

Between 20:00 & 20:30 we will be installing a new version of our mail server on EasyHTTP. IMAP, SMTP & POP will be unavailable during this maintenance window. UPDATE 01 – 20:19 This work is complete.

Between 20:00 & 20:30 we will be installing a new version of our mail server on EasyHTTP. IMAP, SMTP & POP will be unavailable during this maintenance window.

UPDATE 01 – 20:19

This work is complete.

TalkTalk Circuts – 15-07/2015 – 09:45 RESOLVED

Our network monitoring has alerted us to increased latency and packet loss on some Talk Talk EFM circuits. We have raised a fault with our supplier and are currently awaiting there response. UPDATE01 – 10:55 Our network monitoring is showing a reduction in packet loss and latency, however we have not had an official update. … Continue reading “TalkTalk Circuts – 15-07/2015 – 09:45 *RESOLVED*”

Our network monitoring has alerted us to increased latency and packet loss on some Talk Talk EFM circuits. We have raised a fault with our supplier and are currently awaiting there response.

UPDATE01 – 10:55
Our network monitoring is showing a reduction in packet loss and latency, however we have not had an official update.

UPDATE02 – 11:50
Our network monitoring is again showing latency increasing along with packet loss. We have chased for an update.

UPDATE03 – 13:13
Our supplier has advised this is being caused by an overloaded transit link due to windows updates. We have escalated this response to there management as this level of service should not be contended on a wholesale level. We are awaiting there response.

UPDATE04 – 15:00
Latency has returned back to normal and packet loss has gone. We are still awaiting an official clear and response to our escalation.

LON01 – EasyIPT – 13/07/2014 – 20:00 till 21:00 Planned Maintenance COMPLETE

We are aware some users have been seeing stuck SIP sessions today, whereby “BYE” messages have not been completely clearing down resulting in accounts hitting a concurrent call limit. A work around has been put in place for this until this evening when we can reload the core device to drop any sessions still in … Continue reading “LON01 – EasyIPT – 13/07/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*”

UPDATE01 – 20:05
This work has started and the server has been requested to reload.

UPDATE02 – 20:15
The server has reloaded and is accepting SIP requests, however is not processing any call routing.

UPDATE03 – 20:35
We have discovered that the Kamailio Proxy service has failed to start which is why calls are not being processed. We are working to restore the service.

UPDATE04 – 20:40
The service has been restored and test calls are now routing OK.

LON01 – 01/07/2015 – 20:00 – 21:00 – EasyBond – Maintenance COMPLETE

During this window we will be upgrading our bonded platform to the latest software version as per our software vendors recommendation. Customers on the platform may notice drops in service while this takes place. Any customer using the service with our EasyIPT VOIP solution may notice there handsets fail to register with status lights remaining … Continue reading “LON01 – 01/07/2015 – 20:00 – 21:00 – EasyBond – Maintenance *COMPLETE*”

During this window we will be upgrading our bonded platform to the latest software version as per our software vendors recommendation. Customers on the platform may notice drops in service while this takes place. Any customer using the service with our EasyIPT VOIP solution may notice there handsets fail to register with status lights remaining orange. If this occurs please power down all hardware for 5 minutes and power back up once the maintenance window is marked as complete.

UPDATE01 – 21:06
This work is complete.