Author: Adam Clark (Network Operations)

LON01 – EasyXDSL – 14/11/2016 – 23:30 – 0600 – *Maintenance*

We have been advised by one of our wholesale providers that they will be replacing hardware within there core network tonight that will physically disconnect one of our host-link cables we use to terminate DSL traffic with BT. To avoid any unnecessary PPP resets to DSL sessions we will be manually re-routing traffic via alternative … Continue reading “LON01 – EasyXDSL – 14/11/2016 – 23:30 – 0600 – *Maintenance*”

We have been advised by one of our wholesale providers that they will be replacing hardware within there core network tonight that will physically disconnect one of our host-link cables we use to terminate DSL traffic with BT. To avoid any unnecessary PPP resets to DSL sessions we will be manually re-routing traffic via alternative paths before the work starts. No outages or session drops are expected, however DSL services should be classed as “at risk” until these works are fully completed.

UPDATE01 – 09:00

This work was completed as expected and we have restored traffic over this hostlink.

LON01 – EasyLL (Leased Lines) – Core01 – 10/11/2016 – 00:00 – 06:00 – Maintenance

We have been advised by one of our wholesale providers they will be undertaking upgrade works on part of there core in Goswell Road that delivers some of our Leased Lines and Transit services. This upgrade work involves replacement of their hardware and will directly affect any services we provide via core01.lon01.structuredcommunications.co.uk. Customers who take … Continue reading “LON01 – EasyLL (Leased Lines) – Core01 – 10/11/2016 – 00:00 – 06:00 – Maintenance”

We have been advised by one of our wholesale providers they will be undertaking upgrade works on part of there core in Goswell Road that delivers some of our Leased Lines and Transit services. This upgrade work involves replacement of their hardware and will directly affect any services we provide via core01.lon01.structuredcommunications.co.uk. Customers who take backup DSL from us will see see there connection automatically re-route via this as required. Customers who take redundant fibre connections to our border only or customers without backup will see up-to a 30 minutes downtime.

Peering an transit services will also automatically re-route via alternative paths on our network.

LON01 – EasyIPT – 12/10/2016 – 10:45 *Inbound Calls*

We are aware one of our upstream providers is suffering a network issue affecting a small number of 020 prefixed inbound calls routed via them. This issue has been raised with the respective team and they are dealing with it as a priority. We apologies for any inconvenience caused and will continue to follow this … Continue reading “LON01 – EasyIPT – 12/10/2016 – 10:45 *Inbound Calls*”

We are aware one of our upstream providers is suffering a network issue affecting a small number of 020 prefixed inbound calls routed via them.

This issue has been raised with the respective team and they are dealing with it as a priority.

We apologies for any inconvenience caused and will continue to follow this up with them posting updates as they become available.

UPDATE 01 – 11:45

We are starting to see inbound calls connect again from the affected 020 numbers. We have not been officially advised this has been resolved but services from this carrier are starting to restore.

LON01 – EasyIPT – 13/09/2016 – 10:07 *Outbound Calls*

We are aware of a problem with outbound calls. This has been isolated to an upstream carrier and removed from our carrier pool. Traffic is re-routing as expected. We apologise for any inconvenience caused. UPDATE01 – 10:30 Discussions with this upstream carrier have uncovered our account being suspended incorrectly which was re-enabled promptly when there … Continue reading “LON01 – EasyIPT – 13/09/2016 – 10:07 *Outbound Calls*”

We are aware of a problem with outbound calls. This has been isolated to an upstream carrier and removed from our carrier pool. Traffic is re-routing as expected.

We apologise for any inconvenience caused.

UPDATE01 – 10:30
Discussions with this upstream carrier have uncovered our account being suspended incorrectly which was re-enabled promptly when there mistake was discovered. Further discussions are being had with there management as no notice was given to ourselves at the time.

We have also discovered the upstream carrier was sending back a 404 error to our softwitch. This error code prevented our system for trying another route as a 404 is “not found” or number incorrect.

UPDATE02 – 11:44
We have bought this carrier back in to service and as above are conducting a review with there management team in to there actions.

UPDATE03 – 10:03
We are aware this issue reoccurred again this morning between the hours of 09:30 and 10:00. The same upstream carrier was at fault and we have further escalations in place for a response.

LON01 – EasyXDSL Fibre Hostlink– 06/09/2016 – 23:00

We have been advised by one of our fibre wave providers that they will be conducting some emergency maintenance on some fibre interconnects which includes a circuit used by ourselves. To avoid any unexpected issues we will manually re-route traffic and flag that link as unavailable. Some PPP sessions may drop and re-establish for DSL … Continue reading “LON01 – EasyXDSL Fibre Hostlink– 06/09/2016 – 23:00”

We have been advised by one of our fibre wave providers that they will be conducting some emergency maintenance on some fibre interconnects which includes a circuit used by ourselves.

To avoid any unexpected issues we will manually re-route traffic and flag that link as unavailable. Some PPP sessions may drop and re-establish for DSL circuits using this link.

UPDATE 01 – 22:23

Traffic has been re-routed. We where able to maintain reachability to our L2TP endpoints, as such no PPP session drops where seen. The link will be bought back in to service once we have received a clear from that wave provider.

LON01 – CORE03 – 24/08/2016 – 20:00 Till 23:59 *Emergency Work *

Following on from recent repeated hardware failures on core03.structuredcommunications.co.uk as detailed HERE the decision has been taken to fully replace the device. We will also be taking the opportunity to upgrade the IOS image to bring this device in line with the current images across the rest of our network. This work will involve powering … Continue reading “LON01 – CORE03 – 24/08/2016 – 20:00 Till 23:59 *Emergency Work *”

Following on from recent repeated hardware failures on core03.structuredcommunications.co.uk as detailed HERE the decision has been taken to fully replace the device. We will also be taking the opportunity to upgrade the IOS image to bring this device in line with the current images across the rest of our network.

This work will involve powering down and physically moving the current device along with all installed line cards. Due to this all directly connected services (listed below) will be unavailable for the duration of the works.

> Bonded DSL on AG1 & AG2

> un-managed SIP trunking provided via sipwise.easyipt.co.uk

> Managed VoIP services provided via primary-sw.r03.core03 & primary-sw.r04.core03

> Webhosting via server01.easyhttp.co.uk

> VPS sessions on esxi10.r02.structuredcommunications.co.uk

Other services will remain unaffected. Redundant services provided via other parts of the network (Such as DNS & SMTP) will take over. Please ensure you configuration is up to date.

UPDATE01 – 20:10 – 24/08/2016 Engineers are on site and these works have started.

UPDATE02 – 22:06 – 24/08/2016 Engineers have completed the above works ahead of schedule and we can confirm all services have returned to normal. We apologize for the inconvenience caused.

We will continue to monitor the new device to ensure continued operation.

LON01 – CORE03 – 21/08/2016 – 08:33 *At Risk*

Our network monitoring has alerted us to a fault on CORE03 within our Goswell Road network. This fault is a re-occurrence of an issue identified yesterday that was resolved without impact. Additional logging was added at the time to further assist should it be required. The issue has been tracked down to the “Ethernet Out … Continue reading “LON01 – CORE03 – 21/08/2016 – 08:33 *At Risk*”

Our network monitoring has alerted us to a fault on CORE03 within our Goswell Road network. This fault is a re-occurrence of an issue identified yesterday that was resolved without impact. Additional logging was added at the time to further assist should it be required.

The issue has been tracked down to the “Ethernet Out of Band Channel” (EOBC) control channel on the devices back plane.

Due to the number of line cards automatically taken out of service by device, we are currently investigating to see if this is part of a common hardware fault such as the current active supervisor module.

We diversely route our internal backhull fibre up-links across each core to insure that a single line card failure does not result in an outage. This is currently in operation however we have lost a number of links due to the fault and the device is classed as at risk along with any directly connected equipment.

We are currently reviewing the logs and will update with further information / action plan asap.

UPDATE01 – 09:49
After reviewing the logs we have concluded the next action step is to swap between active supervisors within that device. This will cause a brief outage to all services connected to that device. We will monitor the device closely after the change to see if the same issue occurs. This reload has been scheduled for 10:00 today.

UPDATE02 – 10:08
The swap completed as expected, however despite this supervisor showing OK and passing diagnostics, it failed to fully take the system load and was reverted back. We suspect this is now a possible backplane issue on this device. Further updates to follow.

UPDATE03 – 14:08
Further observations have been made and the log files reviewed at depth. At this stage we can advise the backup supervisor within CORE03 has been reporting errors however the Cisco IOS listed these as “Non-fatal” and as such have not been flagged up within our monitoring platform.

We suspect a fault had occurred on the standby supervisor which had not been picked up on by the devices internal diagnostics until we bought the card fully in to operation. This fault we suspect was having an impact on the EOBC reporting and thus causing line cards to be disabled. As the previous fault took 24 hours to resurface we are continuing to monitor. An emergency maintenance window is also going to be scheduled for CORE03 to replace the suspected failed card, along with an IOS update.

UPDATE04 – 14:30 – 22/08/2016
Despite seeing the device operate for over 24 hours without further errors, we have just observed the fault conditions triggering line cards to be disabled. We therefore now suspect this a problem with the chassis its-self and the backplane. We will now be replacing the entire device as a matter of course to prevent this escalating to an outage. Further works will be scheduled and notified via the NOC as we dont have a pre-built device on site.

EasyIPT – 0207 Numbers – 08/08/2016 – 16:56

We are aware of some intermittent inbound call issues on 0207 numbers from a single carrier. We have raised a support query with this carrier and are awaiting an update from them. At the moment we are not aware of this affecting outbound calls, nor are we aware of any other affected inbound area codes. … Continue reading “EasyIPT – 0207 Numbers – 08/08/2016 – 16:56”

We are aware of some intermittent inbound call issues on 0207 numbers from a single carrier.

We have raised a support query with this carrier and are awaiting an update from them. At the moment we are not aware of this affecting outbound calls, nor are we aware of any other affected inbound area codes. If you are having issues we would advise you to contact us so your examples can be logged to further assist with a quicker resolution.

BT Wholesale incident – 21/07/2016

We reported this morning we where seeing issues with our DSL circuits, this has now escalated to issues within BT wholesale themselves affecting multiple services to providers across the UK. We believe the same London datacenter that experienced issue yesterday is once again experiencing issues today however is now affecting other providers. Services affected include … Continue reading “BT Wholesale incident – 21/07/2016”

We reported this morning we where seeing issues with our DSL circuits, this has now escalated to issues within BT wholesale themselves affecting multiple services to providers across the UK.

We believe the same London datacenter that experienced issue yesterday is once again experiencing issues today however is now affecting other providers.

Services affected include DSL circuits dropping and not re-establishing, along with poor quality calls if they can be made / received.

Once again this is affecting providers across the UK and is not just limited to our selves, we are working with BT and other providers to assist BT in restoring services.

This is also affecting our phones and we advise clients to email support.

Once we know more we will provide an update. We apologise for any inconvenience caused.

UPDATE01 – 09:55
We are seeing call quality return to normal, however this has been officially confirmed as resolved. We are still seeing a number of DSL circuits down however.

UPDATE02 – 10:10
As advised in the previous post, we can confirm this is due to another power issue is London that has further affected BT wholesale.

UPDATE03- 10:15
Voice issues have returned and we are pushing BT wholesale for an update.

UPDATE04- 10:21
We have been advised the data centre affected is Telehouse North. We do not have any equipment of our own at Telehouse North, however key service providers do including BT which is why this issue is affecting so many providers and services. Telehouse North is a critical part of the UK infrastructure.

UPDATE05- 10:51
We have started to see all services restore, however no official conformation yet.

UPDATE06- 11:51
Services still remain stable and no further reports have been seen, however the incident has still not been closed off officially.

UPDATE07 – 12:45
This is the latest information we have received from BT about their issue: Due to a power incident at Telehouse North (London) IP Exchange Direct Access has been severely impacted since 07.45 this morning (21/07/2016).Service has been restored for all but a small subset of customers who are connected through a hardware card that requires replacing after the restoration of power. Local spares are being sought. There are still wider power restoration activities on going on site which is still expected to take until approx. 1pm BST. We will update with more information on the next update.

UPDATE08 – 13:07
No further issues have been seen and we are closing this incident for now. We will continue to monitor for the next 24 hours.

LON01 – EasyXDSL – 21/07/2016

We have been made aware of issues on BT wholesale Network whereby all customers connecting though London may experience drops in connectivity but then reconnect to our network via other gateways. If you are experiencing issues reconnecting, please power down your equipment for a few minutes to clear any stuck sessions. BT are investigating the … Continue reading “LON01 – EasyXDSL – 21/07/2016”

We have been made aware of issues on BT wholesale Network whereby all customers connecting though London may experience drops in connectivity but then reconnect to our network via other gateways.

If you are experiencing issues reconnecting, please power down your equipment for a few minutes to clear any stuck sessions.

BT are investigating the issue and updates will be provided when we have them.

We apologies any inconvenience this may cause.

UPDATE01 – 09:29

We have discovered the data centre that experienced issues yesterday for BT in respect to power may also be suffering issues again this morning.