Author: Adam Clark (Network Operations)

12-10-2017 – 01:00 till 03:00 – Planned Maintenance: Level3 – Goswell Road

Level 3 intends to carry out internal maintenance within its network. This has been classified as ESSENTIAL. The nature of this work is to perform a Network Element software upgrade. Level 3 will Upgrade BB2 ALU 7950 routers to code level SROS-14.0.R5-1 This will fix 100gb/10gb uplink error and a p-chip false alarm error. This … Continue reading “12-10-2017 – 01:00 till 03:00 – Planned Maintenance: Level3 – Goswell Road”

Level 3 intends to carry out internal maintenance within its network. This has been classified as ESSENTIAL. The nature of this work is to perform a Network Element software upgrade. Level 3 will Upgrade BB2 ALU 7950 routers to code level SROS-14.0.R5-1 This will fix 100gb/10gb uplink error and a p-chip false alarm error. This will also bring us up to date with Vendor code level recommendation.

During this window we expect to loose our Level3 peering as it requires a reload of the affected router. Traffic will automatically re-route across other paths on our network.

28-09-2017 – 23:00 – Planned Maintenance: Level3 – Goswell Road

We have been advised by Level3 that they will be replacing a line card between 23:00 on the 28/09/2017 & 02:00 on the 29/09/2017. This work will affect our 10GB interconnect with them as it involves the physical removal of hardware and re-termination of our fibre. Traffic will be automatically re-routed across other carriers during … Continue reading “28-09-2017 – 23:00 – Planned Maintenance: Level3 – Goswell Road”

We have been advised by Level3 that they will be replacing a line card between 23:00 on the 28/09/2017 & 02:00 on the 29/09/2017. This work will affect our 10GB interconnect with them as it involves the physical removal of hardware and re-termination of our fibre. Traffic will be automatically re-routed across other carriers during the works and limited impact is expected however traffic paths will change.

UPDATE 01
This work is complete

11-09-2017 – 21:00 – Planned Maintenance: BROADBAND via Telehouse North

We have been advised by one of our wholesale DSL providers that they will be upgrading backhaul capacity within Telehouse North on Monday the 11th September starting from 21:00 continuing until 06:00 on the 12th. We have been advised that this upgrade work may affect broadband (DSL and FTTC services) that are delivered to ourselves … Continue reading “11-09-2017 – 21:00 – Planned Maintenance: BROADBAND via Telehouse North”

We have been advised by one of our wholesale DSL providers that they will be upgrading backhaul capacity within Telehouse North on Monday the 11th September starting from 21:00 continuing until 06:00 on the 12th.

We have been advised that this upgrade work may affect broadband (DSL and FTTC services) that are delivered to ourselves via Telehouse North. DSL sessions we will be automatically re-routed via Telehouse East in the event of disruption. Users may suffer multiple PPP drops during the above window.

We apologize for any inconvenience caused

LON01 – EasyHTTP – Outbound SMTP – 14/08/2017

We are aware of outbound SMTP delivery issues on server01.r02.easyhttp.co.uk This has been caused by a compromised user account on this server that has been used to send out a large volume of SPAM this morning. The account has been terminated and is being dealt with however, it has resulted in temporary delivery restrictions from … Continue reading “LON01 – EasyHTTP – Outbound SMTP – 14/08/2017”

We are aware of outbound SMTP delivery issues on server01.r02.easyhttp.co.uk This has been caused by a compromised user account on this server that has been used to send out a large volume of SPAM this morning. The account has been terminated and is being dealt with however, it has resulted in temporary delivery restrictions from some remote hosts such as Yahoo.

We are monitoring the situation. Any failed emails depending on the remote host will be queued for re-delivery. If delivery restrictions remain then we will look to divert outbound email flow via another internal SMTP array on the sort term.

We apologize for any inconvenience caused.

UPDATE01 – 14:43
We are no longer seeing outbound emails queue on the server what have failed delivery due to the server reputation. Due to this we have agreed not to re-direct outbound SMTP. If users are still seeing bounce back emails then please email support and we will review on a case by case biases.

LON01 – Network Issue – 25/07/2017 – 14:05

We are aware of a network affecting issue. Engineers are currently working to identify the root cause and affected services. UPDATE 01 – 14:15 Following an audit of our network monitoring we can confirm this only affected our DSL services. Initial diagnostics show this to be a Zen WHOLESALE issue further upstream with one of … Continue reading “LON01 – Network Issue – 25/07/2017 – 14:05”

We are aware of a network affecting issue. Engineers are currently working to identify the root cause and affected services.

UPDATE 01 – 14:15

Following an audit of our network monitoring we can confirm this only affected our DSL services. Initial diagnostics show this to be a Zen WHOLESALE issue further upstream with one of our providers as we observed a number of L2TP Tunnels shutdown and BGP prefix withdrawals from THN.

A fault has been logged with their NOC department who confirmed that their was an issue that affected Wholesale L2TP subscribers along with there channel and direct retail customers.

Services have restored and we have provided our diagnostic information to there NOC. At this time we are unaware of the root cause and services should be considered at risk for now.

UPDATE 02 – 21:30

We are awaiting an RFO and have been advised no further repeats are expected.

LON01 – xDSL – 07/07/2017 *At Risk*

Network monitoring has advised us that one of wholesale L2TP fibre links are down for xDSL. This is part of our backup network and is not service affecting, however due to its non operation at this time the DSL network is classed as *At Risk* Further updates will be published once available. This is currently … Continue reading “LON01 – xDSL – 07/07/2017 *At Risk*”

Network monitoring has advised us that one of wholesale L2TP fibre links are down for xDSL. This is part of our backup network and is not service affecting, however due to its non operation at this time the DSL network is classed as *At Risk* Further updates will be published once available.

This is currently suspected to be a fibre break within one of our cross connects at Telehouse East.

UPDATE 01 – 07/07/2017 – 23:30
We have been advised by Telehouse staff that fibre cleaning works took place on our cross connect. This has restored service and we have confirmed this at both ends.

UPDATE 02 – 10/07/2017 – 09:30
Service has remained stable so we are closing this incident.

LON01 – EasyIPT – 17/06/2017 – 10:00 – 12:00

During the above window we will be replacing server components that we have identified as degraded. This will affected “Dedicated VoIP” on the affected hardware for a short period of time. We apologise for any inconvenience caused. UPDATE01 – 12:15 This work is completed and VM PBXs restored.

During the above window we will be replacing server components that we have identified as degraded. This will affected “Dedicated VoIP” on the affected hardware for a short period of time. We apologise for any inconvenience caused.
UPDATE01 – 12:15
This work is completed and VM PBXs restored.

EasyXDSL – 13/06/2017 – Packet Loss

We have been receiving reports of poor quality affecting some FTTC connections. We are currently working to identify the issue. If you are experiencing problems please call in to report the issue so we can continue to collect affected circuits. UPDATE01 – 09:39 We have collated a number of lines and reported these to our … Continue reading “EasyXDSL – 13/06/2017 – Packet Loss”

We have been receiving reports of poor quality affecting some FTTC connections. We are currently working to identify the issue.

If you are experiencing problems please call in to report the issue so we can continue to collect affected circuits.

UPDATE01 – 09:39

We have collated a number of lines and reported these to our wholesale provider who have suggested a possible hardware fault in London within there core switches. It has not been confirmed these are at fault, however we are connected to these devices.

UPDATE02 – 10:00
Ticket updated to reflect Packet Loss over call quality. Our NOC engineers are currently on a call with our supplier for an update. We apologise for the continued delay.

UPDATE03- 10:15
We are seeing packet loss clear and services are starting to recover. We have not yet received an official clear

UPDATE04- 10:15
Services remain stable. we have been advised the problem was with a 10Gbps bearer that provides a number of services from Openreach POPs. Our network monitoring showed this problem starting at 23:15 last night so we have logged a complaint with our wholesale provider to advise why this was not picked up and resolved over night prior to it becoming service affecting this AM.

Once again we apologise for any inconvenience caused and will provide an RFO on request once made available to ourselves.

LON01 – EasyIPT – 22/05/2017

Our network monitoring has just alerted us to a loss of a VOIP interconnect with one of our primary carriers. We are currently investigating the issue as a matter of priority. Calls active at the time may have dropped however traffic should have re-routed. UPDATE01 – 11:05 We can confirm traffic has re-routed as expected. … Continue reading “LON01 – EasyIPT – 22/05/2017”

Our network monitoring has just alerted us to a loss of a VOIP interconnect with one of our primary carriers. We are currently investigating the issue as a matter of priority. Calls active at the time may have dropped however traffic should have re-routed.

UPDATE01 – 11:05
We can confirm traffic has re-routed as expected. We did observe a drop is call traffic as our network adjusted to the routing changes and is not unexpected. A fault has been logged with our remote carrier as all our remote tests are passing.

UPDATE02 – 11:11
We have seen our BGP session restore along with our preferred routing. Calls remained active. We are still awaiting an update for our affected carrier.

UPDATE03 – 11:45
Our remote carrier has confirmed they are not at fault and they simply saw the same as we did in our logs.

UPDATE04 – 12:25
Our fibre wave provider has confirmed they suffered an issue with part of there network. We again apologise for any inconvenience caused.

04-05-2017 TO 11/05/2017 – Planned Maintenance: Leased Lines

We have been advised by one of our wholesale providers they will performing maintenance on there hardware that will affect our wholesale interconnects with them during the following times: Thursday 04 May 22:00 – 23:59 Monday 08 May 22:00 – 23:59 Tuesday 09 May 22:00 – 23:59 Wednesday 10 May 22:00 – 23:59 Thursday 11 … Continue reading “04-05-2017 TO 11/05/2017 – Planned Maintenance: Leased Lines”

We have been advised by one of our wholesale providers they will performing maintenance on there hardware that will affect our wholesale interconnects with them during the following times:

Thursday 04 May 22:00 – 23:59
Monday 08 May 22:00 – 23:59
Tuesday 09 May 22:00 – 23:59
Wednesday 10 May 22:00 – 23:59
Thursday 11 May 22:00 – 23:59

We have been advised downtime to be no longer than 5 minutes per circuit during one of the above windows.

Customers with backup or multiple services wont be affected.