Category: Incidents

We are currently experiencing an issue that is preventing some DSL users connecting to our service and/or constantly reconnecting to our service. Our initial investigations indicate that this is a Entanet issue and we are working with them to resolve as soon as possible. We will update with more information as it becomes available.

— Update 09:17 —
Entanet have identified a hardware failure and advise engineers have been dispatched to replace the faulty hardware. We are still working for a ETA.

— Update – 09:21 —
Any DSL bonded circuits may see reduced capacity but service should remain stable as we use 2 wholesale ISPs with our own interconnects.

— Update – 13:21 —
The issue is still currently on going, Entanet have advised engineers are still en route to replace the hardware. Due to the re routing of traffic you may experience slow speeds and/or packet loss. Please accept our apologies for any inconvenience caused and we will continue to provide updates as they become available.

— Update – 14:01 —
Entanet have advised engineers have arrived on site to replace the faulty hardware.

Our network monitoring has just alerted us to a loss of our C4L layer2 circuit to Frontier. Some active VoIP calls over this carrier / link will have dropped.

C4L have just issued a Connectivity Issue alert on their NOC.

While the interconnect has been restored we are, given historic outages on this link manually re-routing all outbound traffic away from Frontier to avoid any repeats on outbound traffic flow caused by the C4L issue.

If a re-occurrence does occur then we will drop the circuit completely which will result in our network re-routing ALL VoIP traffic over alternative links. This will have an impact while BGP re-converges and is a last resort option.

We apologies for any inconvenience this may have caused.

UPDATE01 16:01
We have been contacted by our account manager at C4L to advise this is a STP problem once again. A complaint has been issued with C4L and we continue to route outbound calls via an alternative carrier. Inbound remains via the effected link and will be transferred if the link suffers another drop. This advised will case a loss of service while BGP re-converges.

UPDATE02 16:45
The link has remained stable since the initial alert and inbound traffic has remained active without any further loss. C4L has advised that primary POPs have been restored. We will re-enable this link for outbound traffic later tonight.

UPDATE02 18:00
Outbound traffic has now been re-enabled on this link and we are monitoring. Once and RFO is provided we will issue it via the NOC.

Our network monitoring has alerted us to a loss of BGP peering with Frontier Systems. Initial calls to our Layer2 provider for this link (C4L) advise they are aware of a spanning tree issue within their network.

Traffic has automatically been re-routed around the link and no service disruption is being seen.

UPDATE01 – 10:20 – 05/10/3014
After a call to C4L following our link still being offline, we had been advised our VLAN had not been rebuilt on the trunk port where STP had been disabled. The service has now been restored and we have esculated this to our account manager.

We are aware of a DSL issue affecting our wholesale provider Entanet. This is not localised to any part of the UK but as a result people are not able to connect to the internet.

Customers on our EasyBOND platform with dual ISP reliance will be seeing a reduced service.

Once more information becomes available we will update this notice.

UPDATE01 – 20:18
We have received an update to advise Entanet are seeing an issue with telehouse-east.core.enta.net and are working to resolve the issue.

UPDATE02 – 20:25
We have seen all DSL tails return to service but are awaiting official confirmation. We have also requested an RFO from Entanet

UPDATE03 – 03/10/2014
RFO can be found HERE

At 17:11 our networking monitoring advised to a failure on our network which was localised to a single rack within LON01.

Initial diagnostics started and it was assumed there was a power issue within Rack03 as all devices on Feed A where showing as “unreachable”

Further diagnostic showed this not to be the case but a line card within LON01-CORE having been reloaded by the primary supervisor due to a series of automated test failures.

UPDATE01 – 17:18

The module has come back online and all traffic has resumed. Engineers are currently working on why this occurred.

We are currently aware of an incident on Virgin Media’s network which is causing a loss of service to several Virgin Media provided leased lines.

Virgin Media are currently investigating and updates will be provided as and when they become available.

UPDATE01 – 10:52

Virgin Media have identified a problem with one of their core routers in Telehouse and they are continuing their investigation although no ETR is available as of yet.

UPDATE02 – 11:00

We are seeing service restore back to Virgin media circuts, however this has not been confirmed by Virgin yet.

UPDATE03 – 12:00
We have requested an RFO from Virgin which we will make avalible via the NOC

At 10:52am our network monitoring alerted us to a issue with our Frontier voice interconnect. At this time we lost our BGP session to them and traffic started routing over diverse paths. This would have caused all active calls to drop while BGP reconverted.

UPDATE01 – 11:10
At 11:02 our network monitoring showed that the interconnect had been resorted and the BGP session has resumed. We have logged a case with our carrier to see if the fault originated from there side as no other session losses were seen from our side.

UPDATE02 – 11:30
We have been advised this is a carrier issues with C4L.

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service.

UPDATE01 12:50
Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier.

UPDATE02 13:23
Service has been restored; An RFO will be issued shortly. We apologise for any inconvenience caused.

UPDATE02 14:00
This issue has reoccured, the root of the problem has been identified and we are building an RFO that will be published shortly.

Outage Report
Major Incident Report – EasyIPT Outage 21-07-14

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available.

UPDATE01 17:30
This has been confirmed as resolved by wholesale.

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause.

UPDATE01 – 16:12
We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be published.