Category: Incidents

We are currently aware of an incident on Virgin Media’s network which is causing a loss of service to several Virgin Media provided leased lines.

Virgin Media are currently investigating and updates will be provided as and when they become available.

UPDATE01 – 10:52

Virgin Media have identified a problem with one of their core routers in Telehouse and they are continuing their investigation although no ETR is available as of yet.

UPDATE02 – 11:00

We are seeing service restore back to Virgin media circuts, however this has not been confirmed by Virgin yet.

UPDATE03 – 12:00
We have requested an RFO from Virgin which we will make avalible via the NOC

At 10:52am our network monitoring alerted us to a issue with our Frontier voice interconnect. At this time we lost our BGP session to them and traffic started routing over diverse paths. This would have caused all active calls to drop while BGP reconverted.

UPDATE01 – 11:10
At 11:02 our network monitoring showed that the interconnect had been resorted and the BGP session has resumed. We have logged a case with our carrier to see if the fault originated from there side as no other session losses were seen from our side.

UPDATE02 – 11:30
We have been advised this is a carrier issues with C4L.

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service.

UPDATE01 12:50
Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier.

UPDATE02 13:23
Service has been restored; An RFO will be issued shortly. We apologise for any inconvenience caused.

UPDATE02 14:00
This issue has reoccured, the root of the problem has been identified and we are building an RFO that will be published shortly.

Outage Report
Major Incident Report – EasyIPT Outage 21-07-14

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available.

UPDATE01 17:30
This has been confirmed as resolved by wholesale.

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause.

UPDATE01 – 16:12
We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be published.

Our network monitoring has alerted us to an issue at the London LINX exchange. This will be having an impact on any customer who uses a 3rd party connection to connect to our services.

UPDATE 01 – 11:32
Service has been restored and we are seeing traffic route via LINX again. As we have no direct relationship or take services from LINX, there wont be a RFO.

We are aware of a problem with an upstream provider that is preventing the termination of OFFNET (Mobile, Local, National and international) calls. We are currently working to resolve the fault and will provide updates shortly.

UPDATE01 – 11:36
Local and national calls should now be terminating correctly, however mobile calls are still resulting in a BUSY response from our carrier.

UPDATE02 – 12:00
Mobile and international calls are now terminating correctly.

We have logged a case with the carrier for details and will provide this once made available.

We apologise for any inconvenience caused

UPDATE03 – 12:24
We have been advised the root cause of the failure was “a major outage on the BT IP exchange interconnect which caused a high proportion of calls to fail.”

In light of this outage, Structured Communications will be implementing carrier diverse outbound routing. This will require large network changes and further details to follow in the coming weeks.

During an internal training exercise a “reload” command was issued to CORE01-LON instead of CORE01-HOR which resulted in a reboot of both the primary and secondary supervisor modules. This reload caused a 5 minute loss of service to LON01 for all directly connected services and customers.

Safe guards will be put in place on LON01 to prevent this happening again in such a scenario.

We apologise for any inconvenience caused.

We are currently aware of fibre break in the London area which is affecting outbound calls for some of our users. We have been informed by suppliers that this is due do fibre damage to which is currently being repaired. We are expecting an update by midday.

Calls from customers outside of London to customers in London may fail. Broadband speeds should NOT be affected. This incident will be impacting BT customers and other operators.

In the mean time if you are experiencing issues please prefix the dialled number with 1280 for example 128002033014000. If you use a telephone system and normally prefix calls with 9 an example would be 9128002033014000.

We apologise for any inconvenience caused.

We are currently experiencing an issue that is preventing some DSL users connecting to our service. Our initial investigations indicate that this is a BT issue and we are working with them to resolve as soon as possible. The issue spans multiple nodes and is not geographically restrained to one area. We are aiming to have the service restored as a matter of urgency.

-Update- 11:40
Customer sessions are now being restored and our monitoring indicates that the majority of services are now working again. We’ve yet to receive details from BT as to the cause but will update when we have further information.