Category: Incidents

We are currently experiencing an issue that is preventing some DSL users connecting to our service. Our initial investigations indicate that this is a BT issue and we are working with them to resolve as soon as possible. The issue spans multiple nodes and is not geographically restrained to one area. We are aiming to have the service restored as a matter of urgency.

-Update- 11:40
Customer sessions are now being restored and our monitoring indicates that the majority of services are now working again. We’ve yet to receive details from BT as to the cause but will update when we have further information.

We are aware of a wholesale issue currently affecting our xDSL customer base, manifesting itself as poor performance.

Our Systems team are currently investigating. Further information will be provided as and when it becomes available.

UPDATE01 – 17:38 – 17/01/2014

We have been advised that the cause of the issue has been a “flapping” port between BT wholesale and our wholesale provider causing a loss in bandwidth. This has now been resolved however will have caused stale PPP sessions to occur. If you don’t have service can you please reboot all your hardware.

We apologise for any inconvenience caused.

UPDATE02 – 12:09 – 21/01/2014

An incident report has been made available for this issue. Major Incident Report – Enta Outage 17-01-2014

We are aware of a network wide issue affecting DSL sync and PPP sessions on some 02 wholesale LLU ADSL2+ circuits.

Engineers are working to locate the fault.

We will provide further updates once they are provided to us.

We apologise for any inconvenience.

We have been advised of a fibre failure within the 02 Wholesale network. A case has been logged with our wholesale provider.

We apologise for any inconvenience caused

At 20:17 we were made aware by our networking monitoring software that both our /21 ranges where being announced by our previous transit provider (AS24867) and not ourselves (AS199283)

We are current working to get this resolved.

UPDATE01 – 21:22

We have discovered the announcement was detected in the Czech Republic learnt by 6881 15685 6939 24867. Engineers are still working on the root cause of the rouge announcement, however this incident is now closed. A report will be made available as and when a full investigation has taken place.

We are aware of a network wide issue affecting DSL sync and PPP sessions on some 02 wholesale LLU ADSL2+ circuits.
Engineers are working to locate the fault.
We will provide further updates once they are provided to us.

UPDATE 01 – 14:23
Engineers are still working within wholesale to find the root cause of the fault. Connectivity has been intermittently restored.

UPDATE 01 – 17:10
We’ve now been advised that the supplier has identified a fibre fault at Telehouse and are now attempting to get it resolved as soon as possible. We’ll continue to update as the updates are available.

We are aware of an issue with one of our carriers where SIP registrations are being refused resulting in a loss of service. We feel this is a repeat of a recent issue experienced with this Carrier and are in discussions for clarification.

We will provide updates as they become available.

UPDATE01 @ 15:19

We are seeing sessions starting to reconnect and service is being restored. We apologise for any inconvenience and will be taking this up with the Carrier and obtaining an outage report which will be provided once available

Between 22:07 22:55 we saw a loss of connectivity to EasyHTTP. An MOR will be available shortly.

UPDATE 01 @ 23:44
MOR is now avaliable. Major Incident Report – EasyHTTP Outage 08-10-13

Between 19:20 & 20:45 we saw a DNS failure on ns01.r00.lon01.structuredcommunications.co.uk and ns02.r00.lon01.structuredcommunications.co.uk. This DNS failure lead to the failure of the SMTP service on EasyHTTP in that is was unable to lookup domains for email delivery. Emails where queued on the outbound SMTP connector and delivered once the issue was resolved at 20:41.

Looking at the logs prior to the above, its possible there may have been small period where this was intermittant resulting in Delivery delays.

The 2 listed name servers remain offline and DNS traffic has been diverted to ns01.r01.lon01.structuredcommunications.co.uk & ns02.r01.lon01.structuredcommunications.co.uk

We will advise when both the servers located in R00 are back in service.

We are aware of an outage on our Hosted EasyIPT services. Engineers are currently assigned remotely and we have dispatched engineers to the data centre.

Update01**Final Resolved** @ 21:37

Engineers attended LON01 and arrived on site at 17:01 where the server was power cycled restoring service at 17:05. Handsets then started to re-register with individual sites coming back online.

The service has remained stable.

An outage report will be made available shortly.