Category: Incidents

We have been receiving reports of poor quality affecting some FTTC connections. We are currently working to identify the issue.

If you are experiencing problems please call in to report the issue so we can continue to collect affected circuits.

UPDATE01 – 09:39

We have collated a number of lines and reported these to our wholesale provider who have suggested a possible hardware fault in London within there core switches. It has not been confirmed these are at fault, however we are connected to these devices.

UPDATE02 – 10:00
Ticket updated to reflect Packet Loss over call quality. Our NOC engineers are currently on a call with our supplier for an update. We apologise for the continued delay.

UPDATE03- 10:15
We are seeing packet loss clear and services are starting to recover. We have not yet received an official clear

UPDATE04- 10:15
Services remain stable. we have been advised the problem was with a 10Gbps bearer that provides a number of services from Openreach POPs. Our network monitoring showed this problem starting at 23:15 last night so we have logged a complaint with our wholesale provider to advise why this was not picked up and resolved over night prior to it becoming service affecting this AM.

Once again we apologise for any inconvenience caused and will provide an RFO on request once made available to ourselves.

Our network monitoring has just alerted us to a loss of a VOIP interconnect with one of our primary carriers. We are currently investigating the issue as a matter of priority. Calls active at the time may have dropped however traffic should have re-routed.

UPDATE01 – 11:05
We can confirm traffic has re-routed as expected. We did observe a drop is call traffic as our network adjusted to the routing changes and is not unexpected. A fault has been logged with our remote carrier as all our remote tests are passing.

UPDATE02 – 11:11
We have seen our BGP session restore along with our preferred routing. Calls remained active. We are still awaiting an update for our affected carrier.

UPDATE03 – 11:45
Our remote carrier has confirmed they are not at fault and they simply saw the same as we did in our logs.

UPDATE04 – 12:25
Our fibre wave provider has confirmed they suffered an issue with part of there network. We again apologise for any inconvenience caused.

We are aware this server has been black listed due to a rouge script on a clients site. This has been removed and the various de-listing requests submitted. This should start to take affect and clear within the next 48 hours.

We are currently and investigating an issue causing a loss of connectivity for a small number of Vodafone based connections. We are working on the problem and further updates will be provided as and when they become available.

At this stage we believe it to be a faultly NTU within Vodafone.

Clients with backup will have automatically rerouted.

UPDATE02 – 19:05
We have seen services restore and await an update from Vodafone.

Our networking monitoring has alerted us to a large number of DSL circuits going off-line.

initial checks shows this to be a supplier outage and we are already engaged with them.

UPDATE01 – 17:00
We have chased our suppliers for an update who advise they are still working to isolate the fault

UPDATE02 – 17:37
We are continuing to chase for an update. We have been advised this has been made more complex due to another possible wholesale fault

We have confirmed the fault is not within our network and this now appears to only be affecting connections in the South of England.

UPDATE03 – 19:00

our supplier has advised there is a line card failure within there backhull network. Replacement hardware is required and a ETA of 21:30 has been placed on service restoration for affected users.

UPDATE04 – 21:25
We are seen services restore and have requested an RFO from our supplier. We apologise for the issues caused.

We are aware a very small proportion of new ZyXel AMG1202-T10B routers supplied by us are subject to the TR069 exploit that is currently in the media. As well as the main stream providers our network engineers have also been able to devise a resolution.

Clients with a ZyXel AMG1202-T10B who currently have no internet access are advised to contact us during working hours so we can log your case as they are dealt with manually due to requiring engineering resources.

Our networking monitoring has alerted us to a failure of several BT wholesale provided leased lines. Initial calls to them advise they have a failure of a core network device. Engineers have been dispatched to site.

circuits with a backup connection will have re-routed.

UPDATE01 01 – 10:00
We have observed a number of connections restore with a number still offline. We are chasing our supplier for an update as they advise this looks to be power related at there rack.

UPDATE01 02 – 11:00
We have been advised this has been resolved and are awaiting and RFO.

We are aware of a problem with one of our media gateways affecting inbound and outbound calls. We are currently investigating as a matter of priority.

UPDATE01 – 13:13
Engineers have found the media gateway to be suffering from a sudden high CPU load. We have been able to stabilize the system remotely and are currently monitoring.

UPDATE02 – 13:20
We have seen a repeat of high CPU demand and are in the process of isolating the service.

UPDATE03 – 13:25
The service has been found and the root fault isolated with remedial works planned over the weekend. We apologies for the issues caused.

Additional media gateways had already been planned for deployment over December to provide additional redundancy, this will now be bought forward.

UPDATE04 – 15:00
We are still seeing some calls fail to route due to stuck sessions on customer trunks. This was caused by the SQL service being overloaded and log entry’s being missed. Work is being planned for tonight to reload the affected services.

UPDATE05 – 20:00
Work has been undertaken to stabilise the platform.

UPDATE06 – 21/11/2016 12:00
Engineers have completed the install of additional hardware and a migration plan will be designed in due course.

Our network monitoring has alerted us to CPU errors being generated on lns01.dsl.structuredcommunications.co.uk

We have raised a ticket with our hardware vendor for them to investigate and advise. The device remains in operation and routing data as expected, however it should be classed as “at rick” during this time due to the possibility of an unexpected failure.

Other LNSs within the pool are unaffected. Should the above LNS fail then circuits will automatically re-terminate on another device.

UPDATE01 – 11:30
Our hardware vendor has reviewed the logs we have provided and concluded the device could be suffering from a physical hardware issue. We are looking at options on replacing this device.

UPDATE02 – 11:52
While options are being reviewed to replace this device, we have taken the decision to remove it from the active termination pool. This means it will not accept any new PPP connection attemps, however will continue to service existing established ones.

UPDATE03 – 13:05
Replacement hardware has been agreed by the vendor. We will advise of a maintenance window in due course.

UPDATE04 -23/11/2016
The LNS was replaced without issue and is now back in service.