LON01 – EasyXDSL – 02/12/2016 – 16:20

Our networking monitoring has alerted us to a large number of DSL circuits going off-line. initial checks shows this to be a supplier outage and we are already engaged with them. UPDATE01 – 17:00 We have chased our suppliers for an update who advise they are still working to isolate the fault UPDATE02 – 17:37 … Continue reading “LON01 – EasyXDSL – 02/12/2016 – 16:20”

Our networking monitoring has alerted us to a large number of DSL circuits going off-line.

initial checks shows this to be a supplier outage and we are already engaged with them.

UPDATE01 – 17:00
We have chased our suppliers for an update who advise they are still working to isolate the fault

UPDATE02 – 17:37
We are continuing to chase for an update. We have been advised this has been made more complex due to another possible wholesale fault

We have confirmed the fault is not within our network and this now appears to only be affecting connections in the South of England.

UPDATE03 – 19:00

our supplier has advised there is a line card failure within there backhull network. Replacement hardware is required and a ETA of 21:30 has been placed on service restoration for affected users.

UPDATE04 – 21:25
We are seen services restore and have requested an RFO from our supplier. We apologise for the issues caused.

TR069 – Routers

We are aware a very small proportion of new ZyXel AMG1202-T10B routers supplied by us are subject to the TR069 exploit that is currently in the media. As well as the main stream providers our network engineers have also been able to devise a resolution. Clients with a ZyXel AMG1202-T10B who currently have no internet … Continue reading “TR069 – Routers”

We are aware a very small proportion of new ZyXel AMG1202-T10B routers supplied by us are subject to the TR069 exploit that is currently in the media. As well as the main stream providers our network engineers have also been able to devise a resolution.

Clients with a ZyXel AMG1202-T10B who currently have no internet access are advised to contact us during working hours so we can log your case as they are dealt with manually due to requiring engineering resources.

LON01- Leased Lines – BT Wholesale – 01/12/2016 – 09:37

Our networking monitoring has alerted us to a failure of several BT wholesale provided leased lines. Initial calls to them advise they have a failure of a core network device. Engineers have been dispatched to site. circuits with a backup connection will have re-routed. UPDATE01 01 – 10:00 We have observed a number of connections … Continue reading “LON01- Leased Lines – BT Wholesale – 01/12/2016 – 09:37”

Our networking monitoring has alerted us to a failure of several BT wholesale provided leased lines. Initial calls to them advise they have a failure of a core network device. Engineers have been dispatched to site.

circuits with a backup connection will have re-routed.

UPDATE01 01 – 10:00
We have observed a number of connections restore with a number still offline. We are chasing our supplier for an update as they advise this looks to be power related at there rack.

UPDATE01 02 – 11:00
We have been advised this has been resolved and are awaiting and RFO.

LON01 – EasyIPT – 18/11/2016 – 13:10

We are aware of a problem with one of our media gateways affecting inbound and outbound calls. We are currently investigating as a matter of priority. UPDATE01 – 13:13 Engineers have found the media gateway to be suffering from a sudden high CPU load. We have been able to stabilize the system remotely and are … Continue reading “LON01 – EasyIPT – 18/11/2016 – 13:10”

We are aware of a problem with one of our media gateways affecting inbound and outbound calls. We are currently investigating as a matter of priority.

UPDATE01 – 13:13
Engineers have found the media gateway to be suffering from a sudden high CPU load. We have been able to stabilize the system remotely and are currently monitoring.

UPDATE02 – 13:20
We have seen a repeat of high CPU demand and are in the process of isolating the service.

UPDATE03 – 13:25
The service has been found and the root fault isolated with remedial works planned over the weekend. We apologies for the issues caused.

Additional media gateways had already been planned for deployment over December to provide additional redundancy, this will now be bought forward.

UPDATE04 – 15:00
We are still seeing some calls fail to route due to stuck sessions on customer trunks. This was caused by the SQL service being overloaded and log entry’s being missed. Work is being planned for tonight to reload the affected services.

UPDATE05 – 20:00
Work has been undertaken to stabilise the platform.

UPDATE06 – 21/11/2016 12:00
Engineers have completed the install of additional hardware and a migration plan will be designed in due course.

LON01 – EasyXDSL – 17/11/2016 – 10:58 *At Risk*

Our network monitoring has alerted us to CPU errors being generated on lns01.dsl.structuredcommunications.co.uk We have raised a ticket with our hardware vendor for them to investigate and advise. The device remains in operation and routing data as expected, however it should be classed as “at rick” during this time due to the possibility of an … Continue reading “LON01 – EasyXDSL – 17/11/2016 – 10:58 *At Risk*”

Our network monitoring has alerted us to CPU errors being generated on lns01.dsl.structuredcommunications.co.uk

We have raised a ticket with our hardware vendor for them to investigate and advise. The device remains in operation and routing data as expected, however it should be classed as “at rick” during this time due to the possibility of an unexpected failure.

Other LNSs within the pool are unaffected. Should the above LNS fail then circuits will automatically re-terminate on another device.

UPDATE01 – 11:30
Our hardware vendor has reviewed the logs we have provided and concluded the device could be suffering from a physical hardware issue. We are looking at options on replacing this device.

UPDATE02 – 11:52
While options are being reviewed to replace this device, we have taken the decision to remove it from the active termination pool. This means it will not accept any new PPP connection attemps, however will continue to service existing established ones.

UPDATE03 – 13:05
Replacement hardware has been agreed by the vendor. We will advise of a maintenance window in due course.

UPDATE04 -23/11/2016
The LNS was replaced without issue and is now back in service.