November 2014 – Structured Communications NOC

We are currently experiencing an issue that is preventing some DSL users connecting to our service and/or constantly reconnecting to our service. Our initial investigations indicate that this is a Entanet issue and we are working with them to resolve as soon as possible. We will update with more information as it becomes available.

— Update 09:17 —
Entanet have identified a hardware failure and advise engineers have been dispatched to replace the faulty hardware. We are still working for a ETA.

— Update – 09:21 —
Any DSL bonded circuits may see reduced capacity but service should remain stable as we use 2 wholesale ISPs with our own interconnects.

— Update – 13:21 —
The issue is still currently on going, Entanet have advised engineers are still en route to replace the hardware. Due to the re routing of traffic you may experience slow speeds and/or packet loss. Please accept our apologies for any inconvenience caused and we will continue to provide updates as they become available.

— Update – 14:01 —
Entanet have advised engineers have arrived on site to replace the faulty hardware.

On the 12/11/2014 between the hours of 20:00 – 23:00 we will be carrying out a series of hardware and software updates on our core network services.

Hardware
esxi01.r00.lon01.structuredcommunications.co.uk
esxi02.r00.lon01.structuredcommunications.co.uk
esxi01.r01.lon01.structuredcommunications.co.uk
esxi02.r01.lon01.structuredcommunications.co.uk
esxi03.r01.lon01.structuredcommunications.co.uk
esxi04.r01.lon01.structuredcommunications.co.uk
esxi05.r01.lon01.structuredcommunications.co.uk

Software
ns01.r00.lon01.structuredcommunications.co.uk
ns02.r00.lon01.structuredcommunications.co.uk
noc.r01.lon01.easybond.co.uk
ag-primary.r01.lon01.easybond.co.uk
ag-backup.r01.lon01.easybond.co.uk
ns01.r01.lon01.easybond.co.uk
ns02.r01.lon01.easybond.co.uk
smtp01.r01.lon01.easybond.co.uk
smtp02.r01.lon01.easybond.co.uk

The works will involve disconnecting and removing all physical servers listed above.

The purposes of these works are to upgrade the physical hardware, ESXi versions and Debain operating systems. Each server will be taken offline and replaced / upgraded 1 at a time. The services are redundant and no service outages are anticipated. However in the case of the Easybond NOC, this will likely result in a small outage once the controller is re-loaded back to operational state.

Further updates will be given on the day.

UPDATE01 – 12/11/2014 – 19:20
Engineers are on-site and getting ready for the works listed above.

UPDATE02 – 12/11/2014 – 19:50
Engineers have advised that they are unable to gain remote access to ESXi01.R01 which is host to the EasyBond NOC. A server restart is required to restore access, however this will create additional outages for bonded circuts.

UPDATE03 – 12/11/2014 – 20:00
This work has now started.

UPDATE04 – 12/11/2014 – 20:20
Engineers where able to shutdown the EasyBond NOC without having to cause an outage at the time. The NOC will be offline for around an hour during this host migration.

If you reboot your bonded modem or if it looses coneection it will not eb able to reconnect until the NOC is back in service.

UPDATE05 – 12/11/2014 – 21:15
The EasyBond NOC is back in service. No other service migrations tonight are expected to cause any outages as they are redundant. We will advise once this work is complete

UPDATE06 – 12/11/2014 – 22:29
All work is now complete. Some of the EasyBond services still require some adational works and will be scheduled at a later date.

Month: November 2014

EasyxDSL Cannot Conenct Issues 12/11/2014 – 09:00 RESOLVED

LON01 – 12/11/2014 – 20:00 till 23:00 Planned Maintenance COMPLETE