Category: Planned Works

We have been advised by Entanet that they will be inserting an additional line card in to goswell.core.enta.net to increase capacity. They have advised that no downtime is expected, however that’s not always been the case and we have classed this work as “at risk”

Any service disruption will result in traffic being automatically re-routed.

Date scheduled: Thursday 5th March 2015 00:01 – 03:00

We have been advised by one of our fiber providers (C4L) that they will be extending there coreTX network at Goswell Road. Works are **At Risk** as engineers will be present and working within a live network environment.

C4L Notice:

Location: Interxion, City Lifeline, Telehouse Metro, Goswell Road, Volta 36 Great Sutton St.

Start: 2015-02-27, 20:00
End: 2015-02-28, 06:00

Reason: To install additional switching equipment within the coreTX network.

Impact: N/A. However, due to the nature of working within live network racks, services are at risk of layer 1 interruption.

—

Should we loose the interconnect then our network will automatically re-route traffic via an alternate carrier.

We have seen some inbound and outbound VoIP calls fail to connect between the hours of 13:30 – 14:30 today with an odd array of messages on the SIP proxy connecting to our softswitch.

Initial diagnostics show nothing obvious, however to be proactive we will be looking deeper in to the platform this evening with the view of rebooting this to clear any stuck session data.

UPDATE01 This work has been rescheduled for the 26/01/2014

UPDATE02 This work has been rescheduled for the 27/01/2014

UPDATE03 20:35 This work has started.

UPDATE04 20:46 The server has requested a disk check to be run. Due to the size of the array this make take 15-20 minutes.

UPDATE05 20:52 The server has returned to operation but the SQL service is now not operating in the correct state following the disk check.

UPDATE06 20:58 The server has returned to operation and the SQL problem resolved. Test calls have been made and they completed successfully.

1) We will be making some changes to our VPS cluster to bring online additional capacity. Unmanaged 3rd party VPS sessions may see a small amount of downtime while there images are moved. No other impact is expected.

2) An audit will be conducted on our existing bulk fibre between our suite on the 4th floor and our meet me rack on the 2nd. This is for upcoming network expansion. No outages are expected however the links should be considered “at risk” due to the physical nature of the works. Should an outage occur on a fibre link then our network will automatically re-route traffic via a alternative path.

UPDATE01 – 23:30 These works have been completed as expected, However the 3rd party ESXi images will not be moved until 22:00 Tuesday night (16/12/14)

On the 12/11/2014 between the hours of 20:00 – 23:00 we will be carrying out a series of hardware and software updates on our core network services.

Hardware
esxi01.r00.lon01.structuredcommunications.co.uk
esxi02.r00.lon01.structuredcommunications.co.uk
esxi01.r01.lon01.structuredcommunications.co.uk
esxi02.r01.lon01.structuredcommunications.co.uk
esxi03.r01.lon01.structuredcommunications.co.uk
esxi04.r01.lon01.structuredcommunications.co.uk
esxi05.r01.lon01.structuredcommunications.co.uk

Software
ns01.r00.lon01.structuredcommunications.co.uk
ns02.r00.lon01.structuredcommunications.co.uk
noc.r01.lon01.easybond.co.uk
ag-primary.r01.lon01.easybond.co.uk
ag-backup.r01.lon01.easybond.co.uk
ns01.r01.lon01.easybond.co.uk
ns02.r01.lon01.easybond.co.uk
smtp01.r01.lon01.easybond.co.uk
smtp02.r01.lon01.easybond.co.uk

The works will involve disconnecting and removing all physical servers listed above.

The purposes of these works are to upgrade the physical hardware, ESXi versions and Debain operating systems. Each server will be taken offline and replaced / upgraded 1 at a time. The services are redundant and no service outages are anticipated. However in the case of the Easybond NOC, this will likely result in a small outage once the controller is re-loaded back to operational state.

Further updates will be given on the day.

UPDATE01 – 12/11/2014 – 19:20
Engineers are on-site and getting ready for the works listed above.

UPDATE02 – 12/11/2014 – 19:50
Engineers have advised that they are unable to gain remote access to ESXi01.R01 which is host to the EasyBond NOC. A server restart is required to restore access, however this will create additional outages for bonded circuts.

UPDATE03 – 12/11/2014 – 20:00
This work has now started.

UPDATE04 – 12/11/2014 – 20:20
Engineers where able to shutdown the EasyBond NOC without having to cause an outage at the time. The NOC will be offline for around an hour during this host migration.

If you reboot your bonded modem or if it looses coneection it will not eb able to reconnect until the NOC is back in service.

UPDATE05 – 12/11/2014 – 21:15
The EasyBond NOC is back in service. No other service migrations tonight are expected to cause any outages as they are redundant. We will advise once this work is complete

UPDATE06 – 12/11/2014 – 22:29
All work is now complete. Some of the EasyBond services still require some adational works and will be scheduled at a later date.

On the 25/09/2014 between the hours of 20:00 – 22:00 we will be replacing a line card in our LON01-CORE following on from the failure seen on the 28/08/2014.

The works will involve disconnecting all devices that are connected to that line card. Any server that has a redundant feed via another line card will remain online but “at risk” Any server with a single feed will go offline for the duration. Where customer’s PBXs are connected to this line card, they will see a loss of service and may require a reboot of their handsets / system following the works to clear any stale NAT sessions.

UPDATE01 – 20:10
Work on this replacement has started and engineers have started to disconnect devices connected to the S3 line card.

UPDATE02 – 20:50
The card has been replaced and devices have been reconnected. The card is now being placed in to a test mode to complete a series of self tests. Ports may reset and services are still classed as at risk.

UPDATE02 – 22:50
Testing complete and the line card is now back in service. Any users who don’t have a connection to there PBX are advised to power down your hardware and then power back on.

In line with the previous correspondence sent out to all unmanaged EasyIPT accounts whereby a customer has a direct connection to the carrier and has not migrated over to our in-house platform (Sipwise), then the following changes are due to take place tonight on our network as we no longer route managed traffic to this IP range.

1)Removal of the 146.101.248.200/26 network prefix via our private Frontier / Voiceflex interconnects.

Non migrated accounts will also not benefit from our redundant outbound routing in the event of a carrier failure, nore the additional in-house call fraud monitoring services offered via Sipwise. Unmanaged customers are reminded that call fraud protection is there sole responsibility to protect against and will be liable for any occurrences and financial costs.

There will be Planned Maintenance on ESXi02.R01 & ESXi03.R01 to resolve the ongoing remote management issue with these systems which a resolution has now been identified.

Work will involve a reload of these boxes. During this reload EasyBOND services will be effected. Connections will however be manually transferred to the backup aggregation service prior to the work but users will see a sort 10-30 second outage.

We apologise for any inconvenience this may cause.

UPDATE 01 – 20:28
This work is complete and all services have been transferred back to their original primary servers. We will continue to monitor the ESXi hosts.

We have been advised by Zen that they have a maintenance window between 22:00 and 05:00 on the 18th/19th of July for “Essential Software upgrades”

Expected impact duration: 2 x 30min outages within a 7hr window.

During this time we may lose our peering to Zen.