Category: Planned Works

We will be upgrading our shared EasyIPT platform (Hosted VoIP) over the course of Friday the 05th March. This upgrade work involves the replacement of the current server to a much larger device with an upgrade of OS and PBX software. The new server will be built and bought online along side the current one. Migration works will start after 20:00 hours on a client by client biases.

As part of the works, each shared session will now be given its own IPv4 address and trunk. This will eliminate the billing issues on call forwarding that have been an issue for some time.

The works also improve redundancy via a dual network topology setup to the main server and enables IPv6 going forward. This work is inline with our upgrade program and allows us to provide a truly unique setup and experience.

Once a client is migrated the site firewall rules and handsets will be updated to reflect the new session via our management network. No action is needed where this is in place. Any site operating outside of this will need to contact us or if you are using a 3rd party firewall with no VPN.

UPDATE 01 – 22:00 – 08/03/2015
Many of the systems have migrated without issue as expected however due to the volume of migrations we have been unable to complete them all within the time allotted. Migrations will continue over the course of Monday next week out of hours as based on the weekend work, there should be no service disruptions seen. The old server will then be disconnected at 22:00 on Tuesday the 10th March

We will be making some changes to our core SIP soft-switch tonight to resolve the issues we have been seeing with some BTIPX calls. This will involve a reload of the session controller and will drop any active calls. This should only need to happen once. We apologize in advance for any inconvenience this may cause.

We have been advised that an engineer working on behalf of Level3 will be in our core rack Tuesday 10-03-2015 terminating our new 24 pair bulk fiber as part of our network upgrade (details to follow) No service impact is expected, however the engineer will be within a live rack environment upon fitment and testing.

We have been advised by Entanet that they will be inserting an additional line card in to goswell.core.enta.net to increase capacity. They have advised that no downtime is expected, however that’s not always been the case and we have classed this work as “at risk”

Any service disruption will result in traffic being automatically re-routed.

Date scheduled: Thursday 5th March 2015 00:01 – 03:00

We have been advised by one of our fiber providers (C4L) that they will be extending there coreTX network at Goswell Road. Works are **At Risk** as engineers will be present and working within a live network environment.

C4L Notice:

Location: Interxion, City Lifeline, Telehouse Metro, Goswell Road, Volta 36 Great Sutton St.

Start: 2015-02-27, 20:00
End: 2015-02-28, 06:00

Reason: To install additional switching equipment within the coreTX network.

Impact: N/A. However, due to the nature of working within live network racks, services are at risk of layer 1 interruption.

—

Should we loose the interconnect then our network will automatically re-route traffic via an alternate carrier.

We have seen some inbound and outbound VoIP calls fail to connect between the hours of 13:30 – 14:30 today with an odd array of messages on the SIP proxy connecting to our softswitch.

Initial diagnostics show nothing obvious, however to be proactive we will be looking deeper in to the platform this evening with the view of rebooting this to clear any stuck session data.

UPDATE01 This work has been rescheduled for the 26/01/2014

UPDATE02 This work has been rescheduled for the 27/01/2014

UPDATE03 20:35 This work has started.

UPDATE04 20:46 The server has requested a disk check to be run. Due to the size of the array this make take 15-20 minutes.

UPDATE05 20:52 The server has returned to operation but the SQL service is now not operating in the correct state following the disk check.

UPDATE06 20:58 The server has returned to operation and the SQL problem resolved. Test calls have been made and they completed successfully.

1) We will be making some changes to our VPS cluster to bring online additional capacity. Unmanaged 3rd party VPS sessions may see a small amount of downtime while there images are moved. No other impact is expected.

2) An audit will be conducted on our existing bulk fibre between our suite on the 4th floor and our meet me rack on the 2nd. This is for upcoming network expansion. No outages are expected however the links should be considered “at risk” due to the physical nature of the works. Should an outage occur on a fibre link then our network will automatically re-route traffic via a alternative path.

UPDATE01 – 23:30 These works have been completed as expected, However the 3rd party ESXi images will not be moved until 22:00 Tuesday night (16/12/14)

On the 12/11/2014 between the hours of 20:00 – 23:00 we will be carrying out a series of hardware and software updates on our core network services.

Hardware
esxi01.r00.lon01.structuredcommunications.co.uk
esxi02.r00.lon01.structuredcommunications.co.uk
esxi01.r01.lon01.structuredcommunications.co.uk
esxi02.r01.lon01.structuredcommunications.co.uk
esxi03.r01.lon01.structuredcommunications.co.uk
esxi04.r01.lon01.structuredcommunications.co.uk
esxi05.r01.lon01.structuredcommunications.co.uk

Software
ns01.r00.lon01.structuredcommunications.co.uk
ns02.r00.lon01.structuredcommunications.co.uk
noc.r01.lon01.easybond.co.uk
ag-primary.r01.lon01.easybond.co.uk
ag-backup.r01.lon01.easybond.co.uk
ns01.r01.lon01.easybond.co.uk
ns02.r01.lon01.easybond.co.uk
smtp01.r01.lon01.easybond.co.uk
smtp02.r01.lon01.easybond.co.uk

The works will involve disconnecting and removing all physical servers listed above.

The purposes of these works are to upgrade the physical hardware, ESXi versions and Debain operating systems. Each server will be taken offline and replaced / upgraded 1 at a time. The services are redundant and no service outages are anticipated. However in the case of the Easybond NOC, this will likely result in a small outage once the controller is re-loaded back to operational state.

Further updates will be given on the day.

UPDATE01 – 12/11/2014 – 19:20
Engineers are on-site and getting ready for the works listed above.

UPDATE02 – 12/11/2014 – 19:50
Engineers have advised that they are unable to gain remote access to ESXi01.R01 which is host to the EasyBond NOC. A server restart is required to restore access, however this will create additional outages for bonded circuts.

UPDATE03 – 12/11/2014 – 20:00
This work has now started.

UPDATE04 – 12/11/2014 – 20:20
Engineers where able to shutdown the EasyBond NOC without having to cause an outage at the time. The NOC will be offline for around an hour during this host migration.

If you reboot your bonded modem or if it looses coneection it will not eb able to reconnect until the NOC is back in service.

UPDATE05 – 12/11/2014 – 21:15
The EasyBond NOC is back in service. No other service migrations tonight are expected to cause any outages as they are redundant. We will advise once this work is complete

UPDATE06 – 12/11/2014 – 22:29
All work is now complete. Some of the EasyBond services still require some adational works and will be scheduled at a later date.

On the 25/09/2014 between the hours of 20:00 – 22:00 we will be replacing a line card in our LON01-CORE following on from the failure seen on the 28/08/2014.

The works will involve disconnecting all devices that are connected to that line card. Any server that has a redundant feed via another line card will remain online but “at risk” Any server with a single feed will go offline for the duration. Where customer’s PBXs are connected to this line card, they will see a loss of service and may require a reboot of their handsets / system following the works to clear any stale NAT sessions.

UPDATE01 – 20:10
Work on this replacement has started and engineers have started to disconnect devices connected to the S3 line card.

UPDATE02 – 20:50
The card has been replaced and devices have been reconnected. The card is now being placed in to a test mode to complete a series of self tests. Ports may reset and services are still classed as at risk.

UPDATE02 – 22:50
Testing complete and the line card is now back in service. Any users who don’t have a connection to there PBX are advised to power down your hardware and then power back on.

In line with the previous correspondence sent out to all unmanaged EasyIPT accounts whereby a customer has a direct connection to the carrier and has not migrated over to our in-house platform (Sipwise), then the following changes are due to take place tonight on our network as we no longer route managed traffic to this IP range.

1)Removal of the 146.101.248.200/26 network prefix via our private Frontier / Voiceflex interconnects.

Non migrated accounts will also not benefit from our redundant outbound routing in the event of a carrier failure, nore the additional in-house call fraud monitoring services offered via Sipwise. Unmanaged customers are reminded that call fraud protection is there sole responsibility to protect against and will be liable for any occurrences and financial costs.