Category: Emergency Work

LON01 – EasyIPT – 13/07/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*

We are aware some users have been seeing stuck SIP sessions today, whereby “BYE” messages have not been completely clearing down resulting in accounts hitting a concurrent call limit. A work around has been put in place for this until this evening when we can reload the core device to drop any sessions still in … Continue reading “LON01 – EasyIPT – 13/07/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*”

We are aware some users have been seeing stuck SIP sessions today, whereby “BYE” messages have not been completely clearing down resulting in accounts hitting a concurrent call limit. A work around has been put in place for this until this evening when we can reload the core device to drop any sessions still in this state.

UPDATE01 – 20:05
This work has started and the server has been requested to reload.

UPDATE02 – 20:15
The server has reloaded and is accepting SIP requests, however is not processing any call routing.

UPDATE03 – 20:35
We have discovered that the Kamailio Proxy service has failed to start which is why calls are not being processed. We are working to restore the service.

UPDATE04 – 20:40
The service has been restored and test calls are now routing OK.

LON01 – 26/06/2015 – 13:00 – CORE01.LON01 *EMERGENCY WORKS* COMPLETE*

Our network monitoring has alerted us to a intermittent fan failure on PSU 1 within CORE01.LON01. This device has redundant PSUs so no impact is expected. Engineers will attend this afternoon to swap out the failing power supply. Again no impact is expected to be seen, however the device will be marked as “at risk” … Continue reading “LON01 – 26/06/2015 – 13:00 – CORE01.LON01 *EMERGENCY WORKS* COMPLETE*”

Our network monitoring has alerted us to a intermittent fan failure on PSU 1 within CORE01.LON01. This device has redundant PSUs so no impact is expected.

Engineers will attend this afternoon to swap out the failing power supply. Again no impact is expected to be seen, however the device will be marked as “at risk” for the duration enginners are working on it.

Updates will be provided once available.

UPDATE01 – 14:00

This work has been completed without any issue.

Internal Systems *RESOLVED*

We are currently experiencing issues with some internal systems that is preventing various customer service / support operations. Engineers are on route to the data centre, however local network rail cancellations and disruptions are delaying engineers from attending. Please also note the above is effectiving email delivery so we are unable to process support tickets. … Continue reading “Internal Systems *RESOLVED*”

We are currently experiencing issues with some internal systems that is preventing various customer service / support operations. Engineers are on route to the data centre, however local network rail cancellations and disruptions are delaying engineers from attending.

Please also note the above is effectiving email delivery so we are unable to process support tickets. We advise calling for any urgent issues.

Apologies for any inconvenience this may cause. Once we know more we will update this notice.

UPDATE01 – 12.00
Engineers have arrived on site.

UPDATE 02 – 12.30
Internal systems have been restored

EasyHTTP – 24/04/2015 – 20:00 – 20:30 – Emergency Maintenance *COMPLETE*

We are aware of a permission issue on the Plesk control panel for EasyHTTP which is generating “Access Denied” errors when changing various account settings. A patch has been released and installed, however this does require a full reload of the server which will take place tonight at 20:00. Due to the size of the … Continue reading “EasyHTTP – 24/04/2015 – 20:00 – 20:30 – Emergency Maintenance *COMPLETE*”

We are aware of a permission issue on the Plesk control panel for EasyHTTP which is generating “Access Denied” errors when changing various account settings.

A patch has been released and installed, however this does require a full reload of the server which will take place tonight at 20:00. Due to the size of the system this reload may take up to 20 minutes to complete. During this time all services will be unavailable.

LON01 – 05-04-2015 – 20:00 – 21:00 – Emergency Network Maintenance *Complete*

We have discovered that one of our internal BGP filters is incorrectly configured. We are currently reviewing all filters to ensure this is only limited to 1 instance. Once the review is complete we will be taking steps to correct the configuration issues found. This will result in routing table changes and disruption to various … Continue reading “LON01 – 05-04-2015 – 20:00 – 21:00 – Emergency Network Maintenance *Complete*”

We have discovered that one of our internal BGP filters is incorrectly configured. We are currently reviewing all filters to ensure this is only limited to 1 instance. Once the review is complete we will be taking steps to correct the configuration issues found. This will result in routing table changes and disruption to various services.

We apologies for any inconvenience this may cause.

UPDATE 01 – 20:20
After reviewing:

core01.lon01.structuredcommunications.co.uk
core02.lon01.structuredcommunications.co.uk
access01.lon01.structuredcommunications.co.uk

We have discovered only 1 miss-configured BGP filter on core01.lon01.structuredcommunications.co.uk. The BGP filter was only client facing and not carrier as originally suspected. No changes have been made to our carrier peering and no loss of service was seen.

LON01 – Core01 – Active Supervisor Swap – 03/03/2015 – 20:00 – 20:30 *COMPLETE*

Following on from the previous NOC advisory it has become necessary to migrate the existing “Active” supervisor to “Hot Standby” so a reload can take place on that slot. This may effect backplane switching on CORE-01 and packet loss may be seen for several seconds while the Standby supervisor becomes the Active unit.

Following on from the previous NOC advisory it has become necessary to migrate the existing “Active” supervisor to “Hot Standby” so a reload can take place on that slot. This may effect backplane switching on CORE-01 and packet loss may be seen for several seconds while the Standby supervisor becomes the Active unit.

LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators. We have corrected this configuration issue and … Continue reading “LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*”

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators.
We have corrected this configuration issue and completed a fail-over test which completed as expected.

Users on EasyBond would have seen 2 x 30 seconds outages. We apologise for any inconvenience caused.

LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration. We have made some changes to our core softswitch to eliminate … Continue reading “LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*”

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration.

We have made some changes to our core softswitch to eliminate this problem. These changes require a reload of the active configuration files on the server and will drop any active call. This reload should take no more than 30 seconds.

We apologise for any inconvenience this may cause.

UPDATE01 – 21:01
This work is complete.

LON01 – EasyBond – 20:00 – 21:00 – 03/07/2014 *Controlled Failover Test* *COMPLETE*

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers. UPDATE01 – 20:54 Testing has completed and the service failed over as expected on 3 simulated failures

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers.

UPDATE01 – 20:54
Testing has completed and the service failed over as expected on 3 simulated failures

EasyHTTP – 16/04/2014 – 20:00 – Maintenance *COMPLETE*

Following on from today’s CPU issues, We have been advised there is a major upgrade for Mail Enable. This will be installed tonight to ensure we are running the latest release.# UPDATE 01 – 20:15 This work is now complete. Any users having problems with account syncing are advised to remove and re-add there account … Continue reading “EasyHTTP – 16/04/2014 – 20:00 – Maintenance *COMPLETE*”

Following on from today’s CPU issues, We have been advised there is a major upgrade for Mail Enable. This will be installed tonight to ensure we are running the latest release.#

UPDATE 01 – 20:15
This work is now complete. Any users having problems with account syncing are advised to remove and re-add there account to their mail client. (Remember to back up your messages first)