Category: Emergency Work

Following on from reports today of SIP channel limits being reached for both inbound and outbound calls affecting some of our managed PBX systems, work has been undertaken on our core softswitch to try and identify the root cause of this issue.

Changes have been made to the platform however a emergency reboot is required. Due to the size of the platform this reboot will take up-to 15 minutes to complete. During this time inbound and outbound calls will be limited.

We will advise once service is restored and calls are routing again.

Apologies for any inconvenience this may cause.

UPDATE01 – 21:40 softswitch reboot is complete, we are monitoring traffic flow

UPDATE02 – 22:55 A random check on managed PBX systems are showing registration, however this is not reflected on our softswitch, traffic is flowing however.

UPDATE03 – 22:10 Systems continue to show inaccurate data about the status of some registrations. We suspect stuck SIP sessions on some managed PBXs as this is affecting a wide range of PBX software versions.

UPDATE04 – 21:40 All affected managed PBXs are showing as on-line correctly.

We have been advised this afternoon by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 tonight. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at Goswell Road. DSL traffic traversing over this link will drop during the maintenance window, however should re-establish over an alternative link via Telehouse East which is unaffected.

The nature of these works include a reload of the Junos hardware at Telehouse North by our wave supplier and in theory should be the only disruption we see. Unfortunately sort disruptions like this will cause multiple DSL PPP session re-terminations as they flip back and forth between Telehouse North & Telehouse East as our network is designed to use the former where possible. Due this concern and avoiding any other unforeseen issues that will cause session drops during the maintenance windows, we will be manually shutting down the Telehouse North link at 22:50 tonight with the view of bringing it backup at 07:00 tomorrow morning. This will limit the session drops to these times and provide a stable service throughout.

We apologise for any inconvenience this may cause.

UPDATE 01 – 18:45

We have reviewed the original manual switch over time and concluded it does not allow any contingency should this be required. We have therefor updated this to 22:00
UPDATE 02 – 07:10 – 12/11/2015

This work is complete.

We are aware some users have been seeing stuck SIP sessions today, whereby “BYE” messages have not been completely clearing down resulting in accounts hitting a concurrent call limit. A work around has been put in place for this until this evening when we can reload the core device to drop any sessions still in this state.

UPDATE01 – 20:05
This work has started and the server has been requested to reload.

UPDATE02 – 20:15
The server has reloaded and is accepting SIP requests, however is not processing any call routing.

UPDATE03 – 20:35
We have discovered that the Kamailio Proxy service has failed to start which is why calls are not being processed. We are working to restore the service.

UPDATE04 – 20:40
The service has been restored and test calls are now routing OK.

Our network monitoring has alerted us to a intermittent fan failure on PSU 1 within CORE01.LON01. This device has redundant PSUs so no impact is expected.

Engineers will attend this afternoon to swap out the failing power supply. Again no impact is expected to be seen, however the device will be marked as “at risk” for the duration enginners are working on it.

Updates will be provided once available.

UPDATE01 – 14:00

This work has been completed without any issue.

We are aware of a permission issue on the Plesk control panel for EasyHTTP which is generating “Access Denied” errors when changing various account settings.

A patch has been released and installed, however this does require a full reload of the server which will take place tonight at 20:00. Due to the size of the system this reload may take up to 20 minutes to complete. During this time all services will be unavailable.

We have discovered that one of our internal BGP filters is incorrectly configured. We are currently reviewing all filters to ensure this is only limited to 1 instance. Once the review is complete we will be taking steps to correct the configuration issues found. This will result in routing table changes and disruption to various services.

We apologies for any inconvenience this may cause.

UPDATE 01 – 20:20
After reviewing:

core01.lon01.structuredcommunications.co.uk
core02.lon01.structuredcommunications.co.uk
access01.lon01.structuredcommunications.co.uk

We have discovered only 1 miss-configured BGP filter on core01.lon01.structuredcommunications.co.uk. The BGP filter was only client facing and not carrier as originally suspected. No changes have been made to our carrier peering and no loss of service was seen.

Following on from the previous NOC advisory it has become necessary to migrate the existing “Active” supervisor to “Hot Standby” so a reload can take place on that slot. This may effect backplane switching on CORE-01 and packet loss may be seen for several seconds while the Standby supervisor becomes the Active unit.

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators.
We have corrected this configuration issue and completed a fail-over test which completed as expected.

Users on EasyBond would have seen 2 x 30 seconds outages. We apologise for any inconvenience caused.

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration.

We have made some changes to our core softswitch to eliminate this problem. These changes require a reload of the active configuration files on the server and will drop any active call. This reload should take no more than 30 seconds.

We apologise for any inconvenience this may cause.

UPDATE01 – 21:01
This work is complete.