LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators. We have corrected this configuration issue and … Continue reading “LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*”

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators.
We have corrected this configuration issue and completed a fail-over test which completed as expected.

Users on EasyBond would have seen 2 x 30 seconds outages. We apologise for any inconvenience caused.

LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration. We have made some changes to our core softswitch to eliminate … Continue reading “LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*”

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration.

We have made some changes to our core softswitch to eliminate this problem. These changes require a reload of the active configuration files on the server and will drop any active call. This reload should take no more than 30 seconds.

We apologise for any inconvenience this may cause.

UPDATE01 – 21:01
This work is complete.

EasyIPT – Inbound Calls – 12:45 – 21/07/2014

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service. UPDATE01 12:50 Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier. UPDATE02 13:23 Service has been restored; An … Continue reading “EasyIPT – Inbound Calls – 12:45 – 21/07/2014”

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service.

UPDATE01 12:50
Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier.

UPDATE02 13:23
Service has been restored; An RFO will be issued shortly. We apologise for any inconvenience caused.

UPDATE02 14:00
This issue has reoccured, the root of the problem has been identified and we are building an RFO that will be published shortly.

Outage Report
Major Incident Report – EasyIPT Outage 21-07-14

EasyXDSL – WBC Cannot connect Issue (ADSL2+) *RESOLVED*

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available. UPDATE01 17:30 This … Continue reading “EasyXDSL – WBC Cannot connect Issue (ADSL2+) *RESOLVED*”

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available.

UPDATE01 17:30
This has been confirmed as resolved by wholesale.

LON01 – Zen BGP Session – 22:00- 05:00 – 18/07/2014 *Planned Maintenance*

We have been advised by Zen that they have a maintenance window between 22:00 and 05:00 on the 18th/19th of July for “Essential Software upgrades” Expected impact duration: 2 x 30min outages within a 7hr window. During this time we may lose our peering to Zen.

We have been advised by Zen that they have a maintenance window between 22:00 and 05:00 on the 18th/19th of July for “Essential Software upgrades”

Expected impact duration: 2 x 30min outages within a 7hr window.

During this time we may lose our peering to Zen.

LON01 – EasyBond – 20:00 – 21:00 – 03/07/2014 *Controlled Failover Test* *COMPLETE*

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers. UPDATE01 – 20:54 Testing has completed and the service failed over as expected on 3 simulated failures

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers.

UPDATE01 – 20:54
Testing has completed and the service failed over as expected on 3 simulated failures

LON01 – EasyBond – 15:32 – 03/07/2014 *Outage*

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause. UPDATE01 – 16:12 We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be … Continue reading “LON01 – EasyBond – 15:32 – 03/07/2014 *Outage*”

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause.

UPDATE01 – 16:12
We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be published.

LON01 – ESXi01.R01 – 30/06/2014 *At Risk*

Our network monitoring has alerted us to an issue with ESXi.R01 whereby this server is not responding to remote management query’s correctly. All VM sessions on this server are still running correctly, however this could change without warning due to the de-graded nature of the server. Should this happen the EasyBond NOC will go offline. … Continue reading “LON01 – ESXi01.R01 – 30/06/2014 *At Risk*”

Our network monitoring has alerted us to an issue with ESXi.R01 whereby this server is not responding to remote management query’s correctly.

All VM sessions on this server are still running correctly, however this could change without warning due to the de-graded nature of the server. Should this happen the EasyBond NOC will go offline. This in itself won’t cause an outage for any EasyBond customers, however when the VM session restores this will cause a 30 second outage while the bonded service on the aggregators is reset by the NOC.

ESXi01.R01 will be rebooted outside of hours tonight to resolve this. The above detailed outage will also be seen.

LON01 – Frontier Systems BGP / Interconnect Session – 21:00- 22:00 – 27/06/2014 *Planned Maintenance* *COMPLETE*

We have just had the completed delivery of our fibre cross connect to Frontier System. We will be enabling a BGP session with them tonight in the above window. During this window our traffic to them will be disrupted while BGP reconverges with the new route information. UPDATE01 – 22:00 We have had issues establishing … Continue reading “LON01 – Frontier Systems BGP / Interconnect Session – 21:00- 22:00 – 27/06/2014 *Planned Maintenance* *COMPLETE*”

We have just had the completed delivery of our fibre cross connect to Frontier System. We will be enabling a BGP session with them tonight in the above window. During this window our traffic to them will be disrupted while BGP reconverges with the new route information.

UPDATE01 – 22:00
We have had issues establishing a BGP session to Frontier, We suspect this is due to a configuration issue on the remote side. We have raised a ticket with Frontier for investigation.

UPDATE02 – 01:00 – 28/06/2014UPDATE03 – 14:00 – 28/06/2014

London LINX

Our network monitoring has alerted us to an issue at the London LINX exchange. This will be having an impact on any customer who uses a 3rd party connection to connect to our services. UPDATE 01 – 11:32 Service has been restored and we are seeing traffic route via LINX again. As we have no … Continue reading “London LINX”

Our network monitoring has alerted us to an issue at the London LINX exchange. This will be having an impact on any customer who uses a 3rd party connection to connect to our services.

UPDATE 01 – 11:32
Service has been restored and we are seeing traffic route via LINX again. As we have no direct relationship or take services from LINX, there wont be a RFO.