LON01 – ESXi02.R01 & ESXi03.R01 – 14/08/2014 – 20:00 – 21:00 *Planned Maintenance* *COMPLETE*

There will be Planned Maintenance on ESXi02.R01 & ESXi03.R01 to resolve the ongoing remote management issue with these systems which a resolution has now been identified. Work will involve a reload of these boxes. During this reload EasyBOND services will be effected. Connections will however be manually transferred to the backup aggregation service prior to … Continue reading “LON01 – ESXi02.R01 & ESXi03.R01 – 14/08/2014 – 20:00 – 21:00 *Planned Maintenance* *COMPLETE*”

There will be Planned Maintenance on ESXi02.R01 & ESXi03.R01 to resolve the ongoing remote management issue with these systems which a resolution has now been identified.

Work will involve a reload of these boxes. During this reload EasyBOND services will be effected. Connections will however be manually transferred to the backup aggregation service prior to the work but users will see a sort 10-30 second outage.

We apologise for any inconvenience this may cause.

UPDATE 01 – 20:28
This work is complete and all services have been transferred back to their original primary servers. We will continue to monitor the ESXi hosts.

LON01 – EasyIPT – 10:52 – 12/08/2014 *Carrier Outage* – *RESOLVED*

At 10:52am our network monitoring alerted us to a issue with our Frontier voice interconnect. At this time we lost our BGP session to them and traffic started routing over diverse paths. This would have caused all active calls to drop while BGP reconverted. UPDATE01 – 11:10 At 11:02 our network monitoring showed that the … Continue reading “LON01 – EasyIPT – 10:52 – 12/08/2014 *Carrier Outage* – *RESOLVED*”

At 10:52am our network monitoring alerted us to a issue with our Frontier voice interconnect. At this time we lost our BGP session to them and traffic started routing over diverse paths. This would have caused all active calls to drop while BGP reconverted.

UPDATE01 – 11:10
At 11:02 our network monitoring showed that the interconnect had been resorted and the BGP session has resumed. We have logged a case with our carrier to see if the fault originated from there side as no other session losses were seen from our side.

UPDATE02 – 11:30
We have been advised this is a carrier issues with C4L.

LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators. We have corrected this configuration issue and … Continue reading “LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*”

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators.
We have corrected this configuration issue and completed a fail-over test which completed as expected.

Users on EasyBond would have seen 2 x 30 seconds outages. We apologise for any inconvenience caused.

LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration. We have made some changes to our core softswitch to eliminate … Continue reading “LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*”

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration.

We have made some changes to our core softswitch to eliminate this problem. These changes require a reload of the active configuration files on the server and will drop any active call. This reload should take no more than 30 seconds.

We apologise for any inconvenience this may cause.

UPDATE01 – 21:01
This work is complete.

EasyIPT – Inbound Calls – 12:45 – 21/07/2014

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service. UPDATE01 12:50 Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier. UPDATE02 13:23 Service has been restored; An … Continue reading “EasyIPT – Inbound Calls – 12:45 – 21/07/2014”

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service.

UPDATE01 12:50
Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier.

UPDATE02 13:23
Service has been restored; An RFO will be issued shortly. We apologise for any inconvenience caused.

UPDATE02 14:00
This issue has reoccured, the root of the problem has been identified and we are building an RFO that will be published shortly.

Outage Report
Major Incident Report – EasyIPT Outage 21-07-14

EasyXDSL – WBC Cannot connect Issue (ADSL2+) *RESOLVED*

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available. UPDATE01 17:30 This … Continue reading “EasyXDSL – WBC Cannot connect Issue (ADSL2+) *RESOLVED*”

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available.

UPDATE01 17:30
This has been confirmed as resolved by wholesale.

LON01 – Zen BGP Session – 22:00- 05:00 – 18/07/2014 *Planned Maintenance*

We have been advised by Zen that they have a maintenance window between 22:00 and 05:00 on the 18th/19th of July for “Essential Software upgrades” Expected impact duration: 2 x 30min outages within a 7hr window. During this time we may lose our peering to Zen.

We have been advised by Zen that they have a maintenance window between 22:00 and 05:00 on the 18th/19th of July for “Essential Software upgrades”

Expected impact duration: 2 x 30min outages within a 7hr window.

During this time we may lose our peering to Zen.

LON01 – EasyBond – 20:00 – 21:00 – 03/07/2014 *Controlled Failover Test* *COMPLETE*

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers. UPDATE01 – 20:54 Testing has completed and the service failed over as expected on 3 simulated failures

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers.

UPDATE01 – 20:54
Testing has completed and the service failed over as expected on 3 simulated failures

LON01 – EasyBond – 15:32 – 03/07/2014 *Outage*

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause. UPDATE01 – 16:12 We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be … Continue reading “LON01 – EasyBond – 15:32 – 03/07/2014 *Outage*”

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause.

UPDATE01 – 16:12
We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be published.

LON01 – ESXi01.R01 – 30/06/2014 *At Risk*

Our network monitoring has alerted us to an issue with ESXi.R01 whereby this server is not responding to remote management query’s correctly. All VM sessions on this server are still running correctly, however this could change without warning due to the de-graded nature of the server. Should this happen the EasyBond NOC will go offline. … Continue reading “LON01 – ESXi01.R01 – 30/06/2014 *At Risk*”

Our network monitoring has alerted us to an issue with ESXi.R01 whereby this server is not responding to remote management query’s correctly.

All VM sessions on this server are still running correctly, however this could change without warning due to the de-graded nature of the server. Should this happen the EasyBond NOC will go offline. This in itself won’t cause an outage for any EasyBond customers, however when the VM session restores this will cause a 30 second outage while the bonded service on the aggregators is reset by the NOC.

ESXi01.R01 will be rebooted outside of hours tonight to resolve this. The above detailed outage will also be seen.