Author: Adam Clark (Network Operations)

LON01 – Frontier Interconnect *Resolved*

Our network monitoring has alerted us to a loss of BGP peering with Frontier Systems. Initial calls to our Layer2 provider for this link (C4L) advise they are aware of a spanning tree issue within their network. Traffic has automatically been re-routed around the link and no service disruption is being seen. UPDATE01 – 10:20 … Continue reading “LON01 – Frontier Interconnect *Resolved*”

Our network monitoring has alerted us to a loss of BGP peering with Frontier Systems. Initial calls to our Layer2 provider for this link (C4L) advise they are aware of a spanning tree issue within their network.

Traffic has automatically been re-routed around the link and no service disruption is being seen.

UPDATE01 – 10:20 – 05/10/3014
After a call to C4L following our link still being offline, we had been advised our VLAN had not been rebuilt on the trunk port where STP had been disabled. The service has now been restored and we have esculated this to our account manager.

LON01 – EasyIPT, Unmanaged SIP Accounts ***Service / Settings Change Notice***

Following on from previous correspondence in respect to unmanaged SIP accounts we have seen the remaining unmanaged 3rd party PBXs connect to our new platform. As a result of this and in order to provide better levels of support we will only be providing accounts via our EasyIPT SIP solution from the 01/11/2014. If you … Continue reading “LON01 – EasyIPT, Unmanaged SIP Accounts ***Service / Settings Change Notice***”

Following on from previous correspondence in respect to unmanaged SIP accounts we have seen the remaining unmanaged 3rd party PBXs connect to our new platform. As a result of this and in order to provide better levels of support we will only be providing accounts via our EasyIPT SIP solution from the 01/11/2014.

If you have not yet migrated over to your new trunk, then we request you do so by the 31/10/2014 to avoid any disruption in voice services.

Changes will be made to the number formatting of your PBX as the invite format has changed to canonical. Example: “442030000000” This number formatting can be added alongside existing PBX numbers to avoid anytime downtime. You will be able to instantly make outbound calls once your system is registered and once you are ready to swap over inbound calls, please contact support who will complete this for you.

If you are having trouble migrating your system or have any questions, please don’t hesitate to contact us directly.

EasyXDSL – Entanet WBC & IPSC Cannot connect Issues – 30/09/2014 – 20:10 *RESOLVED*

We are aware of a DSL issue affecting our wholesale provider Entanet. This is not localised to any part of the UK but as a result people are not able to connect to the internet. Customers on our EasyBOND platform with dual ISP reliance will be seeing a reduced service. Once more information becomes available … Continue reading “EasyXDSL – Entanet WBC & IPSC Cannot connect Issues – 30/09/2014 – 20:10 *RESOLVED*”

We are aware of a DSL issue affecting our wholesale provider Entanet. This is not localised to any part of the UK but as a result people are not able to connect to the internet.

Customers on our EasyBOND platform with dual ISP reliance will be seeing a reduced service.

Once more information becomes available we will update this notice.

UPDATE01 – 20:18
We have received an update to advise Entanet are seeing an issue with telehouse-east.core.enta.net and are working to resolve the issue.

UPDATE02 – 20:25
We have seen all DSL tails return to service but are awaiting official confirmation. We have also requested an RFO from Entanet

UPDATE03 – 03/10/2014
RFO can be found HERE

LON01 – LON01-CORE – 25/09/2014 – 20:00 till 22:00 *Planned Maintenance* *COMPLETE*

On the 25/09/2014 between the hours of 20:00 – 22:00 we will be replacing a line card in our LON01-CORE following on from the failure seen on the 28/08/2014. The works will involve disconnecting all devices that are connected to that line card. Any server that has a redundant feed via another line card will … Continue reading “LON01 – LON01-CORE – 25/09/2014 – 20:00 till 22:00 *Planned Maintenance* *COMPLETE*”

On the 25/09/2014 between the hours of 20:00 – 22:00 we will be replacing a line card in our LON01-CORE following on from the failure seen on the 28/08/2014.

The works will involve disconnecting all devices that are connected to that line card. Any server that has a redundant feed via another line card will remain online but “at risk” Any server with a single feed will go offline for the duration. Where customer’s PBXs are connected to this line card, they will see a loss of service and may require a reboot of their handsets / system following the works to clear any stale NAT sessions.

UPDATE01 – 20:10
Work on this replacement has started and engineers have started to disconnect devices connected to the S3 line card.

UPDATE02 – 20:50
The card has been replaced and devices have been reconnected. The card is now being placed in to a test mode to complete a series of self tests. Ports may reset and services are still classed as at risk.

UPDATE02 – 22:50
Testing complete and the line card is now back in service. Any users who don’t have a connection to there PBX are advised to power down your hardware and then power back on.

LON01 – EasyIPT – Unmanaged Accounts – BGP Changes – 18:00-18:10 – 02/09/2014 *COMPLETE*

In line with the previous correspondence sent out to all unmanaged EasyIPT accounts whereby a customer has a direct connection to the carrier and has not migrated over to our in-house platform (Sipwise), then the following changes are due to take place tonight on our network as we no longer route managed traffic to this … Continue reading “LON01 – EasyIPT – Unmanaged Accounts – BGP Changes – 18:00-18:10 – 02/09/2014 *COMPLETE*”

In line with the previous correspondence sent out to all unmanaged EasyIPT accounts whereby a customer has a direct connection to the carrier and has not migrated over to our in-house platform (Sipwise), then the following changes are due to take place tonight on our network as we no longer route managed traffic to this IP range.

1)Removal of the 146.101.248.200/26 network prefix via our private Frontier / Voiceflex interconnects.

Non migrated accounts will also not benefit from our redundant outbound routing in the event of a carrier failure, nore the additional in-house call fraud monitoring services offered via Sipwise. Unmanaged customers are reminded that call fraud protection is there sole responsibility to protect against and will be liable for any occurrences and financial costs.

LON01 – LON01-CORE – 28/08/2014 – 17:10 *OUTAGE* *RESOLVED*

At 17:11 our networking monitoring advised to a failure on our network which was localised to a single rack within LON01. Initial diagnostics started and it was assumed there was a power issue within Rack03 as all devices on Feed A where showing as “unreachable” Further diagnostic showed this not to be the case but … Continue reading “LON01 – LON01-CORE – 28/08/2014 – 17:10 *OUTAGE* *RESOLVED*”

At 17:11 our networking monitoring advised to a failure on our network which was localised to a single rack within LON01.

Initial diagnostics started and it was assumed there was a power issue within Rack03 as all devices on Feed A where showing as “unreachable”

Further diagnostic showed this not to be the case but a line card within LON01-CORE having been reloaded by the primary supervisor due to a series of automated test failures.

UPDATE01 – 17:18

The module has come back online and all traffic has resumed. Engineers are currently working on why this occurred.

Virgin Media Leased Lines *OUTAGE* *RESOLVED*

We are currently aware of an incident on Virgin Media’s network which is causing a loss of service to several Virgin Media provided leased lines. Virgin Media are currently investigating and updates will be provided as and when they become available. UPDATE01 – 10:52 Virgin Media have identified a problem with one of their core … Continue reading “Virgin Media Leased Lines *OUTAGE* *RESOLVED*”

We are currently aware of an incident on Virgin Media’s network which is causing a loss of service to several Virgin Media provided leased lines.

Virgin Media are currently investigating and updates will be provided as and when they become available.

UPDATE01 – 10:52

Virgin Media have identified a problem with one of their core routers in Telehouse and they are continuing their investigation although no ETR is available as of yet.

UPDATE02 – 11:00

We are seeing service restore back to Virgin media circuts, however this has not been confirmed by Virgin yet.

UPDATE03 – 12:00
We have requested an RFO from Virgin which we will make avalible via the NOC

LON01 – ESXi02.R01 & ESXi03.R01 – 14/08/2014 – 20:00 – 21:00 *Planned Maintenance* *COMPLETE*

There will be Planned Maintenance on ESXi02.R01 & ESXi03.R01 to resolve the ongoing remote management issue with these systems which a resolution has now been identified. Work will involve a reload of these boxes. During this reload EasyBOND services will be effected. Connections will however be manually transferred to the backup aggregation service prior to … Continue reading “LON01 – ESXi02.R01 & ESXi03.R01 – 14/08/2014 – 20:00 – 21:00 *Planned Maintenance* *COMPLETE*”

There will be Planned Maintenance on ESXi02.R01 & ESXi03.R01 to resolve the ongoing remote management issue with these systems which a resolution has now been identified.

Work will involve a reload of these boxes. During this reload EasyBOND services will be effected. Connections will however be manually transferred to the backup aggregation service prior to the work but users will see a sort 10-30 second outage.

We apologise for any inconvenience this may cause.

UPDATE 01 – 20:28
This work is complete and all services have been transferred back to their original primary servers. We will continue to monitor the ESXi hosts.

LON01 – EasyIPT – 10:52 – 12/08/2014 *Carrier Outage* – *RESOLVED*

At 10:52am our network monitoring alerted us to a issue with our Frontier voice interconnect. At this time we lost our BGP session to them and traffic started routing over diverse paths. This would have caused all active calls to drop while BGP reconverted. UPDATE01 – 11:10 At 11:02 our network monitoring showed that the … Continue reading “LON01 – EasyIPT – 10:52 – 12/08/2014 *Carrier Outage* – *RESOLVED*”

At 10:52am our network monitoring alerted us to a issue with our Frontier voice interconnect. At this time we lost our BGP session to them and traffic started routing over diverse paths. This would have caused all active calls to drop while BGP reconverted.

UPDATE01 – 11:10
At 11:02 our network monitoring showed that the interconnect had been resorted and the BGP session has resumed. We have logged a case with our carrier to see if the fault originated from there side as no other session losses were seen from our side.

UPDATE02 – 11:30
We have been advised this is a carrier issues with C4L.

LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators. We have corrected this configuration issue and … Continue reading “LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*”

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators.
We have corrected this configuration issue and completed a fail-over test which completed as expected.

Users on EasyBond would have seen 2 x 30 seconds outages. We apologise for any inconvenience caused.