Adam Clark (Network Operations) – Page 19 – Structured Communications NOC

EasyXDSL – Entanet WBC & IPSC Cannot connect Issues – 30/09/2014 – 20:10 RESOLVED

We are aware of a DSL issue affecting our wholesale provider Entanet. This is not localised to any part of the UK but as a result people are not able to connect to the internet. Customers on our EasyBOND platform with dual ISP reliance will be seeing a reduced service. Once more information becomes available … Continue reading “EasyXDSL – Entanet WBC & IPSC Cannot connect Issues – 30/09/2014 – 20:10 *RESOLVED*”

We are aware of a DSL issue affecting our wholesale provider Entanet. This is not localised to any part of the UK but as a result people are not able to connect to the internet.

Customers on our EasyBOND platform with dual ISP reliance will be seeing a reduced service.

Once more information becomes available we will update this notice.

UPDATE01 – 20:18
We have received an update to advise Entanet are seeing an issue with telehouse-east.core.enta.net and are working to resolve the issue.

UPDATE02 – 20:25
We have seen all DSL tails return to service but are awaiting official confirmation. We have also requested an RFO from Entanet

UPDATE03 – 03/10/2014
RFO can be found HERE

LON01 – LON01-CORE – 25/09/2014 – 20:00 till 22:00 Planned Maintenance COMPLETE

On the 25/09/2014 between the hours of 20:00 – 22:00 we will be replacing a line card in our LON01-CORE following on from the failure seen on the 28/08/2014. The works will involve disconnecting all devices that are connected to that line card. Any server that has a redundant feed via another line card will … Continue reading “LON01 – LON01-CORE – 25/09/2014 – 20:00 till 22:00 *Planned Maintenance* *COMPLETE*”

On the 25/09/2014 between the hours of 20:00 – 22:00 we will be replacing a line card in our LON01-CORE following on from the failure seen on the 28/08/2014.

The works will involve disconnecting all devices that are connected to that line card. Any server that has a redundant feed via another line card will remain online but “at risk” Any server with a single feed will go offline for the duration. Where customer’s PBXs are connected to this line card, they will see a loss of service and may require a reboot of their handsets / system following the works to clear any stale NAT sessions.

UPDATE01 – 20:10
Work on this replacement has started and engineers have started to disconnect devices connected to the S3 line card.

UPDATE02 – 20:50
The card has been replaced and devices have been reconnected. The card is now being placed in to a test mode to complete a series of self tests. Ports may reset and services are still classed as at risk.

UPDATE02 – 22:50
Testing complete and the line card is now back in service. Any users who don’t have a connection to there PBX are advised to power down your hardware and then power back on.

LON01 – EasyIPT – Unmanaged Accounts – BGP Changes – 18:00-18:10 – 02/09/2014 COMPLETE

In line with the previous correspondence sent out to all unmanaged EasyIPT accounts whereby a customer has a direct connection to the carrier and has not migrated over to our in-house platform (Sipwise), then the following changes are due to take place tonight on our network as we no longer route managed traffic to this … Continue reading “LON01 – EasyIPT – Unmanaged Accounts – BGP Changes – 18:00-18:10 – 02/09/2014 *COMPLETE*”

1)Removal of the 146.101.248.200/26 network prefix via our private Frontier / Voiceflex interconnects.

Non migrated accounts will also not benefit from our redundant outbound routing in the event of a carrier failure, nore the additional in-house call fraud monitoring services offered via Sipwise. Unmanaged customers are reminded that call fraud protection is there sole responsibility to protect against and will be liable for any occurrences and financial costs.

LON01 – LON01-CORE – 28/08/2014 – 17:10 OUTAGE RESOLVED

At 17:11 our networking monitoring advised to a failure on our network which was localised to a single rack within LON01. Initial diagnostics started and it was assumed there was a power issue within Rack03 as all devices on Feed A where showing as “unreachable” Further diagnostic showed this not to be the case but … Continue reading “LON01 – LON01-CORE – 28/08/2014 – 17:10 *OUTAGE* *RESOLVED*”

At 17:11 our networking monitoring advised to a failure on our network which was localised to a single rack within LON01.

Initial diagnostics started and it was assumed there was a power issue within Rack03 as all devices on Feed A where showing as “unreachable”

Further diagnostic showed this not to be the case but a line card within LON01-CORE having been reloaded by the primary supervisor due to a series of automated test failures.

UPDATE01 – 17:18

The module has come back online and all traffic has resumed. Engineers are currently working on why this occurred.

Virgin Media Leased Lines OUTAGE RESOLVED

We are currently aware of an incident on Virgin Media’s network which is causing a loss of service to several Virgin Media provided leased lines. Virgin Media are currently investigating and updates will be provided as and when they become available. UPDATE01 – 10:52 Virgin Media have identified a problem with one of their core … Continue reading “Virgin Media Leased Lines *OUTAGE* *RESOLVED*”

We are currently aware of an incident on Virgin Media’s network which is causing a loss of service to several Virgin Media provided leased lines.

Virgin Media are currently investigating and updates will be provided as and when they become available.

UPDATE01 – 10:52

Virgin Media have identified a problem with one of their core routers in Telehouse and they are continuing their investigation although no ETR is available as of yet.

UPDATE02 – 11:00

We are seeing service restore back to Virgin media circuts, however this has not been confirmed by Virgin yet.

UPDATE03 – 12:00
We have requested an RFO from Virgin which we will make avalible via the NOC

LON01 – ESXi02.R01 & ESXi03.R01 – 14/08/2014 – 20:00 – 21:00 Planned Maintenance COMPLETE

There will be Planned Maintenance on ESXi02.R01 & ESXi03.R01 to resolve the ongoing remote management issue with these systems which a resolution has now been identified. Work will involve a reload of these boxes. During this reload EasyBOND services will be effected. Connections will however be manually transferred to the backup aggregation service prior to … Continue reading “LON01 – ESXi02.R01 & ESXi03.R01 – 14/08/2014 – 20:00 – 21:00 *Planned Maintenance* *COMPLETE*”

There will be Planned Maintenance on ESXi02.R01 & ESXi03.R01 to resolve the ongoing remote management issue with these systems which a resolution has now been identified.

Work will involve a reload of these boxes. During this reload EasyBOND services will be effected. Connections will however be manually transferred to the backup aggregation service prior to the work but users will see a sort 10-30 second outage.

We apologise for any inconvenience this may cause.

UPDATE 01 – 20:28
This work is complete and all services have been transferred back to their original primary servers. We will continue to monitor the ESXi hosts.

LON01 – EasyIPT – 10:52 – 12/08/2014 Carrier Outage – RESOLVED

At 10:52am our network monitoring alerted us to a issue with our Frontier voice interconnect. At this time we lost our BGP session to them and traffic started routing over diverse paths. This would have caused all active calls to drop while BGP reconverted. UPDATE01 – 11:10 At 11:02 our network monitoring showed that the … Continue reading “LON01 – EasyIPT – 10:52 – 12/08/2014 *Carrier Outage* – *RESOLVED*”

UPDATE01 – 11:10
At 11:02 our network monitoring showed that the interconnect had been resorted and the BGP session has resumed. We have logged a case with our carrier to see if the fault originated from there side as no other session losses were seen from our side.

UPDATE02 – 11:30
We have been advised this is a carrier issues with C4L.

LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work COMPLETE

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators. We have corrected this configuration issue and … Continue reading “LON01 – EasyBond / LON01-CORE – 20:05 – 03/08/2014 – Emergency Work *COMPLETE*”

During the emergency reboot of ESXI02.R01 following our network monitoring advising the management interface was showing as off-line. We discovered an issue with the ARP timeout settings on LON01-CORE core that could have caused extended downtime in the event that the EasyBond service needed to flip between aggregators.
We have corrected this configuration issue and completed a fail-over test which completed as expected.

Users on EasyBond would have seen 2 x 30 seconds outages. We apologise for any inconvenience caused.

LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 Planned Maintenance COMPLETE

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration. We have made some changes to our core softswitch to eliminate … Continue reading “LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*”

We have made some changes to our core softswitch to eliminate this problem. These changes require a reload of the active configuration files on the server and will drop any active call. This reload should take no more than 30 seconds.

We apologise for any inconvenience this may cause.

UPDATE01 – 21:01
This work is complete.

EasyIPT – Inbound Calls – 12:45 – 21/07/2014

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service. UPDATE01 12:50 Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier. UPDATE02 13:23 Service has been restored; An … Continue reading “EasyIPT – Inbound Calls – 12:45 – 21/07/2014”

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service.

UPDATE01 12:50
Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier.

UPDATE02 13:23
Service has been restored; An RFO will be issued shortly. We apologise for any inconvenience caused.

UPDATE02 14:00
This issue has reoccured, the root of the problem has been identified and we are building an RFO that will be published shortly.

Outage Report
Major Incident Report – EasyIPT Outage 21-07-14