Author: Adam Clark (Network Operations)

LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration. We have made some changes to our core softswitch to eliminate … Continue reading “LON01 – EasyIPT – 21:00- 21:05 – 24/07/2014 *Planned Maintenance* *COMPLETE*”

Following on from a known bug in Asterisks whereby a PBX sends an occasional “unauthorized registration” message to our softswitch following on from a SIP OPTIONS request resulting in an account ban if received more than 3 times during the life of the registration.

We have made some changes to our core softswitch to eliminate this problem. These changes require a reload of the active configuration files on the server and will drop any active call. This reload should take no more than 30 seconds.

We apologise for any inconvenience this may cause.

UPDATE01 – 21:01
This work is complete.

EasyIPT – Inbound Calls – 12:45 – 21/07/2014

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service. UPDATE01 12:50 Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier. UPDATE02 13:23 Service has been restored; An … Continue reading “EasyIPT – Inbound Calls – 12:45 – 21/07/2014”

We are aware of an issue with inbound calls via one of our carriers. Engineers are working to urgently restore service.

UPDATE01 12:50
Service was restored however has been lost again. Outbound calls are routing over alternative carriers. Our senior engineer is currently dealing directly with the carrier.

UPDATE02 13:23
Service has been restored; An RFO will be issued shortly. We apologise for any inconvenience caused.

UPDATE02 14:00
This issue has reoccured, the root of the problem has been identified and we are building an RFO that will be published shortly.

Outage Report
Major Incident Report – EasyIPT Outage 21-07-14

EasyXDSL – WBC Cannot connect Issue (ADSL2+) *RESOLVED*

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available. UPDATE01 17:30 This … Continue reading “EasyXDSL – WBC Cannot connect Issue (ADSL2+) *RESOLVED*”

We have been advised by our wholesale partner ENTANET that where is BT’s network problem preventing some DSL users connecting. The issue spans multiple nodes and is not geographically restrained to one area. They are investigating this as a matter of urgency and further updates will be provided when they become available.

UPDATE01 17:30
This has been confirmed as resolved by wholesale.

LON01 – Zen BGP Session – 22:00- 05:00 – 18/07/2014 *Planned Maintenance*

We have been advised by Zen that they have a maintenance window between 22:00 and 05:00 on the 18th/19th of July for “Essential Software upgrades” Expected impact duration: 2 x 30min outages within a 7hr window. During this time we may lose our peering to Zen.

We have been advised by Zen that they have a maintenance window between 22:00 and 05:00 on the 18th/19th of July for “Essential Software upgrades”

Expected impact duration: 2 x 30min outages within a 7hr window.

During this time we may lose our peering to Zen.

LON01 – EasyBond – 20:00 – 21:00 – 03/07/2014 *Controlled Failover Test* *COMPLETE*

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers. UPDATE01 – 20:54 Testing has completed and the service failed over as expected on 3 simulated failures

We will be conducting a controlled failover test to our backup aggregation server following today’s failure to ensure this transfer process is working correctly. A few small outages will be seen during this window for bonded customers.

UPDATE01 – 20:54
Testing has completed and the service failed over as expected on 3 simulated failures

LON01 – EasyBond – 15:32 – 03/07/2014 *Outage*

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause. UPDATE01 – 16:12 We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be … Continue reading “LON01 – EasyBond – 15:32 – 03/07/2014 *Outage*”

We are aware of an outage that has just taken place on our bonded platform and we are working to find the root cause.

UPDATE01 – 16:12
We have been able to restore service and are reviewing the logs with out software vendor. Once we know the cause of the outage an RFO will be published.

LON01 – ESXi01.R01 – 30/06/2014 *At Risk*

Our network monitoring has alerted us to an issue with ESXi.R01 whereby this server is not responding to remote management query’s correctly. All VM sessions on this server are still running correctly, however this could change without warning due to the de-graded nature of the server. Should this happen the EasyBond NOC will go offline. … Continue reading “LON01 – ESXi01.R01 – 30/06/2014 *At Risk*”

Our network monitoring has alerted us to an issue with ESXi.R01 whereby this server is not responding to remote management query’s correctly.

All VM sessions on this server are still running correctly, however this could change without warning due to the de-graded nature of the server. Should this happen the EasyBond NOC will go offline. This in itself won’t cause an outage for any EasyBond customers, however when the VM session restores this will cause a 30 second outage while the bonded service on the aggregators is reset by the NOC.

ESXi01.R01 will be rebooted outside of hours tonight to resolve this. The above detailed outage will also be seen.

LON01 – Frontier Systems BGP / Interconnect Session – 21:00- 22:00 – 27/06/2014 *Planned Maintenance* *COMPLETE*

We have just had the completed delivery of our fibre cross connect to Frontier System. We will be enabling a BGP session with them tonight in the above window. During this window our traffic to them will be disrupted while BGP reconverges with the new route information. UPDATE01 – 22:00 We have had issues establishing … Continue reading “LON01 – Frontier Systems BGP / Interconnect Session – 21:00- 22:00 – 27/06/2014 *Planned Maintenance* *COMPLETE*”

We have just had the completed delivery of our fibre cross connect to Frontier System. We will be enabling a BGP session with them tonight in the above window. During this window our traffic to them will be disrupted while BGP reconverges with the new route information.

UPDATE01 – 22:00
We have had issues establishing a BGP session to Frontier, We suspect this is due to a configuration issue on the remote side. We have raised a ticket with Frontier for investigation.

UPDATE02 – 01:00 – 28/06/2014UPDATE03 – 14:00 – 28/06/2014

London LINX

Our network monitoring has alerted us to an issue at the London LINX exchange. This will be having an impact on any customer who uses a 3rd party connection to connect to our services. UPDATE 01 – 11:32 Service has been restored and we are seeing traffic route via LINX again. As we have no … Continue reading “London LINX”

Our network monitoring has alerted us to an issue at the London LINX exchange. This will be having an impact on any customer who uses a 3rd party connection to connect to our services.

UPDATE 01 – 11:32
Service has been restored and we are seeing traffic route via LINX again. As we have no direct relationship or take services from LINX, there wont be a RFO.

EasyIPT – Network Changes

In line with a previous NOC announcement whereby a BT IPX outage caused service disruption for our EasyIPT VoIP platform (both managed and unmanaged) Structured Communications have, as a result invested in a redundant carrier grade switch along with private bulk fibre and cross-connects to new and existing voice carriers. (known as upstream providers) This … Continue reading “EasyIPT – Network Changes”

In line with a previous NOC announcement whereby a BT IPX outage caused service disruption for our EasyIPT VoIP platform (both managed and unmanaged) Structured Communications have, as a result invested in a redundant carrier grade switch along with private bulk fibre and cross-connects to new and existing voice carriers. (known as upstream providers)

This solution enables us to directly peer with our upstream providers taking out the public internet and resulting in a more robust connection.

These new and existing providers directly terminate in to our carrier grade switch via private SST trunks and our private interconnects. This gives Structured Communications diverse routing in the event of an upstream network issue.

Customers who have a fully managed system with Structured Communications will be automatically migrated outside of hours.

Customers who simply take a SIP trunk from us via our existing carrier will be required to make changes to your PBX. (You will be contacted shortly via the email address on your account)

We are still awaiting the completion of various cross connects from their respective network providers and expect this to take no more than 2 weeks with the migration completed with 4 weeks.

If you have any questions then please don’t hesitate to contact us.