Our engineers where alerted via our network monitoring to a DDoS attack (Distributed Denial of Service) on one of our ESXi servers. The attack and traffic has been mitigated however for the duration any directly connected services would have seen small amounts of packet loss. We are monitoring the network and apologise for any inconvenience … Continue reading “LON01 – 06/03/2014 – 10:10 – ESXi01 – DDoS Attack *RESOLVED*”
Our engineers where alerted via our network monitoring to a DDoS attack (Distributed Denial of Service) on one of our ESXi servers.
The attack and traffic has been mitigated however for the duration any directly connected services would have seen small amounts of packet loss.
We are monitoring the network and apologise for any inconvenience caused.
During an internal training exercise a “reload” command was issued to CORE01-LON instead of CORE01-HOR which resulted in a reboot of both the primary and secondary supervisor modules. This reload caused a 5 minute loss of service to LON01 for all directly connected services and customers. Safe guards will be put in place on LON01 … Continue reading “LON01 – 27/02/2014 – 16:10 – CORE01 – OUTAGE *RESOLVED*”
During an internal training exercise a “reload” command was issued to CORE01-LON instead of CORE01-HOR which resulted in a reboot of both the primary and secondary supervisor modules. This reload caused a 5 minute loss of service to LON01 for all directly connected services and customers.
Safe guards will be put in place on LON01 to prevent this happening again in such a scenario.
We apologise for any inconvenience caused.
We are currently aware of fibre break in the London area which is affecting outbound calls for some of our users. We have been informed by suppliers that this is due do fibre damage to which is currently being repaired. We are expecting an update by midday. Calls from customers outside of London to customers … Continue reading “Inbound / Outbound call Issue London Fibre Break 24/02/2014”
We are currently aware of fibre break in the London area which is affecting outbound calls for some of our users. We have been informed by suppliers that this is due do fibre damage to which is currently being repaired. We are expecting an update by midday.
Calls from customers outside of London to customers in London may fail. Broadband speeds should NOT be affected. This incident will be impacting BT customers and other operators.
In the mean time if you are experiencing issues please prefix the dialled number with 1280 for example 128002033014000. If you use a telephone system and normally prefix calls with 9 an example would be 9128002033014000.
We apologise for any inconvenience caused.
We are currently experiencing an issue that is preventing some DSL users connecting to our service. Our initial investigations indicate that this is a BT issue and we are working with them to resolve as soon as possible. The issue spans multiple nodes and is not geographically restrained to one area. We are aiming to … Continue reading “EasyxDSL Cannot Conenct Issues 20/02/2014 – 10:00”
We are currently experiencing an issue that is preventing some DSL users connecting to our service. Our initial investigations indicate that this is a BT issue and we are working with them to resolve as soon as possible. The issue spans multiple nodes and is not geographically restrained to one area. We are aiming to have the service restored as a matter of urgency.
-Update- 11:40
Customer sessions are now being restored and our monitoring indicates that the majority of services are now working again. We’ve yet to receive details from BT as to the cause but will update when we have further information.
We will shortly be taking ns02.easybond.co.uk offline to copy the virtual image over to another ESXi host for future upgrade work. No service impact is anticipated UPDATE01 – 22:05 *FINAL* This work is now complete.
We will shortly be taking ns02.easybond.co.uk offline to copy the virtual image over to another ESXi host for future upgrade work. No service impact is anticipated
UPDATE01 – 22:05 *FINAL*
This work is now complete.
Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator … Continue reading “LON01 Data Centre – 17/02/2014 – 22:00 – Planned Power Maintenance (B Feed)”
Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification
This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator banks.
Our core network and customer facing services (EasyHTTP, EasyIPT & EasyBOND) are fully power redundant and no service disruption is expected, however they will be classed as “at risk” for the 24 hours while the work is undertaken starting at 22:00 hours on the 17th Feb until 22:00 hours on the 18th Feb.
We will be attending site for the duration and taking the B feed out of service prior to the works in a controlled manor. Any customer services with only a single fed supply currently on B will be transferred to A prior to the work and returned once complete.
UPDATE 01 – 20:30 – 17/02/2014
Engineers have been onsite all day and prep work is now underway. Traffic is going to be transfered from AG-Primary to AG-Backup, this will cause a 30 second outage while devices transfer.
UPDATE 02 – 22:00 – 17/02/2014
An unexpected outage was seen for EasyBond circuits where an old version of the NOC came online due to ESXi01 not saving it’s startup configuration on shutdown. This has resulted in some stale sessions. Please power down your hardware for 20 minutes where power routers are in PPPOE mode or STATIC IP.
We are aware of an issue with our ticket system whereby the main website is not loading. It’s also possible emailed tickets are not being processed. As we do not host the site internally due to diversity, we are investigating the issue with our 3rd party provider. If your query is urgent then please contact … Continue reading “Ticket System – support.structuredcommunications.co.uk *RESOLVED*”
We are aware of an issue with our ticket system whereby the main website is not loading. It’s also possible emailed tickets are not being processed.
As we do not host the site internally due to diversity, we are investigating the issue with our 3rd party provider.
If your query is urgent then please contact us via phone.
UPDATE 01 – 12:21
We have been advised the root cause has now been addressed and this was caused by a DNS lookup limitation which has been removed while a long time solution is found.
We have had reports of performance issues on EasyHTTP. Engineers are looking at the issue. Services may be restarted without warning after 20:00 UPDATE01 – 18:30 The performance issues seem isolated to MySQL driven websites. UPDATE02 – 23:15 Engineers have installed Microsoft Wincache 1.3.5.5.5 and enabled the extension for PHP5.4 Various updates to the my.ini … Continue reading “EasyHTTP – 30/01/2014 – 18:00 – Maintenance *CLOSED*”
We have had reports of performance issues on EasyHTTP.
Engineers are looking at the issue. Services may be restarted without warning after 20:00
UPDATE01 – 18:30
The performance issues seem isolated to MySQL driven websites.
UPDATE02 – 23:15
Engineers have installed Microsoft Wincache 1.3.5.5.5 and enabled the extension for PHP5.4
Various updates to the my.ini files have been made and a server reload has been completed.
Website loading times have seen a 100% improvement and are around 2.5seconds, which is in line with various Windows based platforms.
We are aware of a fault where outbound calls are taking up to 30 seconds to connect. Engineers are currently investigating this and will update you as more information becomes available. UPDATE01 – 09:45 We are seeing outbound calls starting to connect as normal again. Inbound should also now be working. UPDATE01 – 10:30 We … Continue reading “EasyIPT – VoIP Calls – Outbound Delay *RESOLVED*”
We are aware of a fault where outbound calls are taking up to 30 seconds to connect. Engineers are currently investigating this and will update you as more information becomes available.
UPDATE01 – 09:45
We are seeing outbound calls starting to connect as normal again. Inbound should also now be working.
UPDATE01 – 10:30
We have requested further details from our wholesaler as to the cause of the issue.
We are aware ns02.r01.lon.adapt.easybond.co.uk is not processing DNS lookup requests. We are currently looking in to the issue and will provide an update shortly. NS01 (ns01.r01.lon.adapt.easybond.co.uk) is operating normally so this should not be service effecting. UPDATE 01 – 21:45 This issue has been resolved by re-loading the IPTABLE rules.
We are aware ns02.r01.lon.adapt.easybond.co.uk is not processing DNS lookup requests. We are currently looking in to the issue and will provide an update shortly. NS01 (ns01.r01.lon.adapt.easybond.co.uk) is operating normally so this should not be service effecting.
UPDATE 01 – 21:45
This issue has been resolved by re-loading the IPTABLE rules.