EasyHTTP – 16/04/2014 – 08:45 – CPU Usage *RESOLVED*

We are aware CPU usage on EasyHTTP is starting to climb to 100% across all 16 cores. We are monitoring the process that is causing the high usage with the view of restarting the server should usage not drop. A restart of the service has not resulted in a fix. UPDATE 01 – 08:53 The … Continue reading “EasyHTTP – 16/04/2014 – 08:45 – CPU Usage *RESOLVED*”

We are aware CPU usage on EasyHTTP is starting to climb to 100% across all 16 cores. We are monitoring the process that is causing the high usage with the view of restarting the server should usage not drop. A restart of the service has not resulted in a fix.

UPDATE 01 – 08:53
The IMAP service has continued to consume CPU usage and levels are now above 90%. Looking at the service threads we are unable to locate any sub service or string that would be causing the high CPU usage. We have therefore opted to restart EasyHTTP which will take around 10 minutes due to the size and configuration of the RAID array.

UPDATE 02 – 09:07
Due to a disk check being requested by the server due to uptime, we expect a further 15 minute delay.

UPDATE 03 – 09:30
The server has now rebooted and all services have been restored however the IMAP service is still using large amounts of the CPUs. We have taken action move the service to a single core so we can continue to fault find. Users may experience a slower email service due to the limits enforce.

UPDATE 04 -10:15
We have discovered a user account with over 700,000 emails in there deleted items which are suspected to be causing the high CPU load.

UPDATE 05 – 11:03
These files have been removed and the IMAP service restarted with all cores enabled. CPU usage is at normal levels. We will continue to monitor of the next few hours.

UPDATE 06 – 12:05
Our network monitor has alerted us that CPU levels have started to climb again. Further review of the IMAP process has high-listed another account of large size however items in stored in the users INBOX which we are unable to delete. We have therefore disabled the account and levels have returned back to normal.

LON01 – BGP Route HiJack – 20:40 – 02/04/2014 *RESOLVED*

We have been alerted to a RPKI Validation failure on both our /21 IP ranges. This notice advises that our IP space is now being miss announced by AS4761 (INDOSAT-INP-AP INDOSAT Internet Network Provider,ID) and has been accepted by some of their upstream peers who do not filter there BGP sessions correctly. We are working … Continue reading “LON01 – BGP Route HiJack – 20:40 – 02/04/2014 *RESOLVED*”

We have been alerted to a RPKI Validation failure on both our /21 IP ranges. This notice advises that our IP space is now being miss announced by AS4761 (INDOSAT-INP-AP INDOSAT Internet Network Provider,ID) and has been accepted by some of their upstream peers who do not filter there BGP sessions correctly.

We are working with our upstream providers to limit any routing issues that may be seen and will provide updates shortly.

UPDATE 01 – 21:00 – 02/04/2014

Our Chief Engineer (Adam Clark) has been dealing with the situation and can advise this is a wide spread issue affecting many thousands of networks and is a repeat of the INDOSAT issue seen in 2011. Since then many providers now filter results from INDOSAT and the RIPE RPKI validation being in use, this has limited the acceptance of our hijacked ranges. An email has also gone off to the INDOSAT contact who manages the network. We are awaiting a response.

At the moment no UK provider has accepted the wrong announcement and we are not seeing any UK related issues, however as many thousands of other networks are caught up in this you many experience problems connecting to the effected networks.

UPDATE 02 – 21:15 – 02/04/2014

We can now advise the only upstream provider of INDOSAT who accepted the announcement was AS4651 (THAI-GATEWAY The Communications Authority of Thailand(CAT),TH) Our upstream providers directly peer with AS4651 and therefore filtered the incorrect data.
As no further repeat alerts have been seen we are happy this has been resolved by INDOSAT although no official response has been given by them.
Should a repeat occur then this NOC notice will be reopened.

LON01 – 06/03/2014 – 10:10 – ESXi01 – DDoS Attack *RESOLVED*

Our engineers where alerted via our network monitoring to a DDoS attack (Distributed Denial of Service) on one of our ESXi servers. The attack and traffic has been mitigated however for the duration any directly connected services would have seen small amounts of packet loss. We are monitoring the network and apologise for any inconvenience … Continue reading “LON01 – 06/03/2014 – 10:10 – ESXi01 – DDoS Attack *RESOLVED*”

Our engineers where alerted via our network monitoring to a DDoS attack (Distributed Denial of Service) on one of our ESXi servers.

The attack and traffic has been mitigated however for the duration any directly connected services would have seen small amounts of packet loss.

We are monitoring the network and apologise for any inconvenience caused.

LON01 – 27/02/2014 – 16:10 – CORE01 – OUTAGE *RESOLVED*

During an internal training exercise a “reload” command was issued to CORE01-LON instead of CORE01-HOR which resulted in a reboot of both the primary and secondary supervisor modules. This reload caused a 5 minute loss of service to LON01 for all directly connected services and customers. Safe guards will be put in place on LON01 … Continue reading “LON01 – 27/02/2014 – 16:10 – CORE01 – OUTAGE *RESOLVED*”

During an internal training exercise a “reload” command was issued to CORE01-LON instead of CORE01-HOR which resulted in a reboot of both the primary and secondary supervisor modules. This reload caused a 5 minute loss of service to LON01 for all directly connected services and customers.

Safe guards will be put in place on LON01 to prevent this happening again in such a scenario.

We apologise for any inconvenience caused.

Inbound / Outbound call Issue London Fibre Break 24/02/2014

We are currently aware of fibre break in the London area which is affecting outbound calls for some of our users. We have been informed by suppliers that this is due do fibre damage to which is currently being repaired. We are expecting an update by midday. Calls from customers outside of London to customers … Continue reading “Inbound / Outbound call Issue London Fibre Break 24/02/2014”

We are currently aware of fibre break in the London area which is affecting outbound calls for some of our users. We have been informed by suppliers that this is due do fibre damage to which is currently being repaired. We are expecting an update by midday.

Calls from customers outside of London to customers in London may fail. Broadband speeds should NOT be affected. This incident will be impacting BT customers and other operators.

In the mean time if you are experiencing issues please prefix the dialled number with 1280 for example 128002033014000. If you use a telephone system and normally prefix calls with 9 an example would be 9128002033014000.

We apologise for any inconvenience caused.

EasyxDSL Cannot Conenct Issues 20/02/2014 – 10:00

We are currently experiencing an issue that is preventing some DSL users connecting to our service. Our initial investigations indicate that this is a BT issue and we are working with them to resolve as soon as possible. The issue spans multiple nodes and is not geographically restrained to one area. We are aiming to … Continue reading “EasyxDSL Cannot Conenct Issues 20/02/2014 – 10:00”

We are currently experiencing an issue that is preventing some DSL users connecting to our service. Our initial investigations indicate that this is a BT issue and we are working with them to resolve as soon as possible. The issue spans multiple nodes and is not geographically restrained to one area. We are aiming to have the service restored as a matter of urgency.

-Update- 11:40
Customer sessions are now being restored and our monitoring indicates that the majority of services are now working again. We’ve yet to receive details from BT as to the cause but will update when we have further information.

EasyBOND NS02 – 10/02/2014 – 22:00 – Planned Maintenance *COMPLETE*

We will shortly be taking ns02.easybond.co.uk offline to copy the virtual image over to another ESXi host for future upgrade work. No service impact is anticipated UPDATE01 – 22:05 *FINAL* This work is now complete.

We will shortly be taking ns02.easybond.co.uk offline to copy the virtual image over to another ESXi host for future upgrade work. No service impact is anticipated

UPDATE01 – 22:05 *FINAL*
This work is now complete.

LON01 Data Centre – 17/02/2014 – 22:00 – Planned Power Maintenance (B Feed)

Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator … Continue reading “LON01 Data Centre – 17/02/2014 – 22:00 – Planned Power Maintenance (B Feed)”

Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification

This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator banks.

Our core network and customer facing services (EasyHTTP, EasyIPT & EasyBOND) are fully power redundant and no service disruption is expected, however they will be classed as “at risk” for the 24 hours while the work is undertaken starting at 22:00 hours on the 17th Feb until 22:00 hours on the 18th Feb.

We will be attending site for the duration and taking the B feed out of service prior to the works in a controlled manor. Any customer services with only a single fed supply currently on B will be transferred to A prior to the work and returned once complete.

UPDATE 01 – 20:30 – 17/02/2014
Engineers have been onsite all day and prep work is now underway. Traffic is going to be transfered from AG-Primary to AG-Backup, this will cause a 30 second outage while devices transfer.

UPDATE 02 – 22:00 – 17/02/2014
An unexpected outage was seen for EasyBond circuits where an old version of the NOC came online due to ESXi01 not saving it’s startup configuration on shutdown. This has resulted in some stale sessions. Please power down your hardware for 20 minutes where power routers are in PPPOE mode or STATIC IP.

Ticket System – support.structuredcommunications.co.uk *RESOLVED*

We are aware of an issue with our ticket system whereby the main website is not loading. It’s also possible emailed tickets are not being processed. As we do not host the site internally due to diversity, we are investigating the issue with our 3rd party provider. If your query is urgent then please contact … Continue reading “Ticket System – support.structuredcommunications.co.uk *RESOLVED*”

We are aware of an issue with our ticket system whereby the main website is not loading. It’s also possible emailed tickets are not being processed.

As we do not host the site internally due to diversity, we are investigating the issue with our 3rd party provider.

If your query is urgent then please contact us via phone.

UPDATE 01 – 12:21
We have been advised the root cause has now been addressed and this was caused by a DNS lookup limitation which has been removed while a long time solution is found.

EasyHTTP – 30/01/2014 – 18:00 – Maintenance *CLOSED*

We have had reports of performance issues on EasyHTTP. Engineers are looking at the issue. Services may be restarted without warning after 20:00 UPDATE01 – 18:30 The performance issues seem isolated to MySQL driven websites. UPDATE02 – 23:15 Engineers have installed Microsoft Wincache 1.3.5.5.5 and enabled the extension for PHP5.4 Various updates to the my.ini … Continue reading “EasyHTTP – 30/01/2014 – 18:00 – Maintenance *CLOSED*”

We have had reports of performance issues on EasyHTTP.
Engineers are looking at the issue. Services may be restarted without warning after 20:00

UPDATE01 – 18:30
The performance issues seem isolated to MySQL driven websites.

UPDATE02 – 23:15
Engineers have installed Microsoft Wincache 1.3.5.5.5 and enabled the extension for PHP5.4
Various updates to the my.ini files have been made and a server reload has been completed.

Website loading times have seen a 100% improvement and are around 2.5seconds, which is in line with various Windows based platforms.