EasyHTTP – 25/04/2014 – 10:45 – Blacklist / SMTP

We are aware and confirmed by our network monitoring that one of our SMTP servers (95.87.104.30) was used to send out large volume of emails last night from a compromised account. This account has been suspended and actions have been taken to remove the listings from the various listing DNSBL databases. This may take a few hours for remote server to reflect this.

We apologise for any inconvenience caused

EasyHTTP – 16/04/2014 – 20:00 – Maintenance *COMPLETE*

Following on from today’s CPU issues, We have been advised there is a major upgrade for Mail Enable. This will be installed tonight to ensure we are running the latest release.#

UPDATE 01 – 20:15
This work is now complete. Any users having problems with account syncing are advised to remove and re-add there account to their mail client. (Remember to back up your messages first)

EasyHTTP – 16/04/2014 – 08:45 – CPU Usage *RESOLVED*

We are aware CPU usage on EasyHTTP is starting to climb to 100% across all 16 cores. We are monitoring the process that is causing the high usage with the view of restarting the server should usage not drop. A restart of the service has not resulted in a fix.

UPDATE 01 – 08:53
The IMAP service has continued to consume CPU usage and levels are now above 90%. Looking at the service threads we are unable to locate any sub service or string that would be causing the high CPU usage. We have therefore opted to restart EasyHTTP which will take around 10 minutes due to the size and configuration of the RAID array.

UPDATE 02 – 09:07
Due to a disk check being requested by the server due to uptime, we expect a further 15 minute delay.

UPDATE 03 – 09:30
The server has now rebooted and all services have been restored however the IMAP service is still using large amounts of the CPUs. We have taken action move the service to a single core so we can continue to fault find. Users may experience a slower email service due to the limits enforce.

UPDATE 04 -10:15
We have discovered a user account with over 700,000 emails in there deleted items which are suspected to be causing the high CPU load.

UPDATE 05 – 11:03
These files have been removed and the IMAP service restarted with all cores enabled. CPU usage is at normal levels. We will continue to monitor of the next few hours.

UPDATE 06 – 12:05
Our network monitor has alerted us that CPU levels have started to climb again. Further review of the IMAP process has high-listed another account of large size however items in stored in the users INBOX which we are unable to delete. We have therefore disabled the account and levels have returned back to normal.

LON01 – BGP Route HiJack – 20:40 – 02/04/2014 *RESOLVED*

We have been alerted to a RPKI Validation failure on both our /21 IP ranges. This notice advises that our IP space is now being miss announced by AS4761 (INDOSAT-INP-AP INDOSAT Internet Network Provider,ID) and has been accepted by some of their upstream peers who do not filter there BGP sessions correctly.

We are working with our upstream providers to limit any routing issues that may be seen and will provide updates shortly.

UPDATE 01 – 21:00 – 02/04/2014

Our Chief Engineer (Adam Clark) has been dealing with the situation and can advise this is a wide spread issue affecting many thousands of networks and is a repeat of the INDOSAT issue seen in 2011. Since then many providers now filter results from INDOSAT and the RIPE RPKI validation being in use, this has limited the acceptance of our hijacked ranges. An email has also gone off to the INDOSAT contact who manages the network. We are awaiting a response.

At the moment no UK provider has accepted the wrong announcement and we are not seeing any UK related issues, however as many thousands of other networks are caught up in this you many experience problems connecting to the effected networks.

UPDATE 02 – 21:15 – 02/04/2014

We can now advise the only upstream provider of INDOSAT who accepted the announcement was AS4651 (THAI-GATEWAY The Communications Authority of Thailand(CAT),TH) Our upstream providers directly peer with AS4651 and therefore filtered the incorrect data.
As no further repeat alerts have been seen we are happy this has been resolved by INDOSAT although no official response has been given by them.
Should a repeat occur then this NOC notice will be reopened.