LON01 – 27/02/2014 – 16:10 – CORE01 – OUTAGE *RESOLVED*

During an internal training exercise a “reload” command was issued to CORE01-LON instead of CORE01-HOR which resulted in a reboot of both the primary and secondary supervisor modules. This reload caused a 5 minute loss of service to LON01 for all directly connected services and customers.

Safe guards will be put in place on LON01 to prevent this happening again in such a scenario.

We apologise for any inconvenience caused.

Inbound / Outbound call Issue London Fibre Break 24/02/2014

We are currently aware of fibre break in the London area which is affecting outbound calls for some of our users. We have been informed by suppliers that this is due do fibre damage to which is currently being repaired. We are expecting an update by midday.

Calls from customers outside of London to customers in London may fail. Broadband speeds should NOT be affected. This incident will be impacting BT customers and other operators.

In the mean time if you are experiencing issues please prefix the dialled number with 1280 for example 128002033014000. If you use a telephone system and normally prefix calls with 9 an example would be 9128002033014000.

We apologise for any inconvenience caused.

EasyxDSL Cannot Conenct Issues 20/02/2014 – 10:00

We are currently experiencing an issue that is preventing some DSL users connecting to our service. Our initial investigations indicate that this is a BT issue and we are working with them to resolve as soon as possible. The issue spans multiple nodes and is not geographically restrained to one area. We are aiming to have the service restored as a matter of urgency.

-Update- 11:40
Customer sessions are now being restored and our monitoring indicates that the majority of services are now working again. We’ve yet to receive details from BT as to the cause but will update when we have further information.

LON01 Data Centre – 17/02/2014 – 22:00 – Planned Power Maintenance (B Feed)

Following on from the power disruption back in March 2013 that affected our London Leve3 suite. We have been advised that our supplier is upgrading our suite to a Tier 4 specification

This is a welcome upgrade and means that both our A&B power feeds will be fed from independent UPS rooms and independent generator banks.

Our core network and customer facing services (EasyHTTP, EasyIPT & EasyBOND) are fully power redundant and no service disruption is expected, however they will be classed as “at risk” for the 24 hours while the work is undertaken starting at 22:00 hours on the 17th Feb until 22:00 hours on the 18th Feb.

We will be attending site for the duration and taking the B feed out of service prior to the works in a controlled manor. Any customer services with only a single fed supply currently on B will be transferred to A prior to the work and returned once complete.

UPDATE 01 – 20:30 – 17/02/2014
Engineers have been onsite all day and prep work is now underway. Traffic is going to be transfered from AG-Primary to AG-Backup, this will cause a 30 second outage while devices transfer.

UPDATE 02 – 22:00 – 17/02/2014
An unexpected outage was seen for EasyBond circuits where an old version of the NOC came online due to ESXi01 not saving it’s startup configuration on shutdown. This has resulted in some stale sessions. Please power down your hardware for 20 minutes where power routers are in PPPOE mode or STATIC IP.