We are aware ns02.r01.lon.adapt.easybond.co.uk is not processing DNS lookup requests. We are currently looking in to the issue and will provide an update shortly. NS01 (ns01.r01.lon.adapt.easybond.co.uk) is operating normally so this should not be service effecting. UPDATE 01 – 21:45 This issue has been resolved by re-loading the IPTABLE rules.
We are aware ns02.r01.lon.adapt.easybond.co.uk is not processing DNS lookup requests. We are currently looking in to the issue and will provide an update shortly. NS01 (ns01.r01.lon.adapt.easybond.co.uk) is operating normally so this should not be service effecting.
UPDATE 01 – 21:45
This issue has been resolved by re-loading the IPTABLE rules.
We are aware of a wholesale issue currently affecting our xDSL customer base, manifesting itself as poor performance. Our Systems team are currently investigating. Further information will be provided as and when it becomes available. UPDATE01 – 17:38 – 17/01/2014 We have been advised that the cause of the issue has been a “flapping” port … Continue reading “xDSL connectivity – 16:30 – 17/01/2014 *RESOLVED*”
We are aware of a wholesale issue currently affecting our xDSL customer base, manifesting itself as poor performance.
Our Systems team are currently investigating. Further information will be provided as and when it becomes available.
UPDATE01 – 17:38 – 17/01/2014
We have been advised that the cause of the issue has been a “flapping” port between BT wholesale and our wholesale provider causing a loss in bandwidth. This has now been resolved however will have caused stale PPP sessions to occur. If you don’t have service can you please reboot all your hardware.
We apologise for any inconvenience caused.
UPDATE02 – 12:09 – 21/01/2014
An incident report has been made available for this issue. Major Incident Report – Enta Outage 17-01-2014
We are aware of a network wide issue affecting DSL sync and PPP sessions on some 02 wholesale LLU ADSL2+ circuits. Engineers are working to locate the fault. We will provide further updates once they are provided to us. We apologise for any inconvenience.
We are aware of a network wide issue affecting DSL sync and PPP sessions on some 02 wholesale LLU ADSL2+ circuits.
Engineers are working to locate the fault.
We will provide further updates once they are provided to us.
We apologise for any inconvenience.
The 02 Wholesale network which is used to provide some of our ADSL services is being switched off on the 27th February 2014 following last year’s acquisition of the 02 and BE* retail customer base by Sky. This news is unwelcome within the industry and requires a change in service we provide to you if … Continue reading “Withdrawal of 02/BE* Wholesale LLU DSL services”
The 02 Wholesale network which is used to provide some of our ADSL services is being switched off on the 27th February 2014 following last year’s acquisition of the 02 and BE* retail customer base by Sky.
This news is unwelcome within the industry and requires a change in service we provide to you if you are on an LLU package provided by Structured Communications. BT Wholesale has advised they are accepting migrations from the 02 Wholesale network in light of the recent industry news, This will automatically be arranged by Structured Communications for effected circuits prior to the 27th February 2014.
The current pricing structure and packages will be migrated like for like as will login credentials and IP address settings.
We are starting migrations on the 14th of January 2014 whereby you will experience a brief loss of service while the physical exchange work is completed. (30 minutes average maximum) We hope to have all affected circuits migrated by the 1st February 2014
If you have any questions, please don’t hesitate to contact us on 0203 301 4000
Following on from our planned maintenance works on EasyHTTP we removed the SSL certificate securing the control panel. We have been unable to reinstall this. We have a backup for both IIS and Plesk. We have logged an issue with our software vendor to see if these can be re-installed. If not then we will … Continue reading “NOC: SSL Certificate – LON01 (EasyHTTP Service) *Complete*”
Following on from our planned maintenance works on EasyHTTP we removed the SSL certificate securing the control panel.
We have been unable to reinstall this. We have a backup for both IIS and Plesk. We have logged an issue with our software vendor to see if these can be re-installed. If not then we will simply install a new one.
UPDATE01 – 15:15 – 24/12/2013
A new SSL certificate has been ordered and will be installed shortly.
UPDATE02 – 16:00 – 24/12/2013
The new SSL certificate has just been issed by AlphaSSL and installed on the system. This Notice is closed
We have been advised of a fibre failure within the 02 Wholesale network. A case has been logged with our wholesale provider. We apologise for any inconvenience caused
We have been advised of a fibre failure within the 02 Wholesale network. A case has been logged with our wholesale provider.
We apologise for any inconvenience caused
In line with our current service level agreements and previously advised (where applicable) we are pleased to announce we will be upgrading our current webhosting service (EasyHTTP) to our new hardware platform (HP DL580) very shortly. Our planned migration date for this is the 20th December 2013, however this will likely change depending on the … Continue reading “NOC: Planned Maintenance – 20/12/2013 20:00 till 23/12/2013 23:59 – LON01 (EasyHTTP Service) *COMPLETE*”
In line with our current service level agreements and previously advised (where applicable) we are pleased to announce we will be upgrading our current webhosting service (EasyHTTP) to our new hardware platform (HP DL580) very shortly. Our planned migration date for this is the 20th December 2013, however this will likely change depending on the migration method used and should be considered as TBC (to be confirmed)
There are no planned changes to IP addressing at this moment in time, and is the preferred migration method. However this will result in some downtime in order to ensure no new data is upload to the old server during the migration window.
Inbound and outbound SMTP will be refused at times during this window, unless you are using our backup MX service. In which case inbound mail will be held and delivered once the new server is online.
The new platform consists of:
Quad power supplies 6000 Watts (one per CPU)
4 x Intel quad core 2.93 GHz processers
32GB of RAM
8 x 500GB SAS drives in a RAID 5 configuration (3.2TB of useable space)
Dual 1Gbps uplinks
The upgrade is a very welcome project and more details will follow as the migration plan is finalised.
UPDATE01 – 17:04 – 09/12/2013
During an audit of existing scripting languages we have discovered PHP 4.4.9 is still installed. As this release has gone EOL (End of Life) it will not be installed on to the new platform. All existing sites using 4.4.9 will be re-graded to 5.2.17
We will also be upgrading MS SQL to 2012 which will result in a Instance Name change from “default” to “MSSQLSERVER2012” (excluding the “”) PLease ensure you update any connection strings.
UPDATE02 – 21:30 – 19/12/2013
We have started a FULL manual backup of the platform which will be used as the base / current image on the new array. Should the restore go as planned and once we are happy then we will manually copy over any new email received on the old system between now and midnight tomorrow. Once this is complete we will be taking the old system offline. Any new website data will not be migrated due to the size and time it would take resulting is a further loss of data. We would advise keeping any new uploads / changes to existing website data to a minimum.
We will update this once the backup and restore has completed advising of the next steps.
UPDATE03 – 23:00 – 20/12/2013
Migration is running behind due to an issue that has occurred with the use of the same IP addressing and the way Plesk maps this to the physical network card when not physically enabled. A new VLAN was installed to avoid this however we have had to process the backup / restore another way due to the extreme size of the backup and processing limitations on various protocol.
We are migrating data to a temporary IP range that we will revert once complete.
UPDATE04 – 01:15 – 21/12/2013
Migration work is still on-going and back on track, However due to the additional changes required we have extended this maintenance window for a further 24 hours.
This window will now close Sunday 6am.
UPDATE05 – 15:15 – 21/12/2013
The migration work completed last night however several warnings where encountered as MySQL stopped accepting connections from “localhost” and prevented a small number of domains having their mail boxes created. This issue has been resolved and we are now re-restoring the effected domains.
UPDATE06 – 18:15 – 21/12/2013
We have finished restoring the effected domains and the primary migration is now complete. We have now started the test migration of any email received since the initial copy. We have again extended this maintenance window for a further 24 hours.
This window will now close Monday 6am.
UPDATE07 – 19:00 – 22/12/2013
We have now started the final migration of any email received since the initial copy. Concerns over the size means we will be unable to copy any further email received on the old system from this time (19:00) We advise everyone to make a local copy of email received from this point to their local mail platform which can be copied back after
UPDATE08 – 23:30 – 22/12/2013
We have encountered an error that is preventing us from changing the primary default ip address. The error we are getting is in relation to an IIS module, however we suspect this is a corruption in an MS sql database.
We have been forced to log this with the software vendor as an escalation. As we have no influence on timing we are unlikely to have this resolved within our maintenance window and have therefore suspended this change. We are hopeful however this should be resolved by midday tomorrow so have extended this maintenance window until 23:00 tomorrow (Monday 23rd) This will allow us to migrate email data (again) however we are unable to restore any new website content due to time implications.
UPDATE09 – 12:15 – 23/12/2013
We have been advised by our software vendor that they have resolved the issue effecting the reconfiguration of IP addressing and the issue was related to corruption in the Master Plesk database and missing SSL certificate data following on from the migration. As the migration was completed via the built in migration tool we have raised a BUG with the vendor and ask them to investigate, however this may be related to the way we were forced to complete the migration.
The changes have now been made and the server is ready to come online and take over following the migration of any new email. As the new server now has the same IP addressing as the old system we are unable to complete a direct copy and will be required to copy the data to 3rd party storage device within our network. This additional step will require more time to complete the transfer
We therefore have confirmed the following times:
Copy email: 21:00
Down old server: 22:00
Up new server: 23:30
This will result in a planned 1 hour outage.
If any problems are experanced with the server then we will revert.
UPDATE10 – 20:15 – 23/12/2013
The work above has started. Times have been amended due to delays with weather and engineering.
UPDATE11 – 22:50 – 23/12/2013
We have completed the copy of any new email delivered to the old platform since 21:00 and are currently downloading this to the new system. The additional migration to an external storage source has added delay as expected. We have added an extra hour to the maintenance window.
UPDATE12 – 23:59 – 23/12/2013
We have completed the migration of email and the VLANS have been swapped on our core. This upgrade is now complete
We will be conducting an upgrade of our bonded platform on the above date to Patch 9 Version 2.32 The update will cause a brief interruption to service UPDATE01 *FINAL* @ 21:25 – 29/11/2013 This work is now complete.
We will be conducting an upgrade of our bonded platform on the above date to Patch 9 Version 2.32
The update will cause a brief interruption to service
UPDATE01 *FINAL* @ 21:25 – 29/11/2013
This work is now complete.
At 20:17 we were made aware by our networking monitoring software that both our /21 ranges where being announced by our previous transit provider (AS24867) and not ourselves (AS199283) We are current working to get this resolved. UPDATE01 – 21:22 We have discovered the announcement was detected in the Czech Republic learnt by 6881 15685 … Continue reading “BGP route hi-jack – 20:35 – 18/11/2013 *RESOLVED*”
At 20:17 we were made aware by our networking monitoring software that both our /21 ranges where being announced by our previous transit provider (AS24867) and not ourselves (AS199283)
We are current working to get this resolved.
UPDATE01 – 21:22
We have discovered the announcement was detected in the Czech Republic learnt by 6881 15685 6939 24867. Engineers are still working on the root cause of the rouge announcement, however this incident is now closed. A report will be made available as and when a full investigation has taken place.
We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware. In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically … Continue reading “Entanet BGP Peering – Planned outage – 26/11/2013 – 00:01 till 06:00 *COMPLETE*”
We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware.
In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically re-routed via our other upstream providers, we will be taking our BGP / transit down several hours before the works are due to take place.
No long term service impact is anticipated, however people routing to us via our Enta transit will see between 30-60 second of routing instability while routes re-converge via BGP.
Peering shutdown time: 22:00 (25/11/2013)
Peering start-up time: 08:00 (26/11/2013)
UPDATE01 @ 19:10 – 25/11/2013
We have been advised that there is a problem with the newly run fibre. Due to this we have suspended the above planned works and will re-schedule.
UPDATE02 @ 08:00 – 26/11/2013
Our Upstream provider Entanet still relocated one of their POPs as planned. We will migrate of the existing core once the fibre fault has been resolved.
UPDATE03 @ 09:09 – 26/11/2013
I have just been advised that Level3 re-ran our Fibre cross-connects late last night following our escalation and both tested OK. Our upstream provider continued with the move on our behalf and we can advise that our transit links have now been migrated to the new core. During this move we did lose our BGP session as it was not taken down as originally planned, However other upstream providers took the traffic as our network is designed to with minimum downtime.