Entanet BGP Peering – Planned outage – 26/11/2013 – 00:01 till 06:00 *COMPLETE*

We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware. In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically … Continue reading “Entanet BGP Peering – Planned outage – 26/11/2013 – 00:01 till 06:00 *COMPLETE*”

We have been advised that one of our upstream providers Entanet are relocating there POP within Goswell Road at the request of Level3. This work will involve the re-routing of our fibre cables along with there hardware.

In order to keep disruption to a minimum and to ensure our private traffic to Enta is automatically re-routed via our other upstream providers, we will be taking our BGP / transit down several hours before the works are due to take place.

No long term service impact is anticipated, however people routing to us via our Enta transit will see between 30-60 second of routing instability while routes re-converge via BGP.

Peering shutdown time: 22:00 (25/11/2013)
Peering start-up time: 08:00 (26/11/2013)

UPDATE01 @ 19:10 – 25/11/2013

We have been advised that there is a problem with the newly run fibre. Due to this we have suspended the above planned works and will re-schedule.

UPDATE02 @ 08:00 – 26/11/2013

Our Upstream provider Entanet still relocated one of their POPs as planned. We will migrate of the existing core once the fibre fault has been resolved.

UPDATE03 @ 09:09 – 26/11/2013

I have just been advised that Level3 re-ran our Fibre cross-connects late last night following our escalation and both tested OK. Our upstream provider continued with the move on our behalf and we can advise that our transit links have now been migrated to the new core. During this move we did lose our BGP session as it was not taken down as originally planned, However other upstream providers took the traffic as our network is designed to with minimum downtime.

02 Wholesale LLU – Outage – 13:00 – 21/10/2013

We are aware of a network wide issue affecting DSL sync and PPP sessions on some 02 wholesale LLU ADSL2+ circuits. Engineers are working to locate the fault. We will provide further updates once they are provided to us. UPDATE 01 – 14:23 Engineers are still working within wholesale to find the root cause of … Continue reading “02 Wholesale LLU – Outage – 13:00 – 21/10/2013”

We are aware of a network wide issue affecting DSL sync and PPP sessions on some 02 wholesale LLU ADSL2+ circuits.
Engineers are working to locate the fault.
We will provide further updates once they are provided to us.

UPDATE 01 – 14:23
Engineers are still working within wholesale to find the root cause of the fault. Connectivity has been intermittently restored.

UPDATE 01 – 17:10
We’ve now been advised that the supplier has identified a fibre fault at Telehouse and are now attempting to get it resolved as soon as possible. We’ll continue to update as the updates are available.

EasyIPT – Carrier fault – 15:02 – 22/10/2013

We are aware of an issue with one of our carriers where SIP registrations are being refused resulting in a loss of service. We feel this is a repeat of a recent issue experienced with this Carrier and are in discussions for clarification. We will provide updates as they become available. UPDATE01 @ 15:19 We … Continue reading “EasyIPT – Carrier fault – 15:02 – 22/10/2013”

We are aware of an issue with one of our carriers where SIP registrations are being refused resulting in a loss of service. We feel this is a repeat of a recent issue experienced with this Carrier and are in discussions for clarification.

We will provide updates as they become available.

UPDATE01 @ 15:19

We are seeing sessions starting to reconnect and service is being restored. We apologise for any inconvenience and will be taking this up with the Carrier and obtaining an outage report which will be provided once available

EasyHTTP – Loss of service – *RESOLVED*

Between 22:07 22:55 we saw a loss of connectivity to EasyHTTP. An MOR will be available shortly. UPDATE 01 @ 23:44 MOR is now avaliable. Major Incident Report – EasyHTTP Outage 08-10-13

Between 22:07 22:55 we saw a loss of connectivity to EasyHTTP. An MOR will be available shortly.

UPDATE 01 @ 23:44
MOR is now avaliable. Major Incident Report – EasyHTTP Outage 08-10-13

EasyHTTP (SMTP) – Outbound delivery delay *RESOLVED*

Between 19:20 & 20:45 we saw a DNS failure on ns01.r00.lon01.structuredcommunications.co.uk and ns02.r00.lon01.structuredcommunications.co.uk. This DNS failure lead to the failure of the SMTP service on EasyHTTP in that is was unable to lookup domains for email delivery. Emails where queued on the outbound SMTP connector and delivered once the issue was resolved at 20:41. Looking … Continue reading “EasyHTTP (SMTP) – Outbound delivery delay *RESOLVED*”

Between 19:20 & 20:45 we saw a DNS failure on ns01.r00.lon01.structuredcommunications.co.uk and ns02.r00.lon01.structuredcommunications.co.uk. This DNS failure lead to the failure of the SMTP service on EasyHTTP in that is was unable to lookup domains for email delivery. Emails where queued on the outbound SMTP connector and delivered once the issue was resolved at 20:41.

Looking at the logs prior to the above, its possible there may have been small period where this was intermittant resulting in Delivery delays.

The 2 listed name servers remain offline and DNS traffic has been diverted to ns01.r01.lon01.structuredcommunications.co.uk & ns02.r01.lon01.structuredcommunications.co.uk

We will advise when both the servers located in R00 are back in service.

NOC: Planned Maintenance – 25/09/2013 – 21:00 till 22:00 – LON01 (Hosted EasyIPT) *COMPLETE*

We will be reviewing our out-of-band management on this service and associated servers. If adjustments are required then we will be conducting a controlled reboot in order to adjust the iLO BIOS on each device. An outage is expected during this window as handsets will be unable to register to the platform and will result … Continue reading “NOC: Planned Maintenance – 25/09/2013 – 21:00 till 22:00 – LON01 (Hosted EasyIPT) *COMPLETE*”

We will be reviewing our out-of-band management on this service and associated
servers. If adjustments are required then we will be conducting a controlled
reboot in order to adjust the iLO BIOS on each device. An outage is expected during this window as handsets will be unable to register to the platform and will result in a timeout.

This outage will due to its nature and the individual settings of each handset result in a possible stuck NAT session with a customers router, Where a customer has all our required managed hardware and where Structured Communications have a current and valid maintenance contract with the site, we will reboot the required hardware remotely. Where this is not the case, customers are requested to power-cycle there hardware if they find there handsets have not re-registered following this work.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 22:00 – 07/10/2013

This work is now complete

NOC: Planned Maintenance – 25/09/2013 – 21:00 till 22:00 – LON01 (EasyHTTP Service) *COMPLETE*

We will be reviewing our out-of-band management on this service and associated servers. If adjustments are required then we will be conducting a controlled reboot in order to adjust the iLO BIOS on each device. Several small outages may be seen for HTTP traffic, FTP traffic and Email flow (both inbound and outbound). Email will … Continue reading “NOC: Planned Maintenance – 25/09/2013 – 21:00 till 22:00 – LON01 (EasyHTTP Service) *COMPLETE*”

We will be reviewing our out-of-band management on this service and associated
servers. If adjustments are required then we will be conducting a controlled
reboot in order to adjust the iLO BIOS on each device. Several small outages may
be seen for HTTP traffic, FTP traffic and Email flow (both inbound and outbound). Email will not be lost durning this window as it should be queued on the originating server.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 22:00 – 07/10/2013

This work is now complete

NOC: Planned Maintenance – 25/09/2013 – 20:00 till 21:00 – LON01 (Easybond Service) *COMPLETE*

We will be reviewing our out-of-band management on this service and associated servers. If adjustments are required then we will be conducting a controlled reboot in order to adjust the iLO BIOS on each device. Several small outages may be seen while connections automatically fail-over to redundant hardware and ultimately back to the primary hardware. … Continue reading “NOC: Planned Maintenance – 25/09/2013 – 20:00 till 21:00 – LON01 (Easybond Service) *COMPLETE*”

We will be reviewing our out-of-band management on this service and associated servers. If adjustments are required then we will be conducting a controlled reboot in order to adjust the iLO BIOS on each device. Several small outages may be seen while connections automatically fail-over to redundant hardware and ultimately back to the primary hardware.

UPDATE @ 18:46 – 25/09/2013

This was has been postponed

UPDATE @ 09:30 – 30/09/2013

This work has been rescheduled for Monday the 07/10/2013 (same time window)

UPDATE @ 21:00 – 07/10/2013

This work is now complete

Hosted EasyIPT

We are aware of an outage on our Hosted EasyIPT services. Engineers are currently assigned remotely and we have dispatched engineers to the data centre. Update01**Final Resolved** @ 21:37 Engineers attended LON01 and arrived on site at 17:01 where the server was power cycled restoring service at 17:05. Handsets then started to re-register with individual … Continue reading “Hosted EasyIPT”

We are aware of an outage on our Hosted EasyIPT services. Engineers are currently assigned remotely and we have dispatched engineers to the data centre.

Update01**Final Resolved** @ 21:37

Engineers attended LON01 and arrived on site at 17:01 where the server was power cycled restoring service at 17:05. Handsets then started to re-register with individual sites coming back online.

The service has remained stable.

An outage report will be made available shortly.

LON01 IPv6 Support

Providers continue to deplete there IPv4 allocations and Structured Communications are no different. In preparation for our own depletion we will soon be enabling IPv6 support with our upstream providers. Further information will follow shortly. **UPDATE 01 @ 01:00 – 08/09/2013** We have experienced a delay in getting our current BGP sessions IPv6 enabled with … Continue reading “LON01 IPv6 Support”

Providers continue to deplete there IPv4 allocations and Structured Communications are no different. In preparation for our own depletion we will soon be enabling IPv6 support with our upstream providers. Further information will follow shortly.

**UPDATE 01 @ 01:00 – 08/09/2013**
We have experienced a delay in getting our current BGP sessions IPv6 enabled with our primary upstream peers. For this reason we have entered in to a peering agreement with Hurricane Electric to tunnel our IPv6 subnet over IPv4 for testing. We are announcing this range via our ASN (199283) to Hurricane Electric ASN (6939). The range is visible globally.

**UPDATE 02 @ 22:07 – 13/10/2013**
We are pleased to advise all our existing IPV4 BGP peering has been reconfigured to include IPV6 BGP sessions. At this moment in time we are announcing our entire /29 subnet (2a04:2140::/29) while we continue to test. We have several IPv6 DualStack servers operating within our range without incident and will be looking to fully enable our network for IPV6 in the next coming months. IPv6 will be available to customers once this is complete and will start off with co-location packages.