February 2016 – Structured Communications NOC

LON01 – EasyIPT – 29/02/2016 – 21:30 till 22:00 Emergency Maintenance Complete

Following on from reports today of SIP channel limits being reached for both inbound and outbound calls affecting some of our managed PBX systems, work has been undertaken on our core softswitch to try and identify the root cause of this issue. Changes have been made to the platform however a emergency reboot is required. … Continue reading “LON01 – EasyIPT – 29/02/2016 – 21:30 till 22:00 *Emergency Maintenance* *Complete*”

Changes have been made to the platform however a emergency reboot is required. Due to the size of the platform this reboot will take up-to 15 minutes to complete. During this time inbound and outbound calls will be limited.

We will advise once service is restored and calls are routing again.

Apologies for any inconvenience this may cause.

UPDATE01 – 21:40 softswitch reboot is complete, we are monitoring traffic flow

UPDATE02 – 22:55 A random check on managed PBX systems are showing registration, however this is not reflected on our softswitch, traffic is flowing however.

UPDATE03 – 22:10 Systems continue to show inaccurate data about the status of some registrations. We suspect stuck SIP sessions on some managed PBXs as this is affecting a wide range of PBX software versions.

UPDATE04 – 21:40 All affected managed PBXs are showing as on-line correctly.

LON01 – 26/02/2016 – 15:40 – CORE01.LON01 INCIDENT RESOLVED

We are aware CORE01.LON01 within our network has suffered a software reload affecting all services directly connected to it. Other parts of the network are unaffected. UPDATE01 – 15:45 CORE01.LON01 has now recovered and we are reviewing the crash logs for the root cause. We apologize for any inconvenience caused. UPDATE02 – 16:15 A incident … Continue reading “LON01 – 26/02/2016 – 15:40 – CORE01.LON01 *INCIDENT* *RESOLVED*”

We are aware CORE01.LON01 within our network has suffered a software reload affecting all services directly connected to it. Other parts of the network are unaffected.

UPDATE01 – 15:45
CORE01.LON01 has now recovered and we are reviewing the crash logs for the root cause. We apologize for any inconvenience caused.

UPDATE02 – 16:15
A incident report is now available HERE

LON01 – 22/02/2016 – IPv6 Changes

We have been running IPv6 on our core network for over 2 years now, and during this time offering test IPv6 subnets to Leased Line customers as well as enabling some of our back end platforms. Over the next 4 weeks we will be re-configuring IPv6 on our core network to enable this connectivity on … Continue reading “LON01 – 22/02/2016 – IPv6 Changes”

We have been running IPv6 on our core network for over 2 years now, and during this time offering test IPv6 subnets to Leased Line customers as well as enabling some of our back end platforms.

Over the next 4 weeks we will be re-configuring IPv6 on our core network to enable this connectivity on the various DSL infrastructure for testing purposes.

No service interruption is expected to be seen for IPv4 based connectivity, however IPv6 users may see some degraded service while various parts of the core are reconfigured. Any service affecting works are expected to take place outside of hours.

UPDATE01 21:00 – 23/02/2016
We have reconfigured our core to the new permanent (we hope) IPv6 addressing scheme. BGP sessions have been reconfigured and are back on-line. The next Phase is to reconfigure our Ethernet services.

UPDATE02 22:30 – 05/03/2016
Both LNS01 and LNS02 now have IPv6 built and enabled on there respective interfaces. We plan to build and enable the IPv6 BGP sessions back to our core shortly. This will enable us to enable test accounts with a IPv6 prefix.

UPDATE03 23:30 – 05/03/2016
IPv6 BGP is now enabled on LNS01 & LNS02 between our core. IPv6 prefixes are now available for testing on our DSL network.

UPDATE04 01/04/2016
standard Ethernet services are now also fully enabled for IPv6 and we welcome test clients.
:
UPDATE05 22/04/2016 – 21:19
ALL Ethernet services (services including a backup media) are now also fully enabled for IPv6. Fail over has been tested without issue. We also added some IPv4 config tweaks to these circuits. These again have been tested without fault, however clients may have been a “blip” or two while we checked these config changes worked as expected. We apologise for any inconvenience caused.

Virgin Media Based Circuits – 16/02/2016 – Incident Resolved

We are currently aware of an issue affecting a number of on-net Virgin Media based circuits. We are currently working with our supplier to investigate the issue further. Services with backup media such as DSL will have re-routed. UPDATE01 – 13:28 We have been advised that Virgin have identified this as an NTU failure impacting … Continue reading “Virgin Media Based Circuits – 16/02/2016 – Incident *Resolved*”

We are currently aware of an issue affecting a number of on-net Virgin Media based circuits. We are currently working with our supplier to investigate the issue further.

Services with backup media such as DSL will have re-routed.

UPDATE01 – 13:28
We have been advised that Virgin have identified this as an NTU failure impacting approximately 200 circuits. Work is being undertaken to restore the service.

UPDATE02 – 13:37
Virgin media are and dispatching engineerins to the point of presence at InterXion

UPDATE03 – 15:40
Virgin engineers are continuing to investigate the reported fault to isolate the issue further.

UPDATE04 – 16:27
We have seen services restore, they will be considered at risk until Virgin has closed off the fault with there full findings.

UPDATE05 – 17:11
Virgin has confirmed service has been restored via a temporary work around to bypass a faulty fibre patch at the Poplar Metnet. Virgin Media will schedule future planned maintenance to fix the original fault. Services should no longer be considered at risk however we will update further once the planned maintenance to complete the repair has been scheduled.

DNS Servers – Advisory – 16/02/2016 – 10:30

Following on from yesterdays issue with Google DNS reachability. We are advising all our customers who take our connectivity solutions to ensure they are using our DNS servers. Primary: 95.87.104.23 Secondary: 95.87.104.25 Clients with IPv6 are advised to contact us for further details. Unfortunately issues with 3rd party DNS servers are not covered by our … Continue reading “DNS Servers – Advisory – 16/02/2016 – 10:30”

Following on from yesterdays issue with Google DNS reachability. We are advising all our customers who take our connectivity solutions to ensure they are using our DNS servers.

Primary: 95.87.104.23
Secondary: 95.87.104.25

Clients with IPv6 are advised to contact us for further details.

Unfortunately issues with 3rd party DNS servers are not covered by our network SLA.

LON01 – 09/02/2016 – 20:00 till 20:30 – EasyXDSL – RADIUS Maintenance COMPLETE

During the above maintenance window, we will be updating our RADIUS servers to run a revised version of code. This code will fix an issue with new PPP sessions not being rate limited on connection whereby they have already exceeded there usage limit. The RADIUS process on each server will be shutdown while the new … Continue reading “LON01 – 09/02/2016 – 20:00 till 20:30 – EasyXDSL – RADIUS Maintenance *COMPLETE*”

The RADIUS process on each server will be shutdown while the new code is applied and then re-started. This will be done 1 server at a time. Existing connections will remain online, however some accounting data maybe lost. New connections may be delayed in PPP while they are re-routed to an active RADIUS.

UPDATE01 – 22:45 This work started later than planned however is now complete. All RADIUS servers are running the updated code and have tested OK.

LON01 – 17/02/2016 – 23:00 till 03:00 – EasyXDSL – Maintenance COMPLETE

We have been advised by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 on the 17/02/2016 from 23:00 – 03:00. This upgrade work will affect one of our fibre waves we use for DSL termination back in … Continue reading “LON01 – 17/02/2016 – 23:00 till 03:00 – EasyXDSL – Maintenance *COMPLETE*”

We have been advised by one of our interconnect providers that that they will be conducting a software upgrade on there core Junos router at Telehouse North between 23:00 and 03:00 on the 17/02/2016 from 23:00 – 03:00. This upgrade work will affect one of our fibre waves we use for DSL termination back in to our network at Goswell Road. DSL traffic traversing over this link will drop during the maintenance window, however should re-establish over an alternative link via Telehouse East which is unaffected.

The nature of these works include a reload of the Junos hardware at Telehouse North by our wave supplier and in theory should be the only disruption we see. Unfortunately sort disruptions like this will cause multiple DSL PPP session re-terminations as they flip back and forth between Telehouse North and Telehouse East as our network is designed to use the former where possible. Due this concern and avoiding any other unforeseen issues that will cause session drops during the maintenance windows, we will be manually shutting down the Telehouse North link at 22:00 on the night with the view of bringing it backup at 07:00 the following morning. This will limit the session drops to these times and provide a stable service throughout.

UPDATE01 – 17/02/2016 – 22:00
Traffic has been re-routed

UPDATE02 – 18/02/2016 – 09:05
Traffic has been re-routed

BT Outage – DSL Authentication Issue – 02/02/2016 – 16:25

We are aware of the national BT DSL outage that is sweeping social media. In addition to this we are now starting to see wholesale DSL services affected, whereby if a connection looses sync or PPP then it wont recover. Further diagnostics shows that BT are not delivering the session to us correctly and is … Continue reading “BT Outage – DSL Authentication Issue – 02/02/2016 – 16:25”