Our network monitor has alerted us to a failure of a link within our Entanet port channel that we use for Transit to and from Entanet at Goswell Road(LON01). Existing links have taken over the traffic loads and no service impact has been seen. Diagnostic work is underway to bring the link back in to service and at the moment is suspected to be a fiber break.
UPDATE 01 – 15:30
Further work with Entanet shows there is no light on there side of the link. No work has taken place in the Entanet cabinet or in our suite. Other links on the same bulk are operating without the same failure so this may be a faulty transceiver
UPDATE 02 – 15:45
Further diagnostics show that despite our transceiver showing the link as “Not Connected” in depth diagnostics show the transceiver with a possible fault as its not being reported within the hardware inventory for that line card. Real-time logs don’t show the device as being taken out of service by the IOS and we believe a line card reset may resolve the issue. However this transceiver is currently connected to the Active supervisor for that core. The device does have 2 for redundancy and the load will need to be transferred before a reload can take place. A new emergency maintenance window will now be opened as this may result in several seconds of switching failure.
UPDATE 03 – 20:05
The module has been restarted, however the problem remains. We believe the output given from the Cisco IOS is misleading as it fails to list the device rather than giving DB levels of 0. We have resumed the fiber break theory and have asked our vendor Level3 to confirm if this is the case.
UPDATE 04 – 04/03/2015 – 09:00
A Structured engineer is being sent to site to locate and test the fiber.
UPDATE 05 – 06/03/2015 – 10:00
Engineers discovered this cross connect had been incorrectly unplugged in a 3rd party cabinet. This won’t be an issue going forward as we have our own bulk fibre and all critical circuits will be moved to this.