LON01 – Entanet Transit – 03-03-2015* *COMPLETE*

Our network monitor has alerted us to a failure of a link within our Entanet port channel that we use for Transit to and from Entanet at Goswell Road(LON01). Existing links have taken over the traffic loads and no service impact has been seen. Diagnostic work is underway to bring the link back in to … Continue reading “LON01 – Entanet Transit – 03-03-2015* *COMPLETE*”

Our network monitor has alerted us to a failure of a link within our Entanet port channel that we use for Transit to and from Entanet at Goswell Road(LON01). Existing links have taken over the traffic loads and no service impact has been seen. Diagnostic work is underway to bring the link back in to service and at the moment is suspected to be a fiber break.

UPDATE 01 – 15:30
Further work with Entanet shows there is no light on there side of the link. No work has taken place in the Entanet cabinet or in our suite. Other links on the same bulk are operating without the same failure so this may be a faulty transceiver

UPDATE 02 – 15:45
Further diagnostics show that despite our transceiver showing the link as “Not Connected” in depth diagnostics show the transceiver with a possible fault as its not being reported within the hardware inventory for that line card. Real-time logs don’t show the device as being taken out of service by the IOS and we believe a line card reset may resolve the issue. However this transceiver is currently connected to the Active supervisor for that core. The device does have 2 for redundancy and the load will need to be transferred before a reload can take place. A new emergency maintenance window will now be opened as this may result in several seconds of switching failure.

UPDATE 03 – 20:05
The module has been restarted, however the problem remains. We believe the output given from the Cisco IOS is misleading as it fails to list the device rather than giving DB levels of 0. We have resumed the fiber break theory and have asked our vendor Level3 to confirm if this is the case.

UPDATE 04 – 04/03/2015 – 09:00
A Structured engineer is being sent to site to locate and test the fiber.

UPDATE 05 – 06/03/2015 – 10:00
Engineers discovered this cross connect had been incorrectly unplugged in a 3rd party cabinet. This won’t be an issue going forward as we have our own bulk fibre and all critical circuits will be moved to this.

LON01 – Rack00 *At Risk – Maintenance – 10-03-2015* *COMPLETE*

We have been advised that an engineer working on behalf of Level3 will be in our core rack Tuesday 10-03-2015 terminating our new 24 pair bulk fiber as part of our network upgrade (details to follow) No service impact is expected, however the engineer will be within a live rack environment upon fitment and testing.

We have been advised that an engineer working on behalf of Level3 will be in our core rack Tuesday 10-03-2015 terminating our new 24 pair bulk fiber as part of our network upgrade (details to follow) No service impact is expected, however the engineer will be within a live rack environment upon fitment and testing.

LON01 – Entanet Transit *At Risk Planned Maintenance 05-03-2015* *COMPLETE*

We have been advised by Entanet that they will be inserting an additional line card in to goswell.core.enta.net to increase capacity. They have advised that no downtime is expected, however that’s not always been the case and we have classed this work as “at risk” Any service disruption will result in traffic being automatically re-routed. … Continue reading “LON01 – Entanet Transit *At Risk Planned Maintenance 05-03-2015* *COMPLETE*”

We have been advised by Entanet that they will be inserting an additional line card in to goswell.core.enta.net to increase capacity. They have advised that no downtime is expected, however that’s not always been the case and we have classed this work as “at risk”

Any service disruption will result in traffic being automatically re-routed.

Date scheduled: Thursday 5th March 2015 00:01 – 03:00

LON01 – C4L Frontier Interconnect *At Risk Planned Maintenance* *COMPLETE*

We have been advised by one of our fiber providers (C4L) that they will be extending there coreTX network at Goswell Road. Works are **At Risk** as engineers will be present and working within a live network environment. C4L Notice: Location: Interxion, City Lifeline, Telehouse Metro, Goswell Road, Volta 36 Great Sutton St. Start: 2015-02-27, … Continue reading “LON01 – C4L Frontier Interconnect *At Risk Planned Maintenance* *COMPLETE*”

We have been advised by one of our fiber providers (C4L) that they will be extending there coreTX network at Goswell Road. Works are **At Risk** as engineers will be present and working within a live network environment.

C4L Notice:

Location: Interxion, City Lifeline, Telehouse Metro, Goswell Road, Volta 36 Great Sutton St.

Start: 2015-02-27, 20:00
End: 2015-02-28, 06:00

Reason: To install additional switching equipment within the coreTX network.

Impact: N/A. However, due to the nature of working within live network racks, services are at risk of layer 1 interruption.

—

Should we loose the interconnect then our network will automatically re-route traffic via an alternate carrier.

LON01 – EasyIPT – 26/01/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*

We have seen some inbound and outbound VoIP calls fail to connect between the hours of 13:30 – 14:30 today with an odd array of messages on the SIP proxy connecting to our softswitch. Initial diagnostics show nothing obvious, however to be proactive we will be looking deeper in to the platform this evening with … Continue reading “LON01 – EasyIPT – 26/01/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*”

We have seen some inbound and outbound VoIP calls fail to connect between the hours of 13:30 – 14:30 today with an odd array of messages on the SIP proxy connecting to our softswitch.

Initial diagnostics show nothing obvious, however to be proactive we will be looking deeper in to the platform this evening with the view of rebooting this to clear any stuck session data.

UPDATE01 This work has been rescheduled for the 26/01/2014

UPDATE02 This work has been rescheduled for the 27/01/2014

UPDATE03 20:35 This work has started.

UPDATE04 20:46 The server has requested a disk check to be run. Due to the size of the array this make take 15-20 minutes.

UPDATE05 20:52 The server has returned to operation but the SQL service is now not operating in the correct state following the disk check.

UPDATE06 20:58 The server has returned to operation and the SQL problem resolved. Test calls have been made and they completed successfully.

LON01 – ESXi10 PSU – 12:10 – 26/12/2014 *CLOSED*

Our network monitoring has alerted us to a possable PSU failure on ESXi10. This server has redundant supplys and remains online but in an “at risk” state. Engineers are due to investigate shortly with the aim of restoring the supply remotely if possible should this be confirmed as being in a warning state. Failed hardware … Continue reading “LON01 – ESXi10 PSU – 12:10 – 26/12/2014 *CLOSED*”

Our network monitoring has alerted us to a possable PSU failure
on ESXi10. This server has redundant supplys and remains online but in an “at risk” state.

Engineers are due to investigate shortly with the aim of restoring the supply remotely if possible should this be confirmed as being in a warning state.

Failed hardware will be replaced as soon as possable and the unit shutdown to protect adjacent devices.

UPDATE 01 – 13:00
An Engineer has remotely logged in to the OOB (out of bounds) management interface to get real-time hardware / status information from the server and has been able to confirm both PSUs are in an operational state.

We can only conclude ESXi is feeding our network monitoring incorrect data to our SNMP session.

Further investigation is required as to the cause of this, however we are happy the server is not at risk and closing this risk report

LON01 – **Christmas Network Freeze***

Please note that in order to protect the quality of your services over the busy Christmas period Structured Communications and several of our suppliers will be implementing network freezes. This means that between the dates listed below, no platform changes or other non-essential activity will be conducted. Essential emergency maintenance will continue if required and … Continue reading “LON01 – **Christmas Network Freeze***”

Please note that in order to protect the quality of your services over the busy Christmas period Structured Communications and several of our suppliers will be implementing network freezes. This means that between the dates listed below, no platform changes or other non-essential activity will be conducted. Essential emergency maintenance will continue if required and will be advised via NOC (http://noc.structuredcommunications.co.uk)

Structured From 17:30 on the 19th December 2014 until 09:00 on 5th January 2015

BT No formal freeze but staffing levels will be reduced over the Christmas period

Colt From 15th December to 4th January

Level 3 13th December to January 5th 2015

TalkTalk Business 15th December to January 2nd 2015

Virgin Media From 11th December for project/service launches, 18th December for business as usual changes to 5th January 2015 00:01

Vodafone 12th December 2014 until 5th January 2015

In all cases the period starts at 00:01 and ends at 23:59 unless otherwise stated.

PBX routing changes can be made up until the 23rd Dec 2014 (17:00) however we do request 24 hours notice prior to this cut off due to the increased support demand. Tickets submitted within this 24 hour window will be completed. Customers with their own access can of course make changes past this period.

Other PBX system configuration changes will not be made until the new year.

If you have any questions regarding the network freeze please contact us on 0203 301 4000

LON01 – 15/12/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*

1) We will be making some changes to our VPS cluster to bring online additional capacity. Unmanaged 3rd party VPS sessions may see a small amount of downtime while there images are moved. No other impact is expected. 2) An audit will be conducted on our existing bulk fibre between our suite on the 4th … Continue reading “LON01 – 15/12/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*”

1) We will be making some changes to our VPS cluster to bring online additional capacity. Unmanaged 3rd party VPS sessions may see a small amount of downtime while there images are moved. No other impact is expected.

2) An audit will be conducted on our existing bulk fibre between our suite on the 4th floor and our meet me rack on the 2nd. This is for upcoming network expansion. No outages are expected however the links should be considered “at risk” due to the physical nature of the works. Should an outage occur on a fibre link then our network will automatically re-route traffic via a alternative path.

UPDATE01 – 23:30 These works have been completed as expected, However the 3rd party ESXi images will not be moved until 22:00 Tuesday night (16/12/14)

EasyxDSL Cannot Conenct Issues 12/11/2014 – 09:00 *RESOLVED*

We are currently experiencing an issue that is preventing some DSL users connecting to our service and/or constantly reconnecting to our service. Our initial investigations indicate that this is a Entanet issue and we are working with them to resolve as soon as possible. We will update with more information as it becomes available. — … Continue reading “EasyxDSL Cannot Conenct Issues 12/11/2014 – 09:00 *RESOLVED*”

We are currently experiencing an issue that is preventing some DSL users connecting to our service and/or constantly reconnecting to our service. Our initial investigations indicate that this is a Entanet issue and we are working with them to resolve as soon as possible. We will update with more information as it becomes available.

— Update 09:17 —
Entanet have identified a hardware failure and advise engineers have been dispatched to replace the faulty hardware. We are still working for a ETA.

— Update – 09:21 —
Any DSL bonded circuits may see reduced capacity but service should remain stable as we use 2 wholesale ISPs with our own interconnects.

— Update – 13:21 —
The issue is still currently on going, Entanet have advised engineers are still en route to replace the hardware. Due to the re routing of traffic you may experience slow speeds and/or packet loss. Please accept our apologies for any inconvenience caused and we will continue to provide updates as they become available.

— Update – 14:01 —
Entanet have advised engineers have arrived on site to replace the faulty hardware.

LON01 – 12/11/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*

On the 12/11/2014 between the hours of 20:00 – 23:00 we will be carrying out a series of hardware and software updates on our core network services. Hardware esxi01.r00.lon01.structuredcommunications.co.uk esxi02.r00.lon01.structuredcommunications.co.uk esxi01.r01.lon01.structuredcommunications.co.uk esxi02.r01.lon01.structuredcommunications.co.uk esxi03.r01.lon01.structuredcommunications.co.uk esxi04.r01.lon01.structuredcommunications.co.uk esxi05.r01.lon01.structuredcommunications.co.uk Software ns01.r00.lon01.structuredcommunications.co.uk ns02.r00.lon01.structuredcommunications.co.uk noc.r01.lon01.easybond.co.uk ag-primary.r01.lon01.easybond.co.uk ag-backup.r01.lon01.easybond.co.uk ns01.r01.lon01.easybond.co.uk ns02.r01.lon01.easybond.co.uk smtp01.r01.lon01.easybond.co.uk smtp02.r01.lon01.easybond.co.uk The works will involve disconnecting and removing all physical servers … Continue reading “LON01 – 12/11/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*”

On the 12/11/2014 between the hours of 20:00 – 23:00 we will be carrying out a series of hardware and software updates on our core network services.

Hardware
esxi01.r00.lon01.structuredcommunications.co.uk
esxi02.r00.lon01.structuredcommunications.co.uk
esxi01.r01.lon01.structuredcommunications.co.uk
esxi02.r01.lon01.structuredcommunications.co.uk
esxi03.r01.lon01.structuredcommunications.co.uk
esxi04.r01.lon01.structuredcommunications.co.uk
esxi05.r01.lon01.structuredcommunications.co.uk

Software
ns01.r00.lon01.structuredcommunications.co.uk
ns02.r00.lon01.structuredcommunications.co.uk
noc.r01.lon01.easybond.co.uk
ag-primary.r01.lon01.easybond.co.uk
ag-backup.r01.lon01.easybond.co.uk
ns01.r01.lon01.easybond.co.uk
ns02.r01.lon01.easybond.co.uk
smtp01.r01.lon01.easybond.co.uk
smtp02.r01.lon01.easybond.co.uk

The works will involve disconnecting and removing all physical servers listed above.

The purposes of these works are to upgrade the physical hardware, ESXi versions and Debain operating systems. Each server will be taken offline and replaced / upgraded 1 at a time. The services are redundant and no service outages are anticipated. However in the case of the Easybond NOC, this will likely result in a small outage once the controller is re-loaded back to operational state.

Further updates will be given on the day.

UPDATE01 – 12/11/2014 – 19:20
Engineers are on-site and getting ready for the works listed above.

UPDATE02 – 12/11/2014 – 19:50
Engineers have advised that they are unable to gain remote access to ESXi01.R01 which is host to the EasyBond NOC. A server restart is required to restore access, however this will create additional outages for bonded circuts.

UPDATE03 – 12/11/2014 – 20:00
This work has now started.

UPDATE04 – 12/11/2014 – 20:20
Engineers where able to shutdown the EasyBond NOC without having to cause an outage at the time. The NOC will be offline for around an hour during this host migration.

If you reboot your bonded modem or if it looses coneection it will not eb able to reconnect until the NOC is back in service.

UPDATE05 – 12/11/2014 – 21:15
The EasyBond NOC is back in service. No other service migrations tonight are expected to cause any outages as they are redundant. We will advise once this work is complete

UPDATE06 – 12/11/2014 – 22:29
All work is now complete. Some of the EasyBond services still require some adational works and will be scheduled at a later date.