Author: Adam Clark (Network Operations)

LON01 – C4L Frontier Interconnect *At Risk Planned Maintenance* *COMPLETE*

We have been advised by one of our fiber providers (C4L) that they will be extending there coreTX network at Goswell Road. Works are **At Risk** as engineers will be present and working within a live network environment. C4L Notice: Location: Interxion, City Lifeline, Telehouse Metro, Goswell Road, Volta 36 Great Sutton St. Start: 2015-02-27, … Continue reading “LON01 – C4L Frontier Interconnect *At Risk Planned Maintenance* *COMPLETE*”

We have been advised by one of our fiber providers (C4L) that they will be extending there coreTX network at Goswell Road. Works are **At Risk** as engineers will be present and working within a live network environment.

C4L Notice:

Location: Interxion, City Lifeline, Telehouse Metro, Goswell Road, Volta 36 Great Sutton St.

Start: 2015-02-27, 20:00
End: 2015-02-28, 06:00

Reason: To install additional switching equipment within the coreTX network.

Impact: N/A. However, due to the nature of working within live network racks, services are at risk of layer 1 interruption.

—

Should we loose the interconnect then our network will automatically re-route traffic via an alternate carrier.

LON01 – EasyIPT – 26/01/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*

We have seen some inbound and outbound VoIP calls fail to connect between the hours of 13:30 – 14:30 today with an odd array of messages on the SIP proxy connecting to our softswitch. Initial diagnostics show nothing obvious, however to be proactive we will be looking deeper in to the platform this evening with … Continue reading “LON01 – EasyIPT – 26/01/2014 – 20:00 till 21:00 *Planned Maintenance* *COMPLETE*”

We have seen some inbound and outbound VoIP calls fail to connect between the hours of 13:30 – 14:30 today with an odd array of messages on the SIP proxy connecting to our softswitch.

Initial diagnostics show nothing obvious, however to be proactive we will be looking deeper in to the platform this evening with the view of rebooting this to clear any stuck session data.

UPDATE01 This work has been rescheduled for the 26/01/2014

UPDATE02 This work has been rescheduled for the 27/01/2014

UPDATE03 20:35 This work has started.

UPDATE04 20:46 The server has requested a disk check to be run. Due to the size of the array this make take 15-20 minutes.

UPDATE05 20:52 The server has returned to operation but the SQL service is now not operating in the correct state following the disk check.

UPDATE06 20:58 The server has returned to operation and the SQL problem resolved. Test calls have been made and they completed successfully.

LON01 – ESXi10 PSU – 12:10 – 26/12/2014 *CLOSED*

Our network monitoring has alerted us to a possable PSU failure on ESXi10. This server has redundant supplys and remains online but in an “at risk” state. Engineers are due to investigate shortly with the aim of restoring the supply remotely if possible should this be confirmed as being in a warning state. Failed hardware … Continue reading “LON01 – ESXi10 PSU – 12:10 – 26/12/2014 *CLOSED*”

Our network monitoring has alerted us to a possable PSU failure
on ESXi10. This server has redundant supplys and remains online but in an “at risk” state.

Engineers are due to investigate shortly with the aim of restoring the supply remotely if possible should this be confirmed as being in a warning state.

Failed hardware will be replaced as soon as possable and the unit shutdown to protect adjacent devices.

UPDATE 01 – 13:00
An Engineer has remotely logged in to the OOB (out of bounds) management interface to get real-time hardware / status information from the server and has been able to confirm both PSUs are in an operational state.

We can only conclude ESXi is feeding our network monitoring incorrect data to our SNMP session.

Further investigation is required as to the cause of this, however we are happy the server is not at risk and closing this risk report

LON01 – **Christmas Network Freeze***

Please note that in order to protect the quality of your services over the busy Christmas period Structured Communications and several of our suppliers will be implementing network freezes. This means that between the dates listed below, no platform changes or other non-essential activity will be conducted. Essential emergency maintenance will continue if required and … Continue reading “LON01 – **Christmas Network Freeze***”

Please note that in order to protect the quality of your services over the busy Christmas period Structured Communications and several of our suppliers will be implementing network freezes. This means that between the dates listed below, no platform changes or other non-essential activity will be conducted. Essential emergency maintenance will continue if required and will be advised via NOC (http://noc.structuredcommunications.co.uk)

Structured From 17:30 on the 19th December 2014 until 09:00 on 5th January 2015

BT No formal freeze but staffing levels will be reduced over the Christmas period

Colt From 15th December to 4th January

Level 3 13th December to January 5th 2015

TalkTalk Business 15th December to January 2nd 2015

Virgin Media From 11th December for project/service launches, 18th December for business as usual changes to 5th January 2015 00:01

Vodafone 12th December 2014 until 5th January 2015

In all cases the period starts at 00:01 and ends at 23:59 unless otherwise stated.

PBX routing changes can be made up until the 23rd Dec 2014 (17:00) however we do request 24 hours notice prior to this cut off due to the increased support demand. Tickets submitted within this 24 hour window will be completed. Customers with their own access can of course make changes past this period.

Other PBX system configuration changes will not be made until the new year.

If you have any questions regarding the network freeze please contact us on 0203 301 4000

LON01 – 15/12/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*

1) We will be making some changes to our VPS cluster to bring online additional capacity. Unmanaged 3rd party VPS sessions may see a small amount of downtime while there images are moved. No other impact is expected. 2) An audit will be conducted on our existing bulk fibre between our suite on the 4th … Continue reading “LON01 – 15/12/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*”

1) We will be making some changes to our VPS cluster to bring online additional capacity. Unmanaged 3rd party VPS sessions may see a small amount of downtime while there images are moved. No other impact is expected.

2) An audit will be conducted on our existing bulk fibre between our suite on the 4th floor and our meet me rack on the 2nd. This is for upcoming network expansion. No outages are expected however the links should be considered “at risk” due to the physical nature of the works. Should an outage occur on a fibre link then our network will automatically re-route traffic via a alternative path.

UPDATE01 – 23:30 These works have been completed as expected, However the 3rd party ESXi images will not be moved until 22:00 Tuesday night (16/12/14)

LON01 – 12/11/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*

On the 12/11/2014 between the hours of 20:00 – 23:00 we will be carrying out a series of hardware and software updates on our core network services. Hardware esxi01.r00.lon01.structuredcommunications.co.uk esxi02.r00.lon01.structuredcommunications.co.uk esxi01.r01.lon01.structuredcommunications.co.uk esxi02.r01.lon01.structuredcommunications.co.uk esxi03.r01.lon01.structuredcommunications.co.uk esxi04.r01.lon01.structuredcommunications.co.uk esxi05.r01.lon01.structuredcommunications.co.uk Software ns01.r00.lon01.structuredcommunications.co.uk ns02.r00.lon01.structuredcommunications.co.uk noc.r01.lon01.easybond.co.uk ag-primary.r01.lon01.easybond.co.uk ag-backup.r01.lon01.easybond.co.uk ns01.r01.lon01.easybond.co.uk ns02.r01.lon01.easybond.co.uk smtp01.r01.lon01.easybond.co.uk smtp02.r01.lon01.easybond.co.uk The works will involve disconnecting and removing all physical servers … Continue reading “LON01 – 12/11/2014 – 20:00 till 23:00 *Planned Maintenance* *COMPLETE*”

On the 12/11/2014 between the hours of 20:00 – 23:00 we will be carrying out a series of hardware and software updates on our core network services.

Hardware
esxi01.r00.lon01.structuredcommunications.co.uk
esxi02.r00.lon01.structuredcommunications.co.uk
esxi01.r01.lon01.structuredcommunications.co.uk
esxi02.r01.lon01.structuredcommunications.co.uk
esxi03.r01.lon01.structuredcommunications.co.uk
esxi04.r01.lon01.structuredcommunications.co.uk
esxi05.r01.lon01.structuredcommunications.co.uk

Software
ns01.r00.lon01.structuredcommunications.co.uk
ns02.r00.lon01.structuredcommunications.co.uk
noc.r01.lon01.easybond.co.uk
ag-primary.r01.lon01.easybond.co.uk
ag-backup.r01.lon01.easybond.co.uk
ns01.r01.lon01.easybond.co.uk
ns02.r01.lon01.easybond.co.uk
smtp01.r01.lon01.easybond.co.uk
smtp02.r01.lon01.easybond.co.uk

The works will involve disconnecting and removing all physical servers listed above.

The purposes of these works are to upgrade the physical hardware, ESXi versions and Debain operating systems. Each server will be taken offline and replaced / upgraded 1 at a time. The services are redundant and no service outages are anticipated. However in the case of the Easybond NOC, this will likely result in a small outage once the controller is re-loaded back to operational state.

Further updates will be given on the day.

UPDATE01 – 12/11/2014 – 19:20
Engineers are on-site and getting ready for the works listed above.

UPDATE02 – 12/11/2014 – 19:50
Engineers have advised that they are unable to gain remote access to ESXi01.R01 which is host to the EasyBond NOC. A server restart is required to restore access, however this will create additional outages for bonded circuts.

UPDATE03 – 12/11/2014 – 20:00
This work has now started.

UPDATE04 – 12/11/2014 – 20:20
Engineers where able to shutdown the EasyBond NOC without having to cause an outage at the time. The NOC will be offline for around an hour during this host migration.

If you reboot your bonded modem or if it looses coneection it will not eb able to reconnect until the NOC is back in service.

UPDATE05 – 12/11/2014 – 21:15
The EasyBond NOC is back in service. No other service migrations tonight are expected to cause any outages as they are redundant. We will advise once this work is complete

UPDATE06 – 12/11/2014 – 22:29
All work is now complete. Some of the EasyBond services still require some adational works and will be scheduled at a later date.

LON01 – EasyIPT, Unmanaged SIP Accounts ***Service / Settings Change Notice FINAL REMINDER***

As per our NOC notice on the 02/10/2014 which can be found HERE. This is a final reminder to advice of these imminent changes. Any PBX / System that is not reconfigured to our EasyIPT service tonight will lose all inbound and outbound call routing once the existing 3rd party services are terminated. All numbers … Continue reading “LON01 – EasyIPT, Unmanaged SIP Accounts ***Service / Settings Change Notice FINAL REMINDER***”

As per our NOC notice on the 02/10/2014 which can be found HERE. This is a final reminder to advice of these imminent changes. Any PBX / System that is not reconfigured to our EasyIPT service tonight will lose all inbound and outbound call routing once the existing 3rd party services are terminated.

All numbers will be automatically re-routed to the respective new trunks / accounts once the old services are terminated. In order to restore your services, PBXs will need to be reconfigured as above.

If you require your login information or have any questions, please don’t hesitate to contact us. Any users requiring out of hours assistance can be arranged FOC.

Structured Communications will not be held responsible for any loss of service due to none reconfiguration of PBX / Systems or configuration issues, not limited to incompatible products.

LON01 – C4L Frontier Interconnect *Resolved*

Our network monitoring has just alerted us to a loss of our C4L layer2 circuit to Frontier. Some active VoIP calls over this carrier / link will have dropped. C4L have just issued a Connectivity Issue alert on their NOC. While the interconnect has been restored we are, given historic outages on this link manually … Continue reading “LON01 – C4L Frontier Interconnect *Resolved*”

Our network monitoring has just alerted us to a loss of our C4L layer2 circuit to Frontier. Some active VoIP calls over this carrier / link will have dropped.

C4L have just issued a Connectivity Issue alert on their NOC.

While the interconnect has been restored we are, given historic outages on this link manually re-routing all outbound traffic away from Frontier to avoid any repeats on outbound traffic flow caused by the C4L issue.

If a re-occurrence does occur then we will drop the circuit completely which will result in our network re-routing ALL VoIP traffic over alternative links. This will have an impact while BGP re-converges and is a last resort option.

We apologies for any inconvenience this may have caused.

UPDATE01 16:01
We have been contacted by our account manager at C4L to advise this is a STP problem once again. A complaint has been issued with C4L and we continue to route outbound calls via an alternative carrier. Inbound remains via the effected link and will be transferred if the link suffers another drop. This advised will case a loss of service while BGP re-converges.

UPDATE02 16:45
The link has remained stable since the initial alert and inbound traffic has remained active without any further loss. C4L has advised that primary POPs have been restored. We will re-enable this link for outbound traffic later tonight.

UPDATE02 18:00
Outbound traffic has now been re-enabled on this link and we are monitoring. Once and RFO is provided we will issue it via the NOC.

LON01 – Frontier Interconnect *Resolved*

Our network monitoring has alerted us to a loss of BGP peering with Frontier Systems. Initial calls to our Layer2 provider for this link (C4L) advise they are aware of a spanning tree issue within their network. Traffic has automatically been re-routed around the link and no service disruption is being seen. UPDATE01 – 10:20 … Continue reading “LON01 – Frontier Interconnect *Resolved*”

Our network monitoring has alerted us to a loss of BGP peering with Frontier Systems. Initial calls to our Layer2 provider for this link (C4L) advise they are aware of a spanning tree issue within their network.

Traffic has automatically been re-routed around the link and no service disruption is being seen.

UPDATE01 – 10:20 – 05/10/3014
After a call to C4L following our link still being offline, we had been advised our VLAN had not been rebuilt on the trunk port where STP had been disabled. The service has now been restored and we have esculated this to our account manager.

LON01 – EasyIPT, Unmanaged SIP Accounts ***Service / Settings Change Notice***

Following on from previous correspondence in respect to unmanaged SIP accounts we have seen the remaining unmanaged 3rd party PBXs connect to our new platform. As a result of this and in order to provide better levels of support we will only be providing accounts via our EasyIPT SIP solution from the 01/11/2014. If you … Continue reading “LON01 – EasyIPT, Unmanaged SIP Accounts ***Service / Settings Change Notice***”

Following on from previous correspondence in respect to unmanaged SIP accounts we have seen the remaining unmanaged 3rd party PBXs connect to our new platform. As a result of this and in order to provide better levels of support we will only be providing accounts via our EasyIPT SIP solution from the 01/11/2014.

If you have not yet migrated over to your new trunk, then we request you do so by the 31/10/2014 to avoid any disruption in voice services.

Changes will be made to the number formatting of your PBX as the invite format has changed to canonical. Example: “442030000000” This number formatting can be added alongside existing PBX numbers to avoid anytime downtime. You will be able to instantly make outbound calls once your system is registered and once you are ready to swap over inbound calls, please contact support who will complete this for you.

If you are having trouble migrating your system or have any questions, please don’t hesitate to contact us directly.