Author: Adam Clark (Network Operations)

17/05/2018 – Ethernet Leased Lines – Migration – 20:00

As part of our planed network migration to Telehouse North and our own facility. We are now ready to migrate existing Ethernet fibre and EFM services from Goswell Road to Telehouse. This work is planned to take place from 20:00 tomorrow night and is expected to take around 1 hour to complete with each service … Continue reading “17/05/2018 – Ethernet Leased Lines – Migration – 20:00”

As part of our planed network migration to Telehouse North and our own facility. We are now ready to migrate existing Ethernet fibre and EFM services from Goswell Road to Telehouse.

This work is planned to take place from 20:00 tomorrow night and is expected to take around 1 hour to complete with each service seeing around 5 minutes of downtime while VLANs are moved around the network to there new home.

Customers with backup ADSL or FTTC are unlikely to notice the drop as services will automatically re-route for the duration. Customers that have dual leased lines are also unlikely to notice a service drop as the move is being done in groups, so no 2 services are in the same migration window.

Updates to be posted on the night. Existing test services have been transferred so we do not expect any problems.

UPDATE01 – 19:45
Engineers are getting ready to start

UPDATE02 – 20:00
Work has started

UPDATE03 – 20:10
A problem has been found in our suppliers scripted code. They are attempting to fix this. Unfortunately this does mean additional downtime for the affected circuits.

UPDATE04 – 20:22
Code has been fixed however we are seeing MTU issues. They are being sorted for both this batch of migrations and the 2nd set.

UPDATE05 – 20:30
Stage 1 has been migrated and BGP sessions are back online. Stage 2 will now start.

UPDATE06 – 20:45
Stage 2 is complete, however there is a problem with 2 circuits.

UPDATE07 – 21:10
This work is now complete and all circuits are showing as online.

11/04/2018 – Email SMTP

We are aware of issues affecting SMTP email delivery on smtp02.structuredcommunications.co.uk We always advise anyone using smtp02.structuredcommunications.co.uk to also setup smtp01.structuredcommunications.co.uk (this server is currently unaffected) Due to the blocks in place on smtp02.structuredcommunications.co.uk it will take several days for the server reputation to recover.

We are aware of issues affecting SMTP email delivery on smtp02.structuredcommunications.co.uk

We always advise anyone using smtp02.structuredcommunications.co.uk to also setup smtp01.structuredcommunications.co.uk (this server is currently unaffected)

Due to the blocks in place on smtp02.structuredcommunications.co.uk it will take several days for the server reputation to recover.

04/04/2018 – Goswell Road (GWR) C3 Migration – 19:00

As part of our THN migration we are due to migrate services from C3 to C2 tonight. There will be no impact on major services such as broadband. Works are only expected to take 10 minutes per service and affect a small number of bonded customers and hosted voice services. UPDATE 01 This work is … Continue reading “04/04/2018 – Goswell Road (GWR) C3 Migration – 19:00”

As part of our THN migration we are due to migrate services from C3 to C2 tonight. There will be no impact on major services such as broadband.

Works are only expected to take 10 minutes per service and affect a small number of bonded customers and hosted voice services.

UPDATE 01

This work is complete. However, during the works a configuration issue was made on a 10GB port that resulted in traffic being re-routed. This was spotted and resolved within minutes.

15/03/2018 – Telehouse North (THN) Enablement & Network Expansion

We are pleased to advise that we have recently added Telehouse North to our network location list and will begin enabling this for service over the next few months. During this time we will be making changes to the current network located in Level3 Goswell Road in order to felicitate this. We hope to keep … Continue reading “15/03/2018 – Telehouse North (THN) Enablement & Network Expansion”

We are pleased to advise that we have recently added Telehouse North to our network location list and will begin enabling this for service over the next few months.

During this time we will be making changes to the current network located in Level3 Goswell Road in order to felicitate this. We hope to keep service disruption to a minimum and NOC notices will be published in advance of any works and the resulting impact they may have to your service.

Goswell Road as a result will be retired and all remaining services moved to our own in house data center. Details on this will follow as the THN migration is planned and the final build phase on our facility is completed.

LON01 – EasyHTTP – 03/03/2018

We have been made aware via a number of external mail partners that they are seeing odd patterns of mail flow from this platform which is affecting SMTP delivery to various destinations. Initial investigations have shown this to be down to a rouge EXE on the server likely due to a compromised site. Scans are … Continue reading “LON01 – EasyHTTP – 03/03/2018”

We have been made aware via a number of external mail partners that they are seeing odd patterns of mail flow from this platform which is affecting SMTP delivery to various destinations.

Initial investigations have shown this to be down to a rouge EXE on the server likely due to a compromised site. Scans are currently underway to isolate the EXE as well as integrity checks on the platform to correct any miss configuration on client folders.

Temempory measures have been added to hopefully prevent it from executing and causing further issues.

As a result of the above scans and audit the platform is under high load until this is resolved. Currently there is no ETA, however we will provide updates once we know more.

UPDATE01 – 20:00

Detailed scans are still underway. Due to the volume of files this is taking longer than planned. Server usage remains high as a result

UPDATE02 – 23:55

Scan progress is at 61% and will continue in to the night / morning. Backups will still run however this will only add to the already excessive load.

UPDATE03 – 04/04/2018 – 09:30

Both scans completed and results are being reviewed.

UPDATE04 – 04/04/2018 – 10:30

Removal works are complete and we are planning to reboot the server from 10:30 to restore a number of security policys.

LON01 – EasyHTTP – Outbound SMTP – 29/12/2017

We are aware of outbound SMTP delivery issues on server01.r02.easyhttp.co.uk This has been caused by a compromised user account on this server that has been used to send out a large volume of SPAM. The account has been terminated and is being dealt with however, it has resulted in temporary delivery restrictions from some remote … Continue reading “LON01 – EasyHTTP – Outbound SMTP – 29/12/2017”

We are aware of outbound SMTP delivery issues on server01.r02.easyhttp.co.uk This has been caused by a compromised user account on this server that has been used to send out a large volume of SPAM. The account has been terminated and is being dealt with however, it has resulted in temporary delivery restrictions from some remote hosts such as Yahoo, Hotmail and AOL.

We are monitoring the situation. Any failed emails depending on the remote host will be queued for re-delivery. If delivery restrictions remain for over 24 hours then we will look to divert outbound email flow via another internal SMTP array on the sort term.

We apologize for any inconvenience caused.

UPDATE01 – 16:00
SMTP traffic from server01.r02.easyhttp.co.uk is being relayed via another smart host as this is affecting internal support sites.

Structured Communications – Christmas 2017 Change Freeze

Structured Communications will be implementing a change freeze from Monday 18/12/2017 at 01:00 GMT until Thursday 04/01/2018 at 09:00 GMT. This freeze is implemented every year during the busy Christmas period to ensure that no changes are made to our network or infrastructure which could affect services. The freeze includes all changes to the network, … Continue reading “Structured Communications – Christmas 2017 Change Freeze”

Structured Communications will be implementing a change freeze from Monday 18/12/2017 at 01:00 GMT until Thursday 04/01/2018 at 09:00 GMT. This freeze is implemented every year during the busy Christmas period to ensure that no changes are made to our network or infrastructure which could affect services.

The freeze includes all changes to the network, non-essential maintenance, non-pre-planned change requests and infrastructure changes. Structured won’t be undertaking any planned maintenance during this time, unless we are responding to a fault.

Support tickets will not be affected by this change along with christmas changes to PBXs and current “inflight” orders already processing.

If you have any further questions regarding this freeze, please get in touch.

17-12-2017 – 22:00 till 22:10 – Planned Maintenance: Broadband

We will be upgrading our Broadband LNS gateways to the latest factory firmware release. This will require several reloads of each device as we are also upgrading the boot loaders along with the new firmware. Customer sessions are expected to drop during the above window and may flip between gateways as each one is upgraded … Continue reading “17-12-2017 – 22:00 till 22:10 – Planned Maintenance: Broadband”

We will be upgrading our Broadband LNS gateways to the latest factory firmware release. This will require several reloads of each device as we are also upgrading the boot loaders along with the new firmware.

Customer sessions are expected to drop during the above window and may flip between gateways as each one is upgraded in turn.

UPDATE 01 – 22:52
This work started late however is now complete with sessions reconnecting as expected. Anyone still without internet access are advised to power off there router / modem for 20 minutes.

LON01 – VoIP – 21/11/2017

We are aware 1 of our media gateways is not releasing channels once a call has cleared down. This is causing BUSY tones or limit exceeded messages We are currently working to resolve this ASAP. UPDATE 01 – 19:00 Emergency Works have started. Any active calls on SIP02 have been dropped. We are sorry for … Continue reading “LON01 – VoIP – 21/11/2017”

We are aware 1 of our media gateways is not releasing channels once a call has cleared down. This is causing BUSY tones or limit exceeded messages

We are currently working to resolve this ASAP.

UPDATE 01 – 19:00
Emergency Works have started. Any active calls on SIP02 have been dropped. We are sorry for any inconvenience caused.

UPDATE 02 – 19:03
SIP02 has reloaded and all services have restored. We will now look at SIP01

UPDATE 03 – 19:05
Emergency Works have started. Any active calls on SIP01 have been dropped. We are sorry for any inconvenience caused.

UPDATE 04 – 19:03
SIP01 has reloaded and all services have restored.

LON01 – C002.017 – 06/11/2017 – 11.32 – *At Risk*

Our network monitoring had alerted us to a power issues within one of our Cross-Connect / POP cabinets on the 2nd floor within Goswell Road. Our 4th floor suite is unaffected, however all services routing via C002.017 should be considered at risk until this is resolved. We are current engaged with Level3 and will provide … Continue reading “LON01 – C002.017 – 06/11/2017 – 11.32 – *At Risk*”

Our network monitoring had alerted us to a power issues within one of our Cross-Connect / POP cabinets on the 2nd floor within Goswell Road. Our 4th floor suite is unaffected, however all services routing via C002.017 should be considered at risk until this is resolved. We are current engaged with Level3 and will provide updates shortly.

UPDATE01 – 11.57
Level3 have responded to advise tier 2 Technicians are to investigate the incident and provide feedback on their findings within 1 hour.

UPDATE02 – 12:11
Level3 have responded to advise they have sent a subcase to thier techs in Goswell Road to continue to investigate.

UPDATE03 – 13:20
Level3 have issued the following statement:

A non-service affecting partial power rack failure at the Goswell Gateway in London, England is ongoing. The European NOC has advised that an electrical contractor has been engaged and is onsite working to resolve the power issue. There is no estimate as to when the power issue will be resolved; however, the European NOC has confirmed that this issue is not impacting services. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted.

We do have a number of servers and core network equipment operating in redundancy mode due to the power loss and remain at risk.

UPDATE04 – 14:25
Level3 have issued the following update:

Event The European NOC has advised that the electrical contractor has replaced the faulty power cabling onsite clearing all of the alarms related to the partial power failure. At this time, efforts to contact the impacted clients to confirm service restoral are now underway.

We are still seeing network alarms in relation to power and have asked Level3 to tech to step in as we suspect one of our circuit breakers may have tripped. We will advise once further updates are given.

UPDATE05 – 18:30
We have chased Level3 for an update as no response has been given to our request. A level 1 escalation has also been raised.

UPDATE06 – 20:30
We have chased Level3 for an update as no response has been given to our request. A level 2 escalation has also been raised.

UPDATE07 – 21:58
Power has been restored and services are no longer “at risk” initial investigations advise it was more than a circuit breaker at fault. A complaint has been raised.