Category: Uncategorized

Update: 15:52
Virgin Media have advised that all repair work is now complete and all services restored. Circuits should no longer be considered at risk.

We apologise the inconvenience caused during the course of this incident.

————-

We are currently aware of an issue affecting a number of on-net Virgin Media based circuits. We believe this is due to a fibre break within the Virgin Media network, no ETR has currently been given.

Services with backup media such as DSL will have been re-routed.

Update: 09:35
Virgin Media have advised that the location of the break has been identified as 7.5km away from a local Virgin Media hub. Splicing engineers are currently attending the closest joint to determine the exact location of the break.

Update: 10:26
Following our previous post, several engineers are now on site identifying the exact location of the fibre break. We are expecting more specific details on the break in due course.

We will keep in contact with the relevant escalation points within Virgin Media to obtain the latest information, as it becomes available.

Update: 11:30
Virgin Media have advised engineers have attended the next joint along from the break and completed Optical time-domain reflectometer (OTDR) testing between the joints. Red light testing is currently underway to confirm the precise location of the break. Virgin Media have also confirmed splicing teams are on site to begin work as soon as the break location is confirmed.

We will post updates as soon as they become available.

Update: 12:58
Virgin Media engineers have located the break between Corbet Place and Brick Lane in London. Engineers are currently accessing the pits this cable passes through to visually inspect the cable and identify if any spare fibres can be used. If any spare fibres are identified they will be prioritised for our services.

Further updates will be posted as soon as available.

Update: 13:22
Virgin Media have advised that spare fibres have been located. Splicing will begin on the first end of the cable shortly and should be complete within 30 minutes. Splicing at the second end of the cable involves more work and is expected to take up to a further 60 minutes to complete.

Further updates will be posted when available.

Update: 14:55
Splicing work on the cables that carry our services has been completed and connectivity restored to all affected circuits. Virgin Media advised work is still ongoing at the site of the break so circuits should still be considered at risk.

We will advise once Virgin Media confirm all repair work complete.

Update: 15:52
Virgin Media have advised that all repair work is now complete and all services restored. Circuits should no longer be considered at risk.

We apologise the inconvenience caused during the course of this incident.

We are currently aware of an issue effecting VoIP calls across our network.

Engineers are currently working on this and an update will be provided shortly.

-Update 12:29-

Calls now seem to be flowing correctly, engineers are continuing to investigate.

Our network monitoring has alerted us to a possable PSU failure
on ESXi10. This server has redundant supplys and remains online but in an “at risk” state.

Engineers are due to investigate shortly with the aim of restoring the supply remotely if possible should this be confirmed as being in a warning state.

Failed hardware will be replaced as soon as possable and the unit shutdown to protect adjacent devices.

UPDATE 01 – 13:00
An Engineer has remotely logged in to the OOB (out of bounds) management interface to get real-time hardware / status information from the server and has been able to confirm both PSUs are in an operational state.

We can only conclude ESXi is feeding our network monitoring incorrect data to our SNMP session.

Further investigation is required as to the cause of this, however we are happy the server is not at risk and closing this risk report

Please note that in order to protect the quality of your services over the busy Christmas period Structured Communications and several of our suppliers will be implementing network freezes. This means that between the dates listed below, no platform changes or other non-essential activity will be conducted. Essential emergency maintenance will continue if required and will be advised via NOC (http://noc.structuredcommunications.co.uk)

Structured From 17:30 on the 19th December 2014 until 09:00 on 5th January 2015

BT No formal freeze but staffing levels will be reduced over the Christmas period

Colt From 15th December to 4th January

Level 3 13th December to January 5th 2015

TalkTalk Business 15th December to January 2nd 2015

Virgin Media From 11th December for project/service launches, 18th December for business as usual changes to 5th January 2015 00:01

Vodafone 12th December 2014 until 5th January 2015

In all cases the period starts at 00:01 and ends at 23:59 unless otherwise stated.

PBX routing changes can be made up until the 23rd Dec 2014 (17:00) however we do request 24 hours notice prior to this cut off due to the increased support demand. Tickets submitted within this 24 hour window will be completed. Customers with their own access can of course make changes past this period.

Other PBX system configuration changes will not be made until the new year.

If you have any questions regarding the network freeze please contact us on 0203 301 4000

In preparation of our new transit and private 1Gbps interconnect to Zen Internet we will be removing all static routes for 3rd party providers and completing a fail-over test prior to the interconnect going live. Once the circuit is handed over to Structured Communications we will create a further out of hour’s maintenance window to bring up the session and complete further checks.

Where a client connects to us via a 3rd party connection not managed by Structured Communications, you will see your traffic revert back to normal paths via Level3 / Adapt.

Due to future network planning and peering we have withdrawn the policy of using static routes.

The above works are expected to take around 10-15 minutes and 2 x 30 seconds blips may be seen while traffic re-converges over our other upstream links.

UPDATE01 – 20:45
Work Complete. No outages where seen.

We are aware and confirmed by our network monitoring that one of our SMTP servers (95.87.104.30) was used to send out large volume of emails last night from a compromised account. This account has been suspended and actions have been taken to remove the listings from the various listing DNSBL databases. This may take a few hours for remote server to reflect this.

We apologise for any inconvenience caused

We have been alerted to a RPKI Validation failure on both our /21 IP ranges. This notice advises that our IP space is now being miss announced by AS4761 (INDOSAT-INP-AP INDOSAT Internet Network Provider,ID) and has been accepted by some of their upstream peers who do not filter there BGP sessions correctly.

We are working with our upstream providers to limit any routing issues that may be seen and will provide updates shortly.

UPDATE 01 – 21:00 – 02/04/2014

Our Chief Engineer (Adam Clark) has been dealing with the situation and can advise this is a wide spread issue affecting many thousands of networks and is a repeat of the INDOSAT issue seen in 2011. Since then many providers now filter results from INDOSAT and the RIPE RPKI validation being in use, this has limited the acceptance of our hijacked ranges. An email has also gone off to the INDOSAT contact who manages the network. We are awaiting a response.

At the moment no UK provider has accepted the wrong announcement and we are not seeing any UK related issues, however as many thousands of other networks are caught up in this you many experience problems connecting to the effected networks.

UPDATE 02 – 21:15 – 02/04/2014

We can now advise the only upstream provider of INDOSAT who accepted the announcement was AS4651 (THAI-GATEWAY The Communications Authority of Thailand(CAT),TH) Our upstream providers directly peer with AS4651 and therefore filtered the incorrect data.
As no further repeat alerts have been seen we are happy this has been resolved by INDOSAT although no official response has been given by them.
Should a repeat occur then this NOC notice will be reopened.

Our engineers where alerted via our network monitoring to a DDoS attack (Distributed Denial of Service) on one of our ESXi servers.

The attack and traffic has been mitigated however for the duration any directly connected services would have seen small amounts of packet loss.

We are monitoring the network and apologise for any inconvenience caused.

We are aware of an issue with our ticket system whereby the main website is not loading. It’s also possible emailed tickets are not being processed.

As we do not host the site internally due to diversity, we are investigating the issue with our 3rd party provider.

If your query is urgent then please contact us via phone.

UPDATE 01 – 12:21
We have been advised the root cause has now been addressed and this was caused by a DNS lookup limitation which has been removed while a long time solution is found.

We are aware of a fault where outbound calls are taking up to 30 seconds to connect. Engineers are currently investigating this and will update you as more information becomes available.

UPDATE01 – 09:45
We are seeing outbound calls starting to connect as normal again. Inbound should also now be working.

UPDATE01 – 10:30
We have requested further details from our wholesaler as to the cause of the issue.