EasyIPT – Network Changes

In line with a previous NOC announcement whereby a BT IPX outage caused service disruption for our EasyIPT VoIP platform (both managed and unmanaged) Structured Communications have, as a result invested in a redundant carrier grade switch along with private bulk fibre and cross-connects to new and existing voice carriers. (known as upstream providers) This … Continue reading “EasyIPT – Network Changes”

In line with a previous NOC announcement whereby a BT IPX outage caused service disruption for our EasyIPT VoIP platform (both managed and unmanaged) Structured Communications have, as a result invested in a redundant carrier grade switch along with private bulk fibre and cross-connects to new and existing voice carriers. (known as upstream providers)

This solution enables us to directly peer with our upstream providers taking out the public internet and resulting in a more robust connection.

These new and existing providers directly terminate in to our carrier grade switch via private SST trunks and our private interconnects. This gives Structured Communications diverse routing in the event of an upstream network issue.

Customers who have a fully managed system with Structured Communications will be automatically migrated outside of hours.

Customers who simply take a SIP trunk from us via our existing carrier will be required to make changes to your PBX. (You will be contacted shortly via the email address on your account)

We are still awaiting the completion of various cross connects from their respective network providers and expect this to take no more than 2 weeks with the migration completed with 4 weeks.

If you have any questions then please don’t hesitate to contact us.

LON01 – EasyHTTP / EasyIPT(Hosted) – 19/06/2014 – 20:00-23:00 – Planned Maintenance *COMPLETE*

We will be undertaking work in LON01to relocate servers for the EasyHTTP & EasyIPT(Hosted) platforms within our racks. This work will result in a brief disruption to service. This work is to create space within Rack01 for the insulation of our new carrier grade switch following on from our previous NOC announcement and planned upgrades. … Continue reading “LON01 – EasyHTTP / EasyIPT(Hosted) – 19/06/2014 – 20:00-23:00 – Planned Maintenance *COMPLETE*”

We will be undertaking work in LON01to relocate servers for the EasyHTTP & EasyIPT(Hosted) platforms within our racks. This work will result in a brief disruption to service.

This work is to create space within Rack01 for the insulation of our new carrier grade switch following on from our previous NOC announcement and planned upgrades. A new post will be created in relation to this upgrade as unmanaged trunks will require PBX changes.

UPDATE01 – 20:10
This work has started

UPDATE02 – 20:15
Engineers have advised we also need to power down our network monitoring server. This has been approved.

UPDATE03 – 11:59
This work is now complete

LON01 – Zen BGP Session – 01:00- 04:00 – 12/06/2014 *Planned Maintenance* *COMPLETE*

We have been advised by Zen that they will be upgrading supervisor software on part of their core network where our cross-connect terminates. During this time we expect our BGP session to drop and be unavailable. Minimal service disruption is expected as traffic will re-coverage over other upstream providers while the link is down.

We have been advised by Zen that they will be upgrading supervisor software on part of their core network where our cross-connect terminates. During this time we expect our BGP session to drop and be unavailable.

Minimal service disruption is expected as traffic will re-coverage over other upstream providers while the link is down.

LON01 – BGP Update – 22:30- 00:30 – 04/06/2014 *COMPLETE*

BT Wholesale have today completed the fit and test on our optical bearer to Zen for private peering and transit. We will be bringing the BGP session online tonight in the above maintenance window. There will be service disruption for any customer who route to us via Zen on a single xDSL circuit, bonded customers … Continue reading “LON01 – BGP Update – 22:30- 00:30 – 04/06/2014 *COMPLETE*”

BT Wholesale have today completed the fit and test on our optical bearer to Zen for private peering and transit.

We will be bringing the BGP session online tonight in the above maintenance window.

There will be service disruption for any customer who route to us via Zen on a single xDSL circuit, bonded customers who have a Zen circuit in the cluster will see this go indirect and a reduction in speed and redundancy will be seen for the duration.

We expect the disruption to be minimal and only during BGP convergence (this includes testing)

UPDATE01 – 21:00
maintenance window moved to 22:30 – 00:30

UPDATE02 – 23:45
Work has started

UPDATE03 – 00:10
The BGP session is now up and traffic is flowing as expected.

UPDATE04 – 02:47
maintenance window closed

EasyIPT – VoIP Calls – Outbound OFFNET – 10:50 18/05/2014 *RESOLVED*

We are aware of a problem with an upstream provider that is preventing the termination of OFFNET (Mobile, Local, National and international) calls. We are currently working to resolve the fault and will provide updates shortly. UPDATE01 – 11:36 Local and national calls should now be terminating correctly, however mobile calls are still resulting in … Continue reading “EasyIPT – VoIP Calls – Outbound OFFNET – 10:50 18/05/2014 *RESOLVED*”

We are aware of a problem with an upstream provider that is preventing the termination of OFFNET (Mobile, Local, National and international) calls. We are currently working to resolve the fault and will provide updates shortly.

UPDATE01 – 11:36
Local and national calls should now be terminating correctly, however mobile calls are still resulting in a BUSY response from our carrier.

UPDATE02 – 12:00
Mobile and international calls are now terminating correctly.

We have logged a case with the carrier for details and will provide this once made available.

We apologise for any inconvenience caused

UPDATE03 – 12:24
We have been advised the root cause of the failure was “a major outage on the BT IP exchange interconnect which caused a high proportion of calls to fail.”

In light of this outage, Structured Communications will be implementing carrier diverse outbound routing. This will require large network changes and further details to follow in the coming weeks.

LON01 – BGP Update – 20:30 – 06/05/2014 *COMPLETE*

In preparation of our new transit and private 1Gbps interconnect to Zen Internet we will be removing all static routes for 3rd party providers and completing a fail-over test prior to the interconnect going live. Once the circuit is handed over to Structured Communications we will create a further out of hour’s maintenance window to … Continue reading “LON01 – BGP Update – 20:30 – 06/05/2014 *COMPLETE*”

In preparation of our new transit and private 1Gbps interconnect to Zen Internet we will be removing all static routes for 3rd party providers and completing a fail-over test prior to the interconnect going live. Once the circuit is handed over to Structured Communications we will create a further out of hour’s maintenance window to bring up the session and complete further checks.

Where a client connects to us via a 3rd party connection not managed by Structured Communications, you will see your traffic revert back to normal paths via Level3 / Adapt.

Due to future network planning and peering we have withdrawn the policy of using static routes.

The above works are expected to take around 10-15 minutes and 2 x 30 seconds blips may be seen while traffic re-converges over our other upstream links.

UPDATE01 – 20:45
Work Complete. No outages where seen.

EasyHTTP – 25/04/2014 – 10:45 – Blacklist / SMTP

We are aware and confirmed by our network monitoring that one of our SMTP servers (95.87.104.30) was used to send out large volume of emails last night from a compromised account. This account has been suspended and actions have been taken to remove the listings from the various listing DNSBL databases. This may take a … Continue reading “EasyHTTP – 25/04/2014 – 10:45 – Blacklist / SMTP”

We are aware and confirmed by our network monitoring that one of our SMTP servers (95.87.104.30) was used to send out large volume of emails last night from a compromised account. This account has been suspended and actions have been taken to remove the listings from the various listing DNSBL databases. This may take a few hours for remote server to reflect this.

We apologise for any inconvenience caused

EasyHTTP – 16/04/2014 – 20:00 – Maintenance *COMPLETE*

Following on from today’s CPU issues, We have been advised there is a major upgrade for Mail Enable. This will be installed tonight to ensure we are running the latest release.# UPDATE 01 – 20:15 This work is now complete. Any users having problems with account syncing are advised to remove and re-add there account … Continue reading “EasyHTTP – 16/04/2014 – 20:00 – Maintenance *COMPLETE*”

Following on from today’s CPU issues, We have been advised there is a major upgrade for Mail Enable. This will be installed tonight to ensure we are running the latest release.#

UPDATE 01 – 20:15
This work is now complete. Any users having problems with account syncing are advised to remove and re-add there account to their mail client. (Remember to back up your messages first)

EasyHTTP – 16/04/2014 – 08:45 – CPU Usage *RESOLVED*

We are aware CPU usage on EasyHTTP is starting to climb to 100% across all 16 cores. We are monitoring the process that is causing the high usage with the view of restarting the server should usage not drop. A restart of the service has not resulted in a fix. UPDATE 01 – 08:53 The … Continue reading “EasyHTTP – 16/04/2014 – 08:45 – CPU Usage *RESOLVED*”

We are aware CPU usage on EasyHTTP is starting to climb to 100% across all 16 cores. We are monitoring the process that is causing the high usage with the view of restarting the server should usage not drop. A restart of the service has not resulted in a fix.

UPDATE 01 – 08:53
The IMAP service has continued to consume CPU usage and levels are now above 90%. Looking at the service threads we are unable to locate any sub service or string that would be causing the high CPU usage. We have therefore opted to restart EasyHTTP which will take around 10 minutes due to the size and configuration of the RAID array.

UPDATE 02 – 09:07
Due to a disk check being requested by the server due to uptime, we expect a further 15 minute delay.

UPDATE 03 – 09:30
The server has now rebooted and all services have been restored however the IMAP service is still using large amounts of the CPUs. We have taken action move the service to a single core so we can continue to fault find. Users may experience a slower email service due to the limits enforce.

UPDATE 04 -10:15
We have discovered a user account with over 700,000 emails in there deleted items which are suspected to be causing the high CPU load.

UPDATE 05 – 11:03
These files have been removed and the IMAP service restarted with all cores enabled. CPU usage is at normal levels. We will continue to monitor of the next few hours.

UPDATE 06 – 12:05
Our network monitor has alerted us that CPU levels have started to climb again. Further review of the IMAP process has high-listed another account of large size however items in stored in the users INBOX which we are unable to delete. We have therefore disabled the account and levels have returned back to normal.

LON01 – BGP Route HiJack – 20:40 – 02/04/2014 *RESOLVED*

We have been alerted to a RPKI Validation failure on both our /21 IP ranges. This notice advises that our IP space is now being miss announced by AS4761 (INDOSAT-INP-AP INDOSAT Internet Network Provider,ID) and has been accepted by some of their upstream peers who do not filter there BGP sessions correctly. We are working … Continue reading “LON01 – BGP Route HiJack – 20:40 – 02/04/2014 *RESOLVED*”

We have been alerted to a RPKI Validation failure on both our /21 IP ranges. This notice advises that our IP space is now being miss announced by AS4761 (INDOSAT-INP-AP INDOSAT Internet Network Provider,ID) and has been accepted by some of their upstream peers who do not filter there BGP sessions correctly.

We are working with our upstream providers to limit any routing issues that may be seen and will provide updates shortly.

UPDATE 01 – 21:00 – 02/04/2014

Our Chief Engineer (Adam Clark) has been dealing with the situation and can advise this is a wide spread issue affecting many thousands of networks and is a repeat of the INDOSAT issue seen in 2011. Since then many providers now filter results from INDOSAT and the RIPE RPKI validation being in use, this has limited the acceptance of our hijacked ranges. An email has also gone off to the INDOSAT contact who manages the network. We are awaiting a response.

At the moment no UK provider has accepted the wrong announcement and we are not seeing any UK related issues, however as many thousands of other networks are caught up in this you many experience problems connecting to the effected networks.

UPDATE 02 – 21:15 – 02/04/2014

We can now advise the only upstream provider of INDOSAT who accepted the announcement was AS4651 (THAI-GATEWAY The Communications Authority of Thailand(CAT),TH) Our upstream providers directly peer with AS4651 and therefore filtered the incorrect data.
As no further repeat alerts have been seen we are happy this has been resolved by INDOSAT although no official response has been given by them.
Should a repeat occur then this NOC notice will be reopened.