Some systems are experiencing issues

Ongoing

17th June 2021

UK: Coventry [cov1] uk-cov1 core switch issue

The uk-cov1 core switch has experienced a software bug and is routing L3 in software rather than hardware-offloaded. Performance and stability is not currently impacted, however we need to reboot the switch to resolve the bug. We will reboot on 2021/06/18 at 02:00 UTC, at which time all services in uk-cov1 will encounter a brief 30-120 second outage.

History

16th February 2021

No incidents reported

15th February 2021

No incidents reported

14th February 2021

No incidents reported

13th February 2021

No incidents reported

12th February 2021

No incidents reported

11th February 2021

UK: Reading [rdg1] uk-rdg1 core router reload

Due to an unexpected memory leak we're performing an immediate patch and reboot to both A + B core routers in our uk-rdg1 location. We expect a blip of 30 seconds during the router delegation from B to A.

  • Both routers are now patched and reloaded.

  • We are now patching the B router. No interruption is expected.

  • 10th February 2021

    UK network incident

    We are investigating the cause of a network outage to our uk-lon2 and uk-cov1 locations between 13:52-13:56 UTC.

  • We have not detected a recurrence of this incident so we have restored service to this transit provider.

  • We are seeing a repeat of this incident in multiple locations. We are routing around the problematic transit provider until a permanent solution is proposed.

  • The outage, which started 13:52 UTC, was traced to a transit provider's configuration change that caused some packets to be silently dropped. Our network team responded and successfully worked around the issue at 13:56 UTC.