Internal network issues Monday 14th February 2022 08:00:00


For the last few days, we are experiencing intermittent issues (packet loss) in the IRC network, resulting in e.g. temporary slow connectivity to distinct hosts. We also noticed some extensive hostile port scans during the weekend.

The symptoms are highly variable, complicating our analysis. To identify the source of the issue(s), we may be required to temporarily disable internal (FSVM) network segments ; this will be scheduled outside business hours, and pre-announced in due time ; impact to IRC services availability will be minimized where possible.

12:39 network has been stable for the past day with the interim solution in place ; closing this ticket . (analysis will continue in parallel until root cause solved)

20:40 Interim solution in place

18:40 Network stability not improved

18:20 SW016 cluster replaced

13:00 Replacement of SW016 cluster

15:37 Scheduled analysis completed

14:00-15:30 Scheduled analysis of switches sw014 & sw016

02:23 first elimination run terminated

23:05 starting first elimination run (most network endpoints will be temporarily disconnected)

15:10 network anomalies coincide with topology changes ; preliminary intrusive (= possible network interruptions) subnetting analysis scheduled after 21:00.

13:05 still investigating (setting up extensive cross-reference ping/latency matrix)

10:45 check internal/external network performance

09:20 disabling firewall tarpit (was enabled to mitigate hostile portscans during past weekend)