Scheduled maintenance : network firmware upgrade Saturday 12th August 2023 22:00:00

15/08 20:25 Downgrading to the pre-12/08 firmware has mitigated the packet loss issue.

To detect the earliest firmware release causing the observed issue, multiple release versions will be consecutively installed and tested using the bisecting methodology. Each install takes about 3-4 minutes (during which all network traffic is interrupted) and about 15-30 minutes of observation time.

15/08 20:00 core stack : reverting to original pre-12/08 firmware

15/08 19:37 core stack : reverting to lower firmware release

14/08 15:34 Cold reset yielded no improvement.

Analysis will continue this evening and/or tomorrow (Assumption Day), with test runs causing temporary connectivity interruptions.

14/08 13:24 While the investigation continues, a cold reset of the core stack is scheduled at 15:00.

As the core stack switches are colocated between iGent and FSVM and an in situ shutdown is required , the procedure will take approx. 15-30 minutes, during which all network traffic is interrupted.

Apologies for the inconvenience.

14/08 12:10 All distribution switches have been replaced with new hardware ; continuing the investigation.

14/08 07:16 The issue is not yet solved.

Past night, both distribution switches serving the first floor have been replaced. Work is in progress to replace the ground floor distribution switches too.

13/08 20:50 In order to pinpoint the issue and reduce variables, the network distribution topology will be modified during the evening and night.

During the intervention, segments of both the internal and external network traffic may be temporarily interrupted.

13/08 14:30 After last night's firmware updates, we immediately observed a substantial decrease in core switch latency (good), but are experiencing some packet loss to the internet (bad) ; investigating.

23:33 Updated distribution switches SW016/SW023/SW019 to latest firmware release.

Firmware update session completed.

22:58 SW001 upgrade completed

22:51 SW012 upgrade completed ; upgrading SW001 (core)

22:40 SW010 upgrade completed ; upgrading SW012 (iGent concentrator)

22:33 upgrading SW010 (UGentNet internet uplink)