Scheduled maintenance : network firmware upgrade Saturday 12th August 2023 22:00:00

Estimated timeframe : 22:00-00:00

Actual timeframe : 12/08/2023 22:33 - 15/08/2023 22:08

Objective

Upgrade Core and selected network switches to the latest firmware

Actions

Impact

Mitigation

15/08 22:05 The latest applicable fiormware with 0% packet loss has been determined ; the network is considered stable.

Thank you for your patience ; the vendor will be contacted to further explore this issue.

Posted 9 months ago

15/08 20:25 Downgrading to the pre-12/08 firmware has mitigated the packet loss issue.

To detect the earliest firmware release causing the observed issue, multiple release versions will be consecutively installed and tested using the bisecting methodology. Each install takes about 3-4 minutes (during which all network traffic is interrupted) and about 15-30 minutes of observation time.

Posted 9 months ago

15/08 20:00 core stack : reverting to original pre-12/08 firmware

Posted 9 months ago

15/08 19:37 core stack : reverting to lower firmware release

Posted 9 months ago

14/08 15:34 Cold reset yielded no improvement.

Analysis will continue this evening and/or tomorrow (Assumption Day), with test runs causing temporary connectivity interruptions.

Posted 9 months ago

14/08 13:24 While the investigation continues, a cold reset of the core stack is scheduled at 15:00.

As the core stack switches are colocated between iGent and FSVM and an in situ shutdown is required , the procedure will take approx. 15-30 minutes, during which all network traffic is interrupted.

Apologies for the inconvenience.

Posted 9 months ago

14/08 12:10 All distribution switches have been replaced with new hardware ; continuing the investigation.

Posted 9 months ago

14/08 07:16 The issue is not yet solved.

Past night, both distribution switches serving the first floor have been replaced. Work is in progress to replace the ground floor distribution switches too.

Posted 9 months ago

13/08 20:50 In order to pinpoint the issue and reduce variables, the network distribution topology will be modified during the evening and night.

During the intervention, segments of both the internal and external network traffic may be temporarily interrupted.

Posted 9 months ago

13/08 14:30 After last night's firmware updates, we immediately observed a substantial decrease in core switch latency (good), but are experiencing some packet loss to the internet (bad) ; investigating.

Posted 9 months ago

23:33 Updated distribution switches SW016/SW023/SW019 to latest firmware release.

Firmware update session completed.

Posted 9 months ago

22:58 SW001 upgrade completed

Posted 9 months ago

22:51 SW012 upgrade completed ; upgrading SW001 (core)

Posted 9 months ago

22:40 SW010 upgrade completed ; upgrading SW012 (iGent concentrator)

Posted 9 months ago

22:33 upgrading SW010 (UGentNet internet uplink)

Posted 9 months ago