Actual timeframe : 20:45 - 22:05

Objective

Upgrade the Single Sign On (SSO) to version v26.x

Impact

• During the migration, https://sso.irc.ugent.be will not be available.

This will result in not being able to log in to multiple services including (but not limited to):

• https://schedule.fsvm.be / https://schedule.irc.ugent.be
• https://wiki.irc.ugent.be
• https://rdg.irc.ugent.be
• https://project.irc.ugent.be/kb
• https://intra.irc.ugent.be/collab-hr
• https://rdg.irc.ugent.be

Actual timeframe : 17:00 - 20:00

Objective

Import all PSB meeting locations, past (1 year) and future reservations.

Impact

• During the migration, https://schedule.fsvm.be / https://schedule.irc.ugent.be will not be available.

We are monitoring the situation.

(See https://status.belnet.be/incidents/159 for more info).

Impacted services:

• SLURM compute nodes (cn10293, cn12383, cn12397, cn12408)
• bioit3.irc.ugent.be
• sctcloud.irc.ugent.be
• cloud.irc.ugent.be/public

Actual timeframe : 15:00 - 18:03

Objective

Migrate FS4 to a new virtualization host and expand storage enclosure cluster

Impact

• During the upgrades, M:/N:/X: network volumes and assoicated public cloud shares will not be available or experience performance degradation.

Actual timeframe : 12:04 - 13:46

Objective

Migrate public webserver to a new platfrom

Impact

• During the migration, https://www.irc.ugent.be will not be available.

Actual timeframe : 10:50-00:30

Objective

Upgrade IRCChat to the latest version 7.x

Actions

• upgrade application
• migrate existing data

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 09:02 - 19:24

Objective

Migrate fileservers FS1,FS3,FS4 to Belnet IPAM

Impact

• During the migration, S: , T: , M: , N: , O: , X: network volumes will not be available.
• Associated services (IRCChat, IRCCloud, ...) will be unavailable or experience performance degradation.

Objective

Core switch expansion

In accordance with last weeks rescheduling, the core network switch was successfully expanded today.

Actual timeframe : 10:30 - 18:59

Objective

Core switch expansion and backbone firmware upgrades.

Impact

• During the interventions, all public (mail, web, cloud, chat, ...) and private (file services, printing, WiFi, ...) IRC network services will be impacted.

Actual timeframe : 10:00 - 22:47

Objective

Upgrade FS1 to latest O/S version

Impact

• During the migration, S: and T: network volumes will not be available.
• Associated services (IRCChat, IRCCloud, ...) will be unavailable or experience performance degradation.

Actual timeframe : 09:30 - 19:11

Objective

Upgrade FS4 and its virtualization host to latest O/S version

Impact

• During the upgrades, M:/N:/X: network volumes and Single Cell Cloud will not be available or experience performance degradation.

Actual timeframe : 20:45 - 23:58

Objective

Upgrade SCCloud to latest release

Impact

• During the upgrades, the Single Cell cloud service will be intermittently interrupted.

Actual timeframe : 01/06/2024 22:05 - 02/06/2024 23:46

Objective

Upgrade IRCCloud to latest release

Impact

• During the upgrades, private and public cloud services will be intermittently interrupted.

Actual timeframe : 14:31 - 16:47

Objective

Migrate FS1 VM to new BM

Impact

• During the migration, S: and T: network volumes will not be available.
• Associated services (IRCChat, IRCCloud, ...) will be unavailable or experience performance degradation.

It can still be reached from the wired IRC network, Eduroam or other (e.g home networks).

Actual timeframe : 14:40 - 16:16

Objective

Migrate e-mail infrastructure to Belnet

As only a part of the storage is degraded many webtools still work for the time being. SingleCell compute is affected when accessing recent data...

• BioIT2 Web Server
• BioIT3 Web Server

Investigate baremetal server cp0071.

Impact

The following services are impacted : Mausoleum, FS4, DHCP, DNS, sctCloud

Mitigation

• Consult this page for status updates

This has a major impact on Eduroam / Guestroam functionality. DNS may also be impacted.

For the latest belnet status : https://status.belnet.be/

Actual timeframe : 12/08/2023 22:33 - 15/08/2023 22:08

Objective

Upgrade Core and selected network switches to the latest firmware

Actions

• upgrade & reboot switches
• observe network connectivity

Impact

• During the upgrades, internal FSVM1/FSVM2 network and external server connections will be temporarily interrupted.

Mitigation

• To prevent data loss or corruption, suspend or close applications which process network-sourced data.

We are investigating the issue, status updates can be found here

We are investigating the issue, status updates can be found here

The issue was identified ; we will take countermeasures to mitigate future issues.

Actual timeframe : 18:09 - 08:08

Objective

Migrate e-mail infrastructure to new hardware platform

The affected servers (eduroam, sso, ...) will be restarted as soon as circumstances permit.

Actual timeframe : 19:35-01:21

Objective

Migrate FS1 datasets (Q/R/T/U/netshares) to new RAID array

Note : backups of S: and T: are currently suspended, as the data integrity of the snapshot backup server is of primordial importance.

Actual timeframe : 15:50-16:25

Objective

Identify why Eduroam is no longer accessible

Actions

• Identify the issue
• Fix the issue

Impact

• Eduroam is no longer accessible

Mitigation

• Consult this page for status updates

Actual timeframe : 22:15-03:02

As previously announced, extended network tests are being performed, impacting network-based services.

The symptoms are highly variable, complicating our analysis. To identify the source of the issue(s), we may be required to temporarily disable internal (FSVM) network segments ; this will be scheduled outside business hours, and pre-announced in due time ; impact to IRC services availability will be minimized where possible.

**Actual timeframe : 2021-11-15 12:31 - 2021-11-16 15:15

Objective

Figure out what is/went wrong with SW035

Actions

• Investigate logfiles
• Draw conclusion

Impact

• Cabled network unavailable in W120

Mitigation

• Use IRC/Eduroam wifi
• Consult this page for status updates

Fix kanboard database issues

Actions

• Expand pgsql data volume

Impact

• https://project.irc.ugent.be/kb is not accessible

Mitigation

• Consult this page for status updates

Actual timeframe : ??:??-??:??

Objective

Fix Eduroam authentication issues

Actions

• Check Eduroam logs

Impact

• Eduroam is not accessible

Mitigation

• Consult this page for status updates

Actual timeframe : 15:05-16:30

Objective

Reinstate full resiliency of the network core stack, after a decreased resiliency caused by a stack member switch dropout in the iGent datacenter facility following yesterday's firmware upgrade.

Actions

• investigate stack member dropout
• remediate on-site (if possible)

Impact

• During the intervention, full network availability will be maintained where possible. If required, a rolling reboot may inflict a short interruption (1-2 minutes) of specific services.

Mitigation

• none

Actual timeframe : 14:59-17:40

Objective

Upgrade Core and selected network switches to the latest firmware

Actions

• upgrade & reboot switches
• observe network connectivity

Impact

• During the upgrades, internal FSVM1/FSVM2 network and external server connections will be temporarily interrupted.

Mitigation

• Do prevent data corruption, close network-sourced data and documents
• Consult this page for status updates

Actual timeframe : 21:45-17:08

Objective

Migrate FS4 datasets to intermediary storage volumes

Actions

• migrate flowcyto data
• migrate sequencing data
• migrate irccloud data
• migrate other data

Impact

• During the migration procedure, select data folders will be temporarily unavailable.

Mitigation

• Do not access the respective folders until after migration completion
• Consult this page for status updates

wifi is available again in W118&W120

Actual timeframe : 09:??-09:31

Objective

Transfer from the "irc.ugent.be", "fvms.ugent.be" and "dmbr.ugent.be" zones from UGentNet/DICT servers to IRC DNS servers

Impact

• During and after the zone delegation, no adverse effects should be noticeable.

Mitigation

Zone transfer will be closely monitored by UGent/DICT and IRC staff (internet propagation of the change may take up to 24-48 hours).

Actual timeframe : 21:00-21:03

Objective

Upgrade IRCChat to the latest version 3.12.1

Actions

• upgrade application
• migrate existing data

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 19:01-21:34

Objective

Diagnose FSVM1/FSVM2 network issues

Actions

• sequentially shutdown network zones
• observe topology changes

Impact

• During a shutdown, network connectivty in the affected zone is temporarily suspended.

Mitigation

• Assure that network-critical jobs have finished before 19:00
• Do not start network-critical jobs during the diagnosis
• Consult this page for status updates

Actual timeframe : 15:05-15:39

Objective

Upgrade IRCChat to the latest version 3.9.3

Actions

• upgrade application
• migrate existing data

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Actual timeframe : Fri 30/10/2020 15:54 - Tue 03/11/2020 01:39

Objective

Migrate FS4 instrumentation data to a new storage platform

Actions

• upgrade FS4 fileserver (EL7 -> EL8)
• migrate existing data

Impact

• During the migration, FS4 M: and N: network volumes will not be available.

Mitigation

• Do not access M: and N: during the upgrade procedure
• Consult this page for status updates

"Actual timeframe : 07:35 - 07:56"

Actual timeframe : 12:05-14:21

Objective

Upgrade all IRCCloud instances (public,private,sandbox) to their latest applicable versions

Actions

• upgrade application
• migrate existing data

Impact

• During the upgrade procedure, IRCCloud will not be available.

Mitigation

• Do not use IRCCloud during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 00:15-00:45

Objective

Upgrade IRCChat to the latest version 3.7.1

Actions

• upgrade application
• migrate existing data

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Intranet connectivity is seriously impacted due to an issue with our DNS server.

Incident post-mortem

A configuration error (committed July 5th) surfaced today, when the BIND service was restarted after a security update. Operations resumed after fixing the error.

Actual timeframe : 16:44-17:40

Objective

Upgrade IRCChat to the latest version 3.5.4

Actions

• upgrade application
• migrate existing data

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 14:08-15:02

Objective

Move virtualized servers from EL7 virtualisation host CP0071 to EL8 virtualisation host CP0192

Actions

• upgrade virtualization host CP0192 to EL8
• migrate virtual servers from CP0071 to CP0192
• upgrade RocketChat to the latest version

Impact

_During the upgrade procedure, the following services will not be available : _

• database server (P:\ network volume)

• instrumentation (FlowCyto, Crystallo, etc.) file server (M:\ and N:\ network volumes)

• Schedule (resource calendaring)

• Project (Kanboard kanban process management)

• IRCChat (real-time chat & conferencing)

Mitigation

• Refrain from using the affected services during the upgrade procedure

Friday afternoon at 15:57, the bare metal CP0081 virtualisation host emitted an OOM (Out Of Memory), and executed a shutdown of 2 of the 7 virtualised servers : CP0001 and CP0152, which rendered file services (S: , T:) and print services unavailable ; related services (IRCChat) were affected too.

• file services (and IRCChat) were restored at 16:59 ;
• print services were restored at 21:41.

Actual timeframe : 17:58-18:32

Objective

Upgrade and reconfigure the iGent SW012 2-switch stack

Actions

• reboot stack
• check configuration

Impact

• During the upgrade procedure, network interruptions to IRC servers and the internet may occur.

• Upgrade Schedule to the latest version v2.8.2

Actions

• Upgrade IRC Schedule

Impact

• During the upgrade procedure, IRC Schedule will not be available.

Mitigation

• Do not use IRC Schedule during the upgrade procedure
• Consult this page for status updates

Some services (but not limited to) are: Wiki, BioIT Tools, Chat, Kanboard, Schedule, Cloud, ...

Actual timeframe : 15:45-17:54

Objective

Deconstruct the iGent SW010 4-switch stack

Actions

• migrate iGent server connections from 4 to 2 switches
• split stack
• reconstruct 2-switch stack resiliency

Impact

• During the procedure, short network interruptions may occur.

Actual timeframe : 18:30-18:36

Objective

Upgrade IRCChat to the latest version 2.4.11

Actions

• upgrade IRCChat

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 14:30-20:10

Objective

Migrate distribution switches to new network core

Actions

• migrate FSVM1 distribution switches
• prepare iGent distribution stack migration

Impact

• During the migration, network zones will experience a short interruption.

Mitigation

• Do not rely on uninterrupted network connectivity
• Do not initiate long-term network data transfers

Actual timeframe : 14:30-16:32

Objective

Replace current core switch stack with new core switch stack

Impact

• During the migration, all network connectivity will be intermittently interrupted.

Mitigation

• Do not rely on uninterrupted network connectivity
• Do not initiate long-term network data transfers

Actual timeframe : 09:04-09:50

Objective

Upgrade IRCChat to the latest version 2.4.0

Actions

• upgrade IRCChat

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 2019-11-06 15:00 - 2019-11-08 22:12

Objective

Upgrade IRCChat to the latest version 2.2.0

Actions

• upgrade Rocket.Chat 0.74.3 -> 1.0.0 -> 1.0.1 -> 1.0.2 -> 1.0.3
• upgrade Rocket.Chat 1.0.3 -> 1.1.0 -> 1.1.1 -> 1.1.2 -> 1.1.3
• upgrade Rocket.Chat 1.1.3 -> 1.2.0 -> 1.2.1 (1.0.4) (1.1.4) -> 1.2.3
• upgrade Rocket.Chat 1.2.3 -> 1.3.0 (1.0.5) (1.1.5) (1.2.4) -> 1.3.1 -> 1.3.2
• upgrade MongoDB 3.2 -> 3.4 -> 3.6
• upgrade Rocket.Chat 1.3.2 -> 2.0.0
• upgrade Rocket.Chat 2.0.0 -> 2.1.0 -> 2.1.1 -> 2.1.2 -> 2.2.0

Impact

• During each intermediate upgrade, IRCChat will not be available.

Mitigation

• Do not use IRCChat during an intermediate upgrade
• Consult this page for status updates

Actual timeframe : 00:25-03:05

Objective

Migrate IRC publicCloud server to production status

Actions

• switch cloud - cloud2 presences
• expand cloud backend storage
• deactivate reverse proxy
• execute test scenarios

Impact

• During the migration procedure, IRCCloud instances (sandbox / private / public) will not be available.

Actual timeframe : 10:30-16:21

Objective

Upgrade firewall switching/routing/filtering throughput (1GbE -> 10GbE)

Operations

• switch to new platform
• test filtering/NAT
• test throughput

Impact

• interrupted network traffic between Intranet/DMZ/Internet
• IRC servers will intermittently be unreachable

Mitigation

• Do not rely on IRC networked services (storage, mail, web, print, ...)
• Consult this page for status updates

Actual timeframe : 19:31-19:40

Objective

Upgrade IRCChat to the latest version 0.74.3

Actions

• upgrade IRCChat

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 11:24-13:14

Objective

Migrate virtualized servers from flakey Dell PE R830 virtualization hosts to newer PE R7425 architecture

Actions

• shutdown services
• reassign RAID VM storage
• restart services

Impact

During the migration, the following IRC services will not be available :

• LDAP (authentication)
• e-mail (sending, retrieving, webmail)
• Mausoleum , BCCM/LMBP
• FS4 (M,N:-drives)
• IRCChat, WIKI , Schedule, KanBoard, Helpdesk
• IRCCloud
• Proxy (fs0)
• DB1 Database (Access)

Mitigation

• Do not use the aforementioned services during the migration procedure
• Consult this page for status updates

Actual timeframe : 15:52-18:02

Issue

Several IRC services are currently unavailable.

Cause

While preparing for a server migration, a cable mismanipulation resulted in the interruption of virtual server hosting, causing outage of several IRC services.

Impact

Currently, the following services are interrupted :

• FS4 (M,N:-drives)
• Schedule
• KanBoard
• DB1 Database (Access , publicWWW, intraWWW/OLWOIS)

Follow-up

The incident was used to partially implement the planned server migration, which is now 90% completed.

Actual timeframe : 23:08-06:35

Objective

Replace the current aging firewall with a modernized architecture

Operations

• apply rules
• test packet filters
• test NAT

Impact

• no or heavily interrupted network traffic between Intranet/DMZ/Internet
• IRC servers will not or only very intermittently be reachable

Mitigation

• Do not rely on any IRC services (storage, mail, web, print, ...)
• Consult this page for status updates

Actual timeframe : 14:41-14:45

Objective

We are observing network spikes and delays on the departmental firewall traffic, resulting in detoriated networking services. Restarting the firewall will clear the IRQ queue, and ameliorate networking services.

Actions

• restart firewall

Impact

• During the restart, most networked services will not be available.

We were not informed about this intervention.

Our LDAP certificate has expired ; many services, relying on LDAP authentication, are unable to login.

Actions

• retrieve new SSL certificate

Impact

• Unable to authenticate on web services

Mitigation

• Consult this page for status updates

Actual timeframe : 11:00-11:40

Objective

Replace the current aging firewall with a modernized architecture

Operations

• remotely switch network traffic from old firewall to new firewall
• test remote accessibility

Impact

• no or heavily interrupted network traffic between Intranet/DMZ/Internet
• IRC servers will not or only very intermittently be reachable

Mitigation

• Do not rely on any IRC services (storage, mail, web, print, ...)
• Consult this page for status updates

Actual timeframe : 11:40-17:12

Objective

Replace the current aging firewall with a modernized architecture

Operations

• apply rules
• test packet filters
• test NAT

Impact

• no or heavily interrupted network traffic between Intranet/DMZ/Internet
• IRC servers will not or only very intermittently be reachable

Mitigation

• Do not rely on any IRC services (storage, mail, web, print, ...)
• Consult this page for status updates

Actual timeframe : 16:45-17:57

Objective

Rebase to latest O/S update EL7.6

Actions

• apply latest Dell firmware updates
• restart bare metal server CP0081
• restart hosted virtual servers

Impact

_During the upgrade procedure, the following services will not be available : _

• CP0000 : Windows XP domain logon
• CP0001 : FS1 file services (Q:,R:,S:,T:,U: network storage)
• CP0152 : print services
• CP0385 : DHCP and DNS name services (this affects all network operations)
• CP0492 : FS0 network storage gateway/proxy
• CP0575 : LDAP authentication

Mitigation

• To prevent data loss, close open documents on the affected network storage
• Consult this page for status updates

Actual timeframe : 15:30-15:38

Objective

Upgrade KanBoard to the latest version 1.2.7

Actions

• upgrade application
• migrate existing data

Impact

• During the upgrade procedure, KanBoard will not be available.

Mitigation

• Do not use KanBoard during the upgrade procedure
• Consult this page for status updates

Actual timeframe : 14:32-14:47

Objective

Upgrade IRCChat to the latest version 0.73.0

Actions

• upgrade IRCChat

Impact

• During the upgrade procedure, IRCChat will not be available.

Mitigation

• Do not use IRCChat during the upgrade procedure
• Consult this page for status updates

This issue is preventing user from logging in to the public and private clouds. Existing session should be unaffected.

We are investigating the issue

While we are at it we will also expand the storage on the bioit2 web server. This will cause some BioIT services to be offline at around 15:30.

Actual timeframe : 10:19-16:10

Objective

Replace the current aging firewall with a modernized architecture

Operations

• test routing (connectivity)
• test LACP (resilience)
• test packet filters (security)

Impact

• no or heavily interrupted network traffic between Intranet/DMZ/Internet
• IRC servers will not or only very intermittently be reachable

Mitigation

• Do not rely on any IRC services (storage, mail, web, print, ...)
• Consult this page for status updates