Files lost from Dec 6-10 dCache database incident: continue watching; quiet now
Transfers: 13-Mar GGUS: 2775/682642 Transfers failing with "file not found"
13-Mar scan 10k errors - 892 unique files declared 856 bad/lost that day
14-Mar 9 more files
16-Mar 1 more file
None since.
Ticket not closed. Added discussion of suspected lost logfiles.
Job errors: None since 13-Mar
found a few work nodes with cvmfs issues, some requires reboot to fix.
UM site progress on EL9, migrated more services, including AFS servers, SVN, Mariadb etc to EL9, and started to migrate to nftables for firewall.
EL9 at MSU Status
Campus firewall issue finally resolved 12-Mar
Found local firewall issue on Satellite side 13-Mar
Found Apache setup issue on Capsule side 14-Mar
Still Capsule failing while proxying to Satellite request from provisioning node for its kickstart file
Found error log on capsule. Not very specific.
Verified we use a fresh token.
Current/today lead: yesteday spotted one error message on Satellite GUI related to kickstart and snipet.
Will ask RedHat for help if we (AGLT2+MSUIT) can't resolve soon.