Site is full of jobs except Illinois

Illinois is currently doing its monthly PM

• Down until later tonight
• Performing some GPFS upgrades
• Moving MWT2 GPFS metadata to NVME (more inodes, better performance)

OSG 3.3.21 to be installed on all gatekeepers later this week

• Has many fixes for AGIS reporting (for BDII retirement)
• Will also go to HTCondor 8.4.11 at this time

Working on migration to latest "wrapper" and job flow

• Using wrapper 0.9.15; moving to 0.9.16
• Cleans up job flow
• CONNECT nearly done; MWT2 next

Retirement of CISCO 6509 caused a dCache issue

• CISCO handled the NAT routing to public network
• Two dCache storage servers had a routing misconfiguration to use NAT
• Nodes were cut off from WAN; needed to fix "route" to nodes public IP
• Monitoring caught the problem quickly
• NAT now provided by a dedicated node other than a switch

New switches are in

• All switches are racked and installed
• However, missing 40Gb modules for EX4500 to 9608
• Downtime in few weeks to installed/configuration the 40Gb connections

Network monitoring at Illinois

• Gaining access to snmp on all switches use by MWT2
• Will soon be able to create graphana graphs on network usage

New Purchases

• UChicago
  • 26 R430
  • 1040 cores
  • Installed and some nodes are online
  • Waiting for new switches to bring remaining nodes online
• Indiana
  • 15 R430
  • 600 cores
  • All nodes online
• Illlinois
  • 8 C6320
  • 448 cores
  • Not installed as yet

MWT2 Site total will be

• 18520 cores
• 192K HS06
• Spreadsheet, GIP and OIM have all been updated