The cluster is almost fully back on line following the upgrade.  We planned to update the firmware of all servers during the downtime, which may have been a mistake as there were difficulties updating the firmware of the older servers.  Also, we had a problem with mac address flapping on an old switch which was interfering with pxe booting, if we can't find the origin of the problem we may have to replace that switch.  So the process went more slowly than we were hoping, but it is now almost complete.  The remaining servers (those without enterprise IDRAC licenses) will have their firmware updated today.

The cluster was briefly blacklisted last night after cvmfs failed to start correctly when a node was restarted.

Ticket 1000422 can be closed, the old squid just had to be removed from the CRIC configuration (thanks Ivan).

We had a goot run on the data challange, but we are waiting to hear from Hiro.