CVMFS Operations Meeting, October 2, 2023
Attending: Soeren Fleischer, Fabrizio Furano, Dave Dykstra, Kenneth
Hoste, Laura, Valentin Volki, Carl Vuosalo
Valentin: First item is packaging issues on EL9. It's a failure of a DNF
check. It seems deprecated. It is due to a package dependency. The
server package depends on the init script package from CentOS 7. It
is a legacy package in EL9.
Fabrizio: The reason why it chokes is because the init scripts depends
upon another package. The package script can fail and leave things in
a strange state even after a successful install. The dnf check fails.
The solution is from Redhat discussed in Mattermost. See
https://access.redhat.com/solutions/6969215. The solution is to do
"rm -fr /etc/init.d" and reinstall chkconfig.
Dave: Are you installing from scratch?
Fabrizio: Yes.
Valentin: I tried it on a fresh machine and did not see this problem.
Dave: It is probably not needed anymore because this requirement came
from EL6. We could do this a different way.
Valentin: Should we agree to remove this for EL9?
We will test this, but it is probably only used for legacy
applications, so it can probably be removed.
Dave: If CVMFS has an /etc/init.d file that is being used, it can't be
removed.
Valentin: I checked the GitHub history of CVMFS, and this was already
removed for Debian. It was only needed for Redhat.
This change can be submitted in the patch release in 2.11.1. We also
need a fix for the unresponsive CVMFS client. We will try publish the
first fix as soon as possible and then do another patch release.
Dave: I'm not optimistic about a quick fix for the hanging issue.
Valentin: We'll definitely do a patch release soon. Please send email if
anything else is needed in the patch release.
Dave: On the hanging issue, we are trying to set up a test
environment. We have set aside a Fermilab dCache area that we can
overload. We would like to use http, But there is a blanket government
directive to use https instead of http. We are requesting an exception
to the directive. Another option is an XRootD server that doesn't
need to be unencrypted. Even after reproducing the bug, how to diagnose
and fix it? It just hangs.
Valentin: We could try installing custom builds that would help in
diagnosis.
Dave: This problem hasn't gotten to the testing stage, but we are
working on it.
Soeren: When the CVMFS client hangs, is it possible to recover? A user
reported that a CVMFS repo was not responding. I couldn't get into the
repo, so I rebooted the machine.
Laura: cvmfs_config killall should clean up all repos. wipecache might
also help.
Soeren: In this case, one repo was completely inaccessible.
Laura: cvmfs_config reload could also be tried.
Dave: You might want to do a ps, find the hung process, and kill it
first to avoid killall.
Please first run "cvmfs_config bugreport" to save info so we can
diagnose this problem.
Another topic: There is progress on the /opt campaign. We have
identified which sites are using it. The WLCG Squid Ops team can tell
from the monitor which IP addresses are requesting /opt. It's
around 15-20 sites, and the team is ready to start submitting tickets.
The CMS SAM test also shows this issue. It prints out each Stratum 1 that
it is probing. Some sites have good CMS SAM test results but still are
making the requests from some other cluster at the site. It may
require reloads or reboots to use the fixed configuration.
The CMS CVMFS main repo (cms.cern.ch) on the Stratum 0 is 19 TB, but on the
Stratum 1 it is 10 TB. A fresh snapshot is needed. Create a new snapshot in
parallel and then unmount and move it into place and delete the old one. It
should be cleaned up every few years.
Fabrizio: I will talk to Andrea.
Dave: A snapshot might take a day or two, but once it is done the
switch over takes a few minutes. Stratum 1s usually have fixed sized disks
so when there is plenty of disk space, we often don't bother to take fresh snapshots,
but when we get close to filling up, we do. The CMS repo is the biggest. There are
requests to make it bigger, which I recommend against.
Valentin: The cms-griddata repo is used only by generators. In the end, an
additional repo is not so much overhead.
Dave: There is setup overhead. The cms-griddata repo seems like a good option
for the additional analysis data that another group of CMS people want to add.
There were expired whitelists at DESY and RAL. The DESY repositories are
expired at their origin, but it's not clear whether they have been fixed today.
There are also three from Fermilab that are expired only at RAL so the problem
for those is at RAL.
cvmfs-stratum-operations is a good mailing list to keep track of these
issues.
Valentin: We will continue to investigate the hanging issue.
The next meeting will be November 13.