CVMFS Sync Meeting

Friday 10 Apr 2026, 11:00 → 12:30 Europe/Zurich

31/S-023 (CERN)

1 Operational status & service overview

• Current CVMFS service health (repositories, stratum-0/1, monitoring)
• Any incidents since last meeting
• Short-term operational risks or alerts

The rota machinery says "unknown". Is it broken? How to fix it?

 

Highlight: first cvmfs machine in PDC

• needed an external volume (io2) to accommodate a decent cache size
• added option to puppet manifest to use an external volume
• puppet was not taking it into account for the size calculation, result was suboptimal hit ratio
• fixed in the yaml by specifying the size directly

https://filer-carbon.cern.ch/grafana/d/QR2YUdh7k/cvmfs-status-by-host?orgId=1&viewPanel=15

NOTE: if all is totally silent for 1 week then I will start removing some older S1 caches

NOTE2: the shrinking of the local disk seems to be a constant trend for newer VMs. I think one day we may have to automate the external volume creation and maintenance (painful)

Poll: what if we eliminate the disk full check on the release managers?
Hugo: print a login banner in the case of disk full
Fab: may be a good idea!

 

2 Issues & interventions follow-up

• Review of open issues
• Status of ongoing or planned interventions
• Decisions needed / actions agreed

@hugo, any progress on the two tickets? I updated one yesterday. Plz let me know if I have to take it

RQF3619820
RQF3639647

 

 

3 Onboarding & knowledge transfer

• Onboarding progress for Hugo and Maria

• Open questions or unclear areas

• Documentation gaps or runbooks to improve

• Identify topics needing deeper walkthroughs (future sessions)

Progress on Hugo's welcome ticket?

RQF3555480

4 Collaborator input – SFT / Valentin

• Updates from SFT side

• Cross-team dependencies or changes impacting CVMFS

• Technical discussions or requests for support

• Upcoming developments worth tracking

Any progress with the steps we need?

Proposals so far
• Make it possible to have an lxplus-like setup for
the RMs
• All identical machines for everyone
• No special user identities to login, just krb
• Make it possible to scale up the CVMFS
gateways and make them fault tolerant
• Goal: 5 gateways, identical, shared by all
repos

Valentin: no progress so far, as the priority is the emergency with unpacked
progress on unpacked. Reduced a lot the size of the catalog, checks should be much faster now. Close to 50% in theory
Hypothesis: we may be leaking objects on S3 because they seem to be too many. Valentin is working with Enrico to check for unreferenced objects
Fab: how to run a check now that we don't have it anymore in the local S1 disks?

https://indico.cern.ch/event/1648544/contributions/6928662/attachments/3246827/5793026/CVMFSthoughts.pdf

 

5 Planning & priorities

• Short-term priorities until next sync

• Upcoming milestones or expected changes

• Topics to follow up

6 AOB