CVMFS Sync Meeting

Friday Mar 13, 2026, 11:00 AM → 12:30 PM Europe/Zurich

31/S-023 (CERN)

11:00 AM → 11:10 AM Operational status & service overview

• Current CVMFS service health (repositories, stratum-0/1, monitoring)
• Any incidents since last meeting
• Short-term operational risks or alerts

TBD since last meeting. Good exercise for Hugo/Maria?

Fab should request the quota, then we see.

------------

Got contacted by Luis about PDC. Ideas?

Fabrizio, we have made available some capacity in PDC fort LCG virtual machines (follow-up from an incident in MDC that impacted the only LCG area there).

I see IT CVMFS - LCG is one of the projects with access to those resources.

If you're interested, you can give it a try to spawn some LCG instances in PDC to add some resiliency in case any issue

If you want to start uisng it, you should request quota for the PDC region in that project, we can follow it up from there.

 

 

Conclusion:

S1 will have:
2 ironic in MDC
1 VM in PDC
1 VM in MDC
(ALL in the LCG network)

 

S0 will have:
2 VMs in MDC
2 VMs in PDC
(ALL in the LCG network)

S1-hpc will have
1VM in PDC
1VM in MDC

(ALL in the LCG network)

11:10 AM → 11:20 AM Issues & interventions follow-up

• Review of open issues
• Status of ongoing or planned interventions
• Decisions needed / actions agreed

OF the many things, just one needs real attention:

I changed the swappiness first to 10... boom! Now it's one and the memory is half. IMO squid leaks memory!

 

INC4884548 i can't understand why only this machine gives this issue, and not its sister... to me this makes little sense, as it is using swap with plenty of available ram
MiB Mem :  63772.9 total,    665.7 free,  37414.9 used,  26348.4 buff/cache
MiB Swap:   8192.0 total,     36.7 free,   8155.3 used.  26358.0 avail Mem

... could it be this?

To check the swappiness value

cat /proc/sys/vm/swappiness

To change the swappiness value A temporary change (lost on reboot) with a swappiness value of 10 can be made with

sudo sysctl vm.swappiness=10

11:20 AM → 11:30 AM Onboarding & knowledge transfer

• Onboarding progress for Hugo and Maria

• Open questions or unclear areas

• Documentation gaps or runbooks to improve

• Identify topics needing deeper walkthroughs (future sessions)

11:30 AM → 11:40 AM Collaborator input – SFT / Valentin

• Updates from SFT side

• Cross-team dependencies or changes impacting CVMFS

• Technical discussions or requests for support

• Upcoming developments worth tracking

New replication monitor prototype

Old replication monitor

• unpacked
• DESY
• new replication-monitor prototype: http://cvm-monit02:8080
• GSoC
• preparation of 2.14.0 pre-release

 

some improvements in unpacked to remove the cache called"chains". 90gb less apparently. 2Mobjects, 1.5TB gained.

Valentin met with Ricardo to discuss harbor, registry.cern.ch and replicate from it to unpacked

11:40 AM → 11:45 AM Planning & priorities

• Short-term priorities until next sync

• Upcoming milestones or expected changes

• Topics to follow up

Brainstorming on a possible future architecture, for which:

 

• publishers should be a proper cluster, all identical
• users would be allowed to publih using their own credentials (instead than the "cvmfs user")
• gateways can be replicated identical (Valentin calls this "high availability gateway")
•  

11:45 AM → 11:50 AM AOB