Ceph/CVMFS/Filer Service Meeting

600/R-001 (CERN)



Show room on map
    • 14:00 14:05
      CVMFS 5m
      Speaker: Enrico Bocchi (CERN)
      • Debugging // profiling atlas-nightlies.cern.ch (S3 + Gateway):
        • Long transaction: 937 packages, 17 GB
        • Many packages built // patched on the RM (ceph-fuse heavily spinning)
        • After 2h15, 648 pkgs out of 937 --> publishing not even started but transaction open
        • Lease time: 6h
      • Restarting lhcbdev-test on S3
      • Buggy cvmfs_server check with S3 storage (does not honor "429 Too Many Requests"): CVM-1736
      • Merge request for dedicated atlas squid proxies
    • 14:05 14:10
      Ceph Upstream News 5m

      Releases, Tickets, Testing, Board, ...

      Speaker: Dan van der Ster (CERN)
    • 14:10 14:15
      Ceph Backends & Block Storage 5m

      Cluster upgrades, capacity changes, rebalancing, ...
      News from OpenStack block storage.

      Speaker: Theofilos Mouratidis (National and Kapodistrian University of Athens (GR))


      • Setup devstack to test automatic manila snapshots
      • Helped Julien with lemon->collectd
      • Backfill optimization, almost done except:
        • Figure out how to fill each PG tree, now only primary
        • Figure out a persist scheme (interval, number of updates)
    • 14:15 14:20
      Ceph Disk Management 5m

      OSD Replacements, Liaison with CF, Failure Predictions

      Speaker: Julien Collet (CERN)


      • IT-CF successfully carried out their first disk replacement autonomously after last week's presentation
      • They will be fully operational within 2w
    • 14:20 14:25
      S3 5m

      Ops, Use-cases (backup, DB), ...

      Speakers: Julien Collet (CERN), Roberto Valverde Cameselle (Universidad de Oviedo (ES))


      • Nexus buckets are not deletable by end-users. INC1983576
        • CDA has configured the Nexus OpenShift app to enable object versioning, so even though users see a empty bucket, there are still thousands of hidden objects.
        • We can bucket rm --purge-objects, and I've asked Alex Lossent if Nexus has a better cleanup tool.


      • RGW VMs replaced by larger ones, now all in prod
      • Investigating an issue (apparently unable to retrieve bucket contents when over than 1000 objects)
      • CEPH-697: still in progress - deadline?
    • 14:25 14:30
      CephFS/HPC/FILER/Manila 5m

      Filer Migration, CephFS/Manila, HPC status and plans.

      Speakers: Dan van der Ster (CERN), Pablo Llopis Sanmillan (CERN)


    • 14:30 14:35
      HyperConverged 5m
      Speakers: Jose Castro Leon (CERN), Julien Collet (CERN), Roberto Valverde Cameselle (Universidad de Oviedo (ES))
      • Several Kopano load tests on the kelly cluster last 2 weeks. Even with highest load we only see ~5MBps, likely indicating the client cache on the Kopano server is effective.
      • Numbers estimate:
        • ~700 million mails in Exchange, roughly 2/3rds have an attachment.
        • Kopano shards to 200 attachment dirs, and we'll have 10 Kopano servers.
        • 500 million / 200 / 10 = 250,000 attachments per directory.
    • 14:35 14:40
      Monitoring 5m


      • Updated prometheus + alertmanager + blackbox exporter
    • 14:40 14:45
      AOB 5m
Your browser is out of date!

If you are using Internet Explorer, please use Firefox, Chrome or Edge instead.

Otherwise, please update your browser to the latest version to use Indico without problems.