Ceph/CVMFS/Filer Service Meeting

600/R-001 (CERN)



Show room on map

Zoom: Ceph Zoom

    • 2:00 PM 2:20 PM
      Ceph: Operations Reports 20m
      • Teo (cta, erin, kelly, levinson) 5m
        Speaker: Theofilos Mouratidis (CERN)
        • upgraded cephcta to centos 8
        • plan with julien to split ceph-kelly into dwight and cta
          • we start with the dev part, since it can be done automatically
      • Enrico (barn, beesly, gabe, meredith, nethub, vault) 5m
        Speaker: Enrico Bocchi (CERN)

        Barn, Beesly, Meredith, Nethub, Vault:

        • All running 14.2.20-2 since ~10 days


        • Many disks to be replaced // OSDs to be recreated. Will take care in the afternoon.
        • Found another case of LVM bug on the block-db. Now draining the osd to then re-create properly.


        • Last one on 14.2.11. Will update to 14.2.21-1 with the security patches (and progress bar fix) asap
        • Draining of old machines is RJs is over. Some in RJ4* are used by Arthur for testing (see CEPH-972). No need to rush for decommissioning.
        • Discussion with MONIT to review logs ingestion: CEPH-1132
          • Suggest to wait for comments from security team to understand requirements
        • New S3 accounting is ready and tested. Will switch today to "prod".
      • Dan (dwight, flax, kopano, jim) 5m
        Speaker: Dan van der Ster (CERN)
        • CEPH-1145: checked all clusters and hw types for zombie spanning blobs. Beesly has a few, but other clusters have zero. Gabe still TODO after the v21 upgrade.
      • Arthur 5m
        Speaker: Arthur Outhenin-Chalandre (CERN)

        Still a few fixes to do on ceph rpm CI for building a release, should be fine now!

    • 2:20 PM 2:30 PM
      Ceph: Operations Tools (ceph-scripts, puppet, monitoring, etc...) 10m
    • 2:30 PM 2:40 PM
      Ceph: R&D Projects Reports 10m
      • Reva/CephFS 5m
        Speaker: Theofilos Mouratidis (CERN)
        • Implemented file versions (revisions)
          • ListRevisions, DownloadRevision, RestoreRevision
          • They use snapshots v2
          • We have some nice wrappers that create snapshots for the subvolumes for us
            • implemented in golang as well
          • They still work the way from when they were introduced
            • .snap from user root has exactly the snapshot name
            • .snap from subdirs have the _<snapshot name>_<user root ino>
            • added user ino to the connections cache to avoid listing all snapshots and grepping for name, since the mount starts from the uuid4 directory after user root
      • Disaster Recovery 5m
        Speaker: Arthur Outhenin-Chalandre (CERN)
        • Done some slide for RBD replication
        • The additional SSD on my test cluster did not really helped the journaling performance after all...
          • Mirror snapshots will definitely be my main target now
        • Did some test with many image to be replicated
          • startup times seems suuuuper slow for 1k images after a restart
        • rbd-mirror logs have a few errors that is commonly printed but not super useful/normal (i.e.: trying to create the image locally but it already exists)
          • Will probably try to come up with fixes for stuff like that, should help bug tracking and prod in the future...
    • 2:40 PM 2:50 PM
      Ceph: Upstream News 10m
      • v14.2.21 released for some dashboard and SWIFT/S3 CVEs. 14.2.21-1 building in koji with mon osdmap and progress patches.
    • 2:50 PM 3:05 PM
      CVMFS 15m
      Speakers: Enrico Bocchi (CERN) , Fabrizio Furano (CERN)


      • From devs about the upgrade of gateways:
        "On the gateway nodes and the publisher nodes, packages should be updated to cvmfs-gateway-1.2.0 and cvmfs-2.8.1 and cvmfs-server-2.8.1"


      Jan asked about feeding an IT accounting infrastructure. So far it's unclear what numbers will be needed to evaluate the cost of the cvmfs service, and how to produce them


    • 3:05 PM 3:10 PM
      AOB 5m