Ceph/CVMFS/Filer Service Meeting

600/R-001 (CERN)



Show room on map

Zoom: Ceph Zoom

    • 14:00 14:15
      CVMFS 15m
      Speakers: Enrico Bocchi (CERN) , Fabrizio Furano (CERN)
      • Network intervention Oct 05, 2020 09:00:
        • Release managers -- OTG0059292
        • All zeros and ones on LCG network
        • 2 caches
      • Network issues on 24/09 at around 15:40?
      • Migrations:
        • sft + sft-nightlies on 08/10 (OTG0059306, OTG0059307)
        • Ready for ams, compass, atlas-nightlies, lhcbdev
        • Missing cms, cms-ib
    • 14:15 14:30
      Ceph: Operations 15m
      • Incidents, Requests, Capacity Planning 5m
        Speaker: Dan van der Ster (CERN)
        • News from CFCCM:
          • servers in RJ+BA+EC rows to be decommissioned early 2021.
          • servers in RA not on the list yet: conclusion, migrate beesly fully to bluestore. (CEPH-966)
      • Cluster Upgrades, Migrations 5m
        Speaker: Theofilos Mouratidis (CERN)
        • gabe upgrade to nautilus went well, small config issues reported in CEPH-858
          • It slightly improved the gitlab bucket listing latency, but we will need another optimization to fully resolve this. (filter common delimiters in the cls_rgw on the osd)
        • flax mds's have all been replaced. upgrade to nautilus can be scheduled.
      • Hardware Repairs 5m
        Speaker: Julien Collet (CERN)


        • New beesly procedures in place
          • we have 3 osds ready to be recreated
          • will try to have them recreated when we convert the host to bs
      • Puppet and Tools 5m
    • 14:30 14:45
      Ceph: Projects, News, Other 15m
      • Kopano/Dovecot 5m
        Speaker: Dan van der Ster (CERN)
      • REVA/CephFS 5m
        Speaker: Theofilos Mouratidis (CERN)

        made smashbox tests work

    • 14:45 14:55
      S3 10m
      Speakers: Julien Collet (CERN) , Roberto Valverde Cameselle (CERN)
      • CEPH-967: mattermost ran out of quota. Can we revive the quota emails?
      • CEPH-970: some PGs have not been scrubbed for a few months, due osd_max_scrubs=1 and too much contention. Increased to osd_max_scrubs=3 and watching. See ceph-scripts/tools/scrubbing/ceph-scrub-summary for some debugging on this.
      • CEPH-965: gitlab s3 latency. We can leave it if the users are happy, or re-shard to fewer shards to bring back some perf.


      • CEPH-916: regular warp benchmark running against gabe/nethub on the S3 dashboard
        • running every 2hr, put/get/stat/delete.
        • there seem to be a slight performance increase for s3.cern.ch since last thursday ...
      • CEPH-949: removed half of the empty personal accounts, disabled the other half
    • 14:55 15:05
      Filer/CephFS 10m
      Speakers: Dan van der Ster (CERN) , Theofilos Mouratidis (CERN)
      • Network intervention (07/10) affecting itnfs30 (openshift-dev-appdata, openshift-dev-registry). OTG?
    • 15:05 15:10
      AOB 5m