Ceph/CVMFS/Filer Service Meeting

600/R-001 (CERN)



Show room on map
    • 1
      Speaker: Enrico Bocchi (CERN)


      • Signing whitelist with YubiKey (OTG0053606)
      • Re-signed all the other repos
        • Safe until 2020
      • atlas-nightlies.cern.ch release manager as Stratum1 (CVMFSOPS-212)
        • Our squids
        • pic.es
        • hep.technion.ac.il
        • hep.tau.ac.il
        • ...
      • atlas.cern.ch migrated to new volume and new release manager
        • old RM (lxcvmfs69) and volume still available
      • cms-ib `cvmfs_server eliminate-hardlinks`: Done and confirmed successful
      • cms.cern.ch migration to CC7 on hold
        • `cvmfs_server eliminate-hardlinks` can still crash and cms has a huge catalog
      • ams.cern.ch unable to complete transaction due to root full
        • Met and discussed with Baosong
        • Not heard back from them since
    • 2
      Ceph Upstream News

      Releases, Tickets, Testing, Board, ...

      Speaker: Dan van der Ster (CERN)
      • 13.2.7 has a missing backport (see https://tracker.ceph.com/issues/38330). I build ceph-13.2.7-0.1 in koji with the missing patch. 13.2.8 includes the backport https://github.com/ceph/ceph/pull/32000 and should come out any day now.
      • 14.2.5 is imminent, which should be a nautilus release candidate for us. I don't foresee any upgrades before xmas, so we can profit from other users' experience with this release.
      • Bluestore compression: pool specific compression options are still not loaded at osd boot time, so only the global cluster options are usable. The backport of https://github.com/ceph/ceph/pull/29093 to mimic/nautilus is needed, but its quite non-trivial.
    • 3
      Ceph Backends & Block Storage

      Cluster upgrades, capacity changes, rebalancing, ...
      News from OpenStack block storage.

      Speaker: Theofilos Mouratidis (National and Kapodistrian University of Athens (GR))


      Beesly filestore to bluestore dangerous conversion

      converted two hosts, created a script for that along with Dan

      waiting for last to backfill and monitoring for any problems

    • 4
      Ceph Disk Management

      OSD Replacements, Liaison with CF, Failure Predictions

      Speaker: Julien Collet (CERN)


      • You may receive several emails from smartd -- I have pushed smartmontools v7.0 to the ceph machines, so this restarts the daemon and re-sends mails about pendingsectors etc.



      • Pushed a warning message to prevent recreation of OSD in beesly/critical
      • Resume work on "no-osd-left-behind"
        • Periodic checks of "available" drives to be put in production
        • Sending weekly report/update JIRA ticket?


    • 5

      Ops, Use-cases (backup, DB), ...

      Speakers: Julien Collet (CERN) , Roberto Valverde Cameselle (Universidad de Oviedo (ES))


      • S3 accounting finally finished!
        • Hugo to add the table in cern.ch/storage
        • Data stored daily in an s3-bucket
      • Kicad bucket about to overload
        • Owner was notified
        • After clean-up: dashboard
      • Elastic-search dashboard broken due to a change in templates
    • 6

      Filer Migration, CephFS/Manila, HPC status and plans.

      Speakers: Dan van der Ster (CERN) , Pablo Llopis Sanmillan (CERN)

      HPC guys wanted to cleanly power off ceph/jim in case of UPS -- decided not to.

    • 7
      Speakers: Jose Castro Leon (CERN) , Julien Collet (CERN) , Roberto Valverde Cameselle (Universidad de Oviedo (ES))


    • 8
    • 9