Ceph/CVMFS/Filer Service Meeting

Europe/Zurich
600/R-001 (CERN)

600/R-001

CERN

15
Show room on map
    • 14:00 14:05
      CVMFS 5m
      Speaker: Enrico Bocchi (CERN)

      Enrico:

      • Signing whitelist with YubiKey (OTG0053606)
      • Re-signed all the other repos
        • Safe until 2020
      • atlas-nightlies.cern.ch release manager as Stratum1 (CVMFSOPS-212)
        • Our squids
        • pic.es
        • hep.technion.ac.il
        • hep.tau.ac.il
        • ...
      • atlas.cern.ch migrated to new volume and new release manager
        • old RM (lxcvmfs69) and volume still available
      • cms-ib `cvmfs_server eliminate-hardlinks`: Done and confirmed successful
      • cms.cern.ch migration to CC7 on hold
        • `cvmfs_server eliminate-hardlinks` can still crash and cms has a huge catalog
      • ams.cern.ch unable to complete transaction due to root full
        • Met and discussed with Baosong
        • Not heard back from them since
    • 14:05 14:10
      Ceph Upstream News 5m

      Releases, Tickets, Testing, Board, ...

      Speaker: Dan van der Ster (CERN)
      • 13.2.7 has a missing backport (see https://tracker.ceph.com/issues/38330). I build ceph-13.2.7-0.1 in koji with the missing patch. 13.2.8 includes the backport https://github.com/ceph/ceph/pull/32000 and should come out any day now.
      • 14.2.5 is imminent, which should be a nautilus release candidate for us. I don't foresee any upgrades before xmas, so we can profit from other users' experience with this release.
      • Bluestore compression: pool specific compression options are still not loaded at osd boot time, so only the global cluster options are usable. The backport of https://github.com/ceph/ceph/pull/29093 to mimic/nautilus is needed, but its quite non-trivial.
    • 14:10 14:15
      Ceph Backends & Block Storage 5m

      Cluster upgrades, capacity changes, rebalancing, ...
      News from OpenStack block storage.

      Speaker: Theofilos Mouratidis (National and Kapodistrian University of Athens (GR))

      Theo:

      Beesly filestore to bluestore dangerous conversion

      converted two hosts, created a script for that along with Dan

      waiting for last to backfill and monitoring for any problems

    • 14:15 14:20
      Ceph Disk Management 5m

      OSD Replacements, Liaison with CF, Failure Predictions

      Speaker: Julien Collet (CERN)

      (Dan)

      • You may receive several emails from smartd -- I have pushed smartmontools v7.0 to the ceph machines, so this restarts the daemon and re-sends mails about pendingsectors etc.

       

      Giuliano

      • Pushed a warning message to prevent recreation of OSD in beesly/critical
      • Resume work on "no-osd-left-behind"
        • Periodic checks of "available" drives to be put in production
        • Sending weekly report/update JIRA ticket?

       

    • 14:20 14:25
      S3 5m

      Ops, Use-cases (backup, DB), ...

      Speakers: Julien Collet (CERN), Roberto Valverde Cameselle (Universidad de Oviedo (ES))

      Giuliano

      • S3 accounting finally finished!
        • Hugo to add the table in cern.ch/storage
        • Data stored daily in an s3-bucket
      • Kicad bucket about to overload
        • Owner was notified
        • After clean-up: dashboard
      • Elastic-search dashboard broken due to a change in templates
    • 14:25 14:30
      CephFS/HPC/FILER/Manila 5m

      Filer Migration, CephFS/Manila, HPC status and plans.

      Speakers: Dan van der Ster (CERN), Pablo Llopis Sanmillan (CERN)

      HPC guys wanted to cleanly power off ceph/jim in case of UPS -- decided not to.

    • 14:30 14:35
      HyperConverged 5m
      Speakers: Jose Castro Leon (CERN), Julien Collet (CERN), Roberto Valverde Cameselle (Universidad de Oviedo (ES))

      (Dan)

    • 14:35 14:40
      Monitoring 5m
    • 14:40 14:45
      AOB 5m