Ceph/CVMFS/Filer Service Meeting

600/R-001 (CERN)



Show room on map
    • 14:00 14:05
      CVMFS 5m
      Speaker: Enrico Bocchi (CERN)
      • Radu developed an S3 benchmark which simulates the CVMFS publish workload.
        • Goal is 1kHz HEAD/PUT. HEAD is currently around 2.5kHz, but PUT is around 600Hz.
        • We should shard their bucket (from 32->128 shards) and check if it has any impact.
        • Once ceph/gabe is converted to bluestore it will be better (all bucket indicies will be on the ssd block.db).
    • 14:05 14:10
      Ceph Upstream News 5m

      Releases, Tickets, Testing, Board, ...

      Speaker: Dan van der Ster (CERN)
    • 14:10 14:15
      Ceph Backends & Block Storage 5m

      Cluster upgrades, capacity changes, rebalancing, ...
      News from OpenStack block storage.

      Speaker: Theofilos Mouratidis (National and Kapodistrian University of Athens (GR))

      ceph/flax: 3rd host converted to bluestore, backfilling
      ceph/flax: balancer enabled to avoid osd backfill-full
      ceph: useless alerts should be less prevalent

    • 14:15 14:20
      Ceph Disk Management 5m

      OSD Replacements, Liaison with CF, Failure Predictions

      Speaker: Julien Collet (CERN)


      • Update in disk replacement procedure (stop of osd service after disk has been drained)
      • Remy/Paul ramping up with scsi tickets
    • 14:20 14:25
      S3 5m

      Ops, Use-cases (backup, DB), ...

      Speakers: Julien Collet (CERN), Roberto Valverde Cameselle (Universidad de Oviedo (ES))
      • Alarms for TLS certificate are now firing. Renewal request sent to Andreas Wagner last week.

      Backup (Roberto)



      • S3 clean-up in progress
      • Renewal of certificate w/Andreas
    • 14:25 14:30
      CephFS/Manila/FILER 5m

      Filer Migration, CephFS/Manila status and plans.

      Speaker: Dan van der Ster (CERN)
      • Kernel client bug: a writer loops on open O_APPEND, write, close; and tail -f from another client, can corrupt the written file. Fix available in centosplus kernel, Red Hat plans a fix in CentOS 7.6.z bugfixes kernels.
        • All ceph kernel users notified directly. Most are unaffected, though HPC is affected.
        • HPC pushed the ceph.ko to their machines, then remounted.
      • Upgrade dwight (Geneva CephFS Testing in Manila) to mimic tomorrow: https://cern.service-now.com/service-portal/view-outage.do?n=OTG0049128
    • 14:30 14:35
      HPC 5m

      Performance testing, HPC storage status and plans

      Speakers: Alberto Chiusole (Universita e INFN Trieste (IT)), Pablo Llopis Sanmillan (CERN)
      • Jim cluster - one mountpoint deadlocked during 1.95TiB ior test. Checking with ceph ML.
    • 14:35 14:40
      HyperConverged 5m
      Speakers: Jose Castro Leon (CERN), Julien Collet (CERN), Roberto Valverde Cameselle (Universidad de Oviedo (ES))
    • 14:40 14:45
      Monitoring 5m


      • Prophetstore:
        • Guys wants to have a whitepaper, we asked them to wait
        • We told them about the overly pessimistic failure predictions, did not receive a satisfying answers as of today...
    • 14:45 14:50
      AOB 5m