Ceph/CVMFS/Filer Service Meeting

600/R-001 (CERN)



Show room on map

Zoom: Ceph Zoom

    • 14:00 14:20
      Ceph: Operations Reports 20m
      • Teo (cta, erin, kelly, levinson) 5m
        Speaker: Theofilos Mouratidis (CERN)
        • mon leaking memory bug solved with workaround
          • remove obsolete config, they leak memory
        • cooperating with leduc and vlado to get cta machines for ceph
      • Enrico (barn, beesly, gabe, meredith, nethub, vault) 5m
        Speaker: Enrico Bocchi (CERN)


        • (from last week) Benchmarking and finalization for enrollment in OpenStack


        • Slowly removing OSDs on new HW
          • 1800 OSDs in crush -- PG map getting too big
          • Will remove 10x48 OSDs --> 1500 remain in crush
          • Hosts will remain in root=incoming for easier recreation
        • MGR struggles to export metrics to prometheus
          • Likely due to big PG map
          • `ceph config set mgr mgr/prometheus/scrape_interval 120`


        • (from last week) 2 nodes require reinstallation to get raid1 on system disk (CEPH-1045)
        • CEPH-1078: one rgw marked out for network switch replcaement
          • Now back. Thanks @Dan!

        Meredith, Nethub, Vault: NTR


        Discussion with OpenStack guys:

        • Embrace AZ model for Cinder volumes provisioning
        • One cluster missing (Oscar?) for "standard" volumes, then Beesly + Vault
        • IO volumes provided by Meredith (io2, io3) + Kelly (hyperc)
        • Barn will do the critical-power volumes
      • Dan (dwight, flax, kopano, jim) 5m
        Speaker: Dan van der Ster (CERN)
        • flax: slowness reported Thursday:
          • linuxsoft reported slow early in the day, around 9:20am on ~Ceph
          • Dan checked, no slow reqs. Increased MDS cache size around 9:40 from 4GB to 8GB.
          • Cephfs load plots don't show any correlated increases. None of the clients seemed exceptionally busy/active.
          • Wojciech reported all looks ok to him.
          • Dan started checking samba gw's around 2pm -- they were hammering an msg.sock dir with hundreds of socket files. Guisseppe cleaned that up over the afternoon -- not sure exactly when. After ?restarting samba? the msg.sock files were ~normal.
          • Around 17h00 dan moved JIRA and Webcast to mds.2 from mds.0 -- they are relatively metadata active, but not clear these were related to any slowness during the day.
          • My best theory is that the samba thrashing caused the mds.0 to be so busy that some md requests were slowed down.
          • Giuseppe is reviewing if msg.sock even needs to be on a shared fs. And he will move to Levinson.
        • CEPH-1068: dwight crush and peering issue (pgs undersized / clean depending on the order osds boot) -- reproduced the issue again with debug_osd = 20 and posted the logs to the tracker. during this testing I managed to break dwight for a couple minutes by making huge crush map changes in a short time.
        • CEPH-1078: one rgw marked out for network switch replcaement.
        • CEPH-1005: pg removal slowness. There is a new config to test this week on gabe for the pg removal issue. (Devs found that the rocksdb cache entries are not removed even after an index entry is deleted from rocksdb -- so eventually the effective rocksdb cache size drops to zero and this might explain why we see thrashing onthe ssds during removal).


        • FILER-140: filer-carbon moving to io2 volume -- all but one io1 volume has been zpool removed. No noticeable performance change, afaict.
    • 14:20 14:30
      Ceph: Operations Tools (ceph-scripts, puppet, monitoring, etc...) 10m
      • CEPH-1077: mds cache memory target is no longer in the ceph.conf -- you need to check that the `ceph config set mds..` is correct for each cluster. (Normally it should be the default, 4GB).
    • 14:30 14:40
      Ceph: R&D Projects Reports 10m
    • 14:40 14:50
      Ceph: Upstream News 10m
      • octopus 15.2.9 in final qa testing right now. release imminent, then they move to next nautilus.
        • 15.2.9 has merged all of the PRs that we need for next nautilus. But these two octopus issues are blocking a major version upgrade:
          • https://github.com/ceph/ceph/pull/39320
          • https://github.com/ceph/ceph/pull/39377
          • (Both related to upgrading ondisk format for huge omaps, e.g. seen on gabe/nethub/flax).
    • 14:50 15:05
      CVMFS 15m
      Speakers: Enrico Bocchi (CERN) , Fabrizio Furano (CERN)

      Network intervention this morning:

      • zero04 (no impact)
      • lxcvmfs152 (alpha, bbp.epfl.ch, clicbp, delphi, lhec) -- OTG0062354

      CVMFS PoW:

      • Slides at https://indico.cern.ch/event/1008722/
      • No big surprises for us: consolidation, publishing statistics with JSROOT, fix shortcomings of the gateway
      • 2.8 is out since few weeks, plan for 2.9 in Q4
    • 15:05 15:10
      AOB 5m
      • Teo will present the merkle tree work at Ceph Performance Weekly this Thursday at 5pm.