RAL Tier1 Experiments Liaison Meeting

Wednesday 18 Dec 2019, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

13:30 → 14:30 CMS PG Down

Speakers: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB)), Dan van der Ster (CERN), Rob Appleyard (STFC)

Logfiles

http://www-public.gridpp.rl.ac.uk/tape_accounting/ceph-osd.log.today.tar.gz

http://www-public.gridpp.rl.ac.uk/tape_accounting/ceph-osd.3849.log.1.gz

http://www-public.gridpp.rl.ac.uk/tape_accounting/ceph-osd.2162.log.1.gz

 

[root@ceph-adm1 ~]# ceph status
  cluster:
    id:     2336dcf1-b1b1-4c6a-b793-61cfb29f014c
    health: HEALTH_ERR
            noscrub,nodeep-scrub flag(s) set
            14300646/4241134807 objects misplaced (0.337%)
            22 scrub errors
            Reduced data availability: 1 pg inactive, 1 pg down
            Possible data damage: 20 pgs inconsistent
            Degraded data redundancy: 24074/4241134807 objects degraded (0.001%), 19 pgs degraded, 1 pg undersized
            Degraded data redundancy (low space): 1 pg backfill_toofull
            mons ceph-mon1,ceph-mon2,ceph-mon3,ceph-mon4,ceph-mon5 are using a lot of disk space

  services:
    mon: 5 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3,ceph-mon4,ceph-mon5
    mgr: ceph-mon3(active), standbys: ceph-mon2, ceph-mon5, ceph-mon1, ceph-mon4
    osd: 5412 osds: 5236 up, 5236 in; 581 remapped pgs
         flags noscrub,nodeep-scrub
    rgw: 8 daemons active

  data:
    pools:   22 pools, 25088 pgs
    objects: 388.91M objects, 19.9PiB
    usage:   27.8PiB used, 13.2PiB / 40.9PiB avail
    pgs:     0.004% pgs not active
             24074/4241134807 objects degraded (0.001%)
             14300646/4241134807 objects misplaced (0.337%)
             24384 active+clean
             515   active+remapped+backfill_wait
             74    active+clean+scrubbing+deep
             60    active+remapped+backfilling
             18    active+clean+inconsistent
             17    active+recovery_wait+degraded
             9     active+recovery_wait
             5     active+clean+scrubbing
             2     active+remapped+inconsistent+backfill_wait
             1     active+undersized+degraded+remapped+backfill_wait
             1     down+remapped
             1     active+recovery_wait+degraded+remapped
             1     active+remapped+backfill_wait+backfill_toofull

  io:
    client:   20.0GiB/s rd, 1.45GiB/s wr, 14.29kop/s rd, 4.84kop/s wr
    recovery: 7.88GiB/s, 144objects/s

 

 

[root@ceph-adm1 ~]# ceph health detail
HEALTH_ERR noscrub,nodeep-scrub flag(s) set; 14296173/4241130586 objects misplaced (0.337%); 22 scrub errors; Reduced data availability: 1 pg inactive, 1 pg down; Possible data damage: 20 pgs inconsistent; Degraded data redundancy: 24074/4241130586 objects degraded (0.001%), 19 pgs degraded, 1 pg undersized; Degraded data redundancy (low space): 1 pg backfill_toofull; mons ceph-mon1,ceph-mon2,ceph-mon3,ceph-mon4,ceph-mon5 are using a lot of disk space
OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set
OBJECT_MISPLACED 14296173/4241130586 objects misplaced (0.337%)
OSD_SCRUB_ERRORS 22 scrub errors
PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg down
    pg 2.e2b is down+remapped, acting [2147483647,2147483647,2147483647,3903,4090,3811,40,4966,1346,323,4078]
PG_DAMAGED Possible data damage: 20 pgs inconsistent
    pg 1.361 is active+clean+inconsistent, acting [420,395,4634,1691,607,4023,1259,1579,534,2423,1638]
    pg 1.369 is active+clean+inconsistent, acting [2956,5081,4600,2232,5096,3772,5127,4901,3307,2230,2664]
    pg 1.692 is active+clean+inconsistent, acting [5027,5005,1023,4687,4571,858,114,1106,3589,2455,4454]
    pg 1.6a2 is active+clean+inconsistent, acting [4293,2952,2253,3361,3430,4606,5321,4464,4253,207,3228]
    pg 1.d19 is active+clean+inconsistent, acting [5188,2426,419,5280,5134,1051,5162,626,211,2073,1704]
    pg 1.eaa is active+clean+inconsistent, acting [3724,3519,4665,1898,3243,3061,4472,1211,2611,4311,4611]
    pg 1.10fa is active+clean+inconsistent, acting [556,2656,4824,1392,2164,3639,3593,5347,2938,3617,3790]
    pg 1.1486 is active+clean+inconsistent, acting [3086,4310,5356,5400,3970,1202,2073,3263,2963,1199,1383]
    pg 1.16b5 is active+clean+inconsistent, acting [1623,4654,4264,408,3765,4307,145,3082,4936,1702,2249]
    pg 1.1768 is active+clean+inconsistent, acting [3571,2453,987,4724,4341,2779,886,660,3904,1302,4570]
    pg 1.192c is active+clean+inconsistent, acting [3184,4068,3811,1489,962,1312,1558,3576,4063,2523,2917]
    pg 1.1cbc is active+clean+inconsistent, acting [3881,1211,3860,5390,1474,2651,3577,2872,2947,4396,5129]
    pg 2.3e8 is active+clean+inconsistent, acting [785,4917,1837,4904,5410,1253,5238,1167,4683,227,5261]
    pg 2.6f2 is active+remapped+inconsistent+backfill_wait, acting [4255,4832,2854,4593,5179,1146,3536,4787,2933,2192,1089]
    pg 2.905 is active+clean+inconsistent, acting [2772,2478,736,1177,3507,3314,3901,1746,4619,4326,3971]
    pg 2.994 is active+clean+inconsistent, acting [4544,1211,2836,3864,5330,3672,5159,4980,108,2187,3333]
    pg 2.adc is active+clean+inconsistent, acting [4324,5022,1592,2123,93,5106,3302,937,5237,2041,1741]
    pg 2.dba is active+clean+inconsistent, acting [3866,4914,3717,4457,303,479,470,4593,2997,4024,5357]
    pg 2.ded is active+clean+inconsistent, acting [4949,3836,1537,337,4679,3742,95,4402,3480,1383,2143]
    pg 2.ffa is active+remapped+inconsistent+backfill_wait, acting [1542,887,3648,3956,95,175,171,4484,1801,7,2793]
PG_DEGRADED Degraded data redundancy: 24074/4241130586 objects degraded (0.001%), 19 pgs degraded, 1 pg undersized
    pg 1.17a is active+recovery_wait+degraded, acting [4226,5367,2320,2089,3463,789,1588,2945,821,3987,3944]
    pg 1.3b7 is active+recovery_wait+degraded, acting [3218,14,3696,486,1339,3987,2773,5020,507,945,2792]
    pg 1.40c is active+recovery_wait+degraded, acting [175,4259,19,876,2041,216,5381,3987,3567,1490,611]
    pg 1.be3 is active+recovery_wait+degraded, acting [4676,5184,5132,3620,3470,4437,4187,2762,4414,3667,4688]
    pg 1.edf is stuck undersized for 19850.746619, current state active+undersized+degraded+remapped+backfill_wait, last acting [4926,2313,2495,1078,2265,18,3253,1611,2147483647,2408,632]
    pg 1.128a is active+recovery_wait+degraded, acting [4912,661,1749,5076,3987,3579,2964,4902,4997,3312,1685]
    pg 1.1392 is active+recovery_wait+degraded, acting [4433,2704,3282,4340,5170,433,4515,3987,1839,4195,1596]
    pg 1.1a31 is active+recovery_wait+degraded, acting [3266,3987,3603,3802,3788,3541,2108,4402,2952,2985,3178]
    pg 1.1d19 is active+recovery_wait+degraded, acting [4449,3177,3752,5092,5196,5067,3987,4776,3604,3410,4412]
    pg 1.1d9b is active+recovery_wait+degraded, acting [4589,2503,453,4663,949,3725,5226,4261,5120,3901,4503]
    pg 1.1e57 is active+recovery_wait+degraded, acting [2935,2912,3987,2364,3396,5153,4651,4346,3646,3963,2404]
    pg 2.3b8 is active+recovery_wait+degraded, acting [4410,2767,3652,4022,1125,5108,1820,585,3987,3594,4687]
    pg 2.542 is active+recovery_wait+degraded, acting [4676,4525,1570,4401,3508,1403,3580,1519,3752,1703,758]
    pg 2.e75 is active+recovery_wait+degraded, acting [3923,3065,2055,3381,319,5184,4414,4114,1347,2564,2995]
    pg 3.e1c is active+recovery_wait+degraded+remapped, acting [4112,388,4453,4414,93,4083,4431,3500,4173,2221,644]
    pg 3.ecc is active+recovery_wait+degraded, acting [5019,73,2639,3901,796,4445,4157,751,2841,1608,2580]
    pg 3.14b2 is active+recovery_wait+degraded, acting [1569,4829,792,5401,4997,1846,927,3987,3319,4145,1098]
    pg 3.16db is active+recovery_wait+degraded, acting [4404,277,1555,160,3733,4434,2107,3394,3901,3670,11]
    pg 3.1979 is active+recovery_wait+degraded, acting [981,759,3987,2394,4186,1506,4314,1524,452,2622,1699]
PG_DEGRADED_FULL Degraded data redundancy (low space): 1 pg backfill_toofull
    pg 2.209 is active+remapped+backfill_wait+backfill_toofull, acting [3899,839,3377,403,3618,366,2231,223,4244,4390,3315]
MON_DISK_BIG mons ceph-mon1,ceph-mon2,ceph-mon3,ceph-mon4,ceph-mon5 are using a lot of disk space
    mon.ceph-mon1 is 59.3GiB >= mon_data_size_warn (15GiB)
    mon.ceph-mon2 is 59.3GiB >= mon_data_size_warn (15GiB)
    mon.ceph-mon3 is 59.3GiB >= mon_data_size_warn (15GiB)
    mon.ceph-mon4 is 59.3GiB >= mon_data_size_warn (15GiB)
    mon.ceph-mon5 is 59.3GiB >= mon_data_size_warn (15GiB)

 

 

 

 

 

[root@ceph-adm1 ~]# ceph osd tree down
ID   CLASS WEIGHT      TYPE NAME            STATUS    REWEIGHT PRI-AFF
  -1       42938.10156 root default
-265         239.05200     host ceph-sn1016
3701   hdd    10.86600         osd.3701     destroyed        0 1.00000
-285         239.05200     host ceph-sn1026
3922   hdd    10.86600         osd.3922     destroyed        0 1.00000
-291         239.05200     host ceph-sn1030
3989   hdd    10.86600         osd.3989     destroyed        0 1.00000
 -43         196.48801     host ceph-sn834
1505   hdd     5.45799         osd.1505     destroyed        0 1.00000
  -3         196.48700     host ceph-sn835
  36   hdd     5.45799         osd.36       destroyed        0 1.00000
 -20         196.48801     host ceph-sn836
 715   hdd     5.45799         osd.715      destroyed        0 1.00000
 -17         185.57201     host ceph-sn837
 179   hdd     5.45799         osd.179      destroyed        0 1.00000
 206   hdd     5.45799         osd.206      destroyed        0 1.00000
 236   hdd     5.45799         osd.236      destroyed        0 1.00000
 267   hdd     5.45799         osd.267      destroyed        0 1.00000
 322   hdd     5.45799         osd.322      destroyed        0 1.00000
1008   hdd     5.45799         osd.1008     destroyed        0 1.00000
1066   hdd     5.45799         osd.1066     destroyed        0 1.00000
3222   hdd     5.45799         osd.3222     destroyed        0 1.00000
 -33         196.48801     host ceph-sn838
1161   hdd     5.45799         osd.1161     destroyed        0 1.00000
1229   hdd     5.45799         osd.1229     destroyed        0 1.00000
 -27         191.03000     host ceph-sn840
 190   hdd     5.45799         osd.190      destroyed        0 1.00000
  -4         196.48801     host ceph-sn842
 189   hdd     5.45799         osd.189      destroyed        0 1.00000
  -6         191.03000     host ceph-sn843
 118   hdd     5.45799         osd.118      destroyed        0 1.00000
 271   hdd     5.45799         osd.271      destroyed        0 1.00000
 300   hdd     5.45799         osd.300      destroyed        0 1.00000
 417   hdd     5.45799         osd.417      destroyed        0 1.00000
 -12         196.48801     host ceph-sn845
 229   hdd     5.45799         osd.229      destroyed        0 1.00000
 628   hdd     5.45799         osd.628      destroyed        0 1.00000
 713   hdd     5.45799         osd.713      destroyed        0 1.00000
 830   hdd     5.45799         osd.830      destroyed        0 1.00000
 -45         147.36600     host ceph-sn846
1562   hdd     5.45799         osd.1562     destroyed        0 1.00000
-365         196.48801     host ceph-sn847
4821   hdd     5.45799         osd.4821     destroyed        0 1.00000
 -23         185.57201     host ceph-sn848
  57   hdd     5.45799         osd.57       destroyed        0 1.00000
 320   hdd     5.45799         osd.320      destroyed        0 1.00000
 530   hdd     5.45799         osd.530      destroyed        0 1.00000
 559   hdd     5.45799         osd.559      destroyed        0 1.00000
 613   hdd     5.45799         osd.613      destroyed        0 1.00000
 645   hdd     5.45799         osd.645      destroyed        0 1.00000
 704   hdd     5.45799         osd.704      destroyed        0 1.00000
 756   hdd     5.45799         osd.756      destroyed        0 1.00000
 816   hdd     5.45799         osd.816      destroyed        0 1.00000
 846   hdd     5.45799         osd.846      destroyed        0 1.00000
 875   hdd     5.45799         osd.875      destroyed        0 1.00000
 905   hdd     5.45799         osd.905      destroyed        0 1.00000
 930   hdd     5.45799         osd.930      destroyed        0 1.00000
 959   hdd     5.45799         osd.959      destroyed        0 1.00000
 -19         196.48801     host ceph-sn849
1007   hdd     5.45799         osd.1007     destroyed        0 1.00000
 -47         109.16000     host ceph-sn851
1620   hdd     5.45799         osd.1620     destroyed        0 1.00000
1630   hdd     5.45799         osd.1630     destroyed        0 1.00000
1640   hdd     5.45799         osd.1640     destroyed        0 1.00000
1645   hdd     5.45799         osd.1645     destroyed        0 1.00000
1648   hdd     5.45799         osd.1648     destroyed        0 1.00000
1651   hdd     5.45799         osd.1651     destroyed        0 1.00000
1652   hdd     5.45799         osd.1652     destroyed        0 1.00000
1653   hdd     5.45799         osd.1653     destroyed        0 1.00000
  -5         185.57201     host ceph-sn852
  58   hdd     5.45799         osd.58       destroyed        0 1.00000
 142   hdd     5.45799         osd.142      destroyed        0 1.00000
 194   hdd     5.45799         osd.194      destroyed        0 1.00000
 245   hdd     5.45799         osd.245      destroyed        0 1.00000
 423   hdd     5.45799         osd.423      destroyed        0 1.00000
 485   hdd     5.45799         osd.485      destroyed        0 1.00000
 515   hdd     5.45799         osd.515      destroyed        0 1.00000
 545   hdd     5.45799         osd.545      destroyed        0 1.00000
 595   hdd     5.45799         osd.595      destroyed        0 1.00000
 795   hdd     5.45799         osd.795      destroyed        0 1.00000
 943   hdd     5.45799         osd.943      destroyed        0 1.00000
1025   hdd     5.45799         osd.1025     destroyed        0 1.00000
 -18         185.57201     host ceph-sn853
  64   hdd     5.45799         osd.64       destroyed        0 1.00000
 120   hdd     5.45799         osd.120      destroyed        0 1.00000
 203   hdd     5.45799         osd.203      destroyed        0 1.00000
 311   hdd     5.45799         osd.311      destroyed        0 1.00000
 345   hdd     5.45799         osd.345      destroyed        0 1.00000
 476   hdd     5.45799         osd.476      destroyed        0 1.00000
 594   hdd     5.45799         osd.594      destroyed        0 1.00000
 624   hdd     5.45799         osd.624      destroyed        0 1.00000
 655   hdd     5.45799         osd.655      destroyed        0 1.00000
 685   hdd     5.45799         osd.685      destroyed        0 1.00000
 803   hdd     5.45799         osd.803      destroyed        0 1.00000
 865   hdd     5.45799         osd.865      destroyed        0 1.00000
 921   hdd     5.45799         osd.921      destroyed        0 1.00000
  -7         169.19800     host ceph-sn854
 128   hdd     5.45799         osd.128      destroyed        0 1.00000
 162   hdd     5.45799         osd.162      destroyed        0 1.00000
 390   hdd     5.45799         osd.390      destroyed        0 1.00000
 451   hdd     5.45799         osd.451      destroyed        0 1.00000
 475   hdd     5.45799         osd.475      destroyed        0 1.00000
 561   hdd     5.45799         osd.561      destroyed        0 1.00000
 619   hdd     5.45799         osd.619      destroyed        0 1.00000
 793   hdd     5.45799         osd.793      destroyed        0 1.00000
 883   hdd     5.45799         osd.883      destroyed        0 1.00000
1002   hdd     5.45799         osd.1002     destroyed        0 1.00000
1026   hdd     5.45799         osd.1026     destroyed        0 1.00000
1053   hdd     5.45799         osd.1053     destroyed        0 1.00000
 -48         196.48801     host ceph-sn855
1656   hdd     5.45799         osd.1656     destroyed        0 1.00000
1672   hdd     5.45799         osd.1672     destroyed        0 1.00000
1675   hdd     5.45799         osd.1675     destroyed        0 1.00000
1676   hdd     5.45799         osd.1676     destroyed        0 1.00000
1683   hdd     5.45799         osd.1683     destroyed        0 1.00000
 -14         163.74001     host ceph-sn856
 182   hdd     5.45799         osd.182      destroyed        0 1.00000
 318   hdd     5.45799         osd.318      destroyed        0 1.00000
 405   hdd     5.45799         osd.405      destroyed        0 1.00000
 744   hdd     5.45799         osd.744      destroyed        0 1.00000
 775   hdd     5.45799         osd.775      destroyed        0 1.00000
 805   hdd     5.45799         osd.805      destroyed        0 1.00000
 860   hdd     5.45799         osd.860      destroyed        0 1.00000
 940   hdd     5.45799         osd.940      destroyed        0 1.00000
1031   hdd     5.45799         osd.1031     destroyed        0 1.00000
1060   hdd     5.45799         osd.1060     destroyed        0 1.00000
 -50         169.19800     host ceph-sn858
1739   hdd     5.45799         osd.1739     destroyed        0 1.00000
 -11         196.48801     host ceph-sn859
 851   hdd     5.45799         osd.851      destroyed        0 1.00000
 -28         196.48801     host ceph-sn860
 993   hdd     5.45799         osd.993      destroyed        0 1.00000
1049   hdd     5.45799         osd.1049     destroyed        0 1.00000
 -22         196.48801     host ceph-sn862
 199   hdd     5.45799         osd.199      destroyed        0 1.00000
 383   hdd     5.45799         osd.383      destroyed        0 1.00000
 -51         169.19800     host ceph-sn863
1769   hdd     5.45799         osd.1769     destroyed        0 1.00000
1772   hdd     5.45799         osd.1772     destroyed        0 1.00000
1785   hdd     5.45799         osd.1785     destroyed        0 1.00000
 -30         191.03000     host ceph-sn864
  63   hdd     5.45799         osd.63       destroyed        0 1.00000
 -52         185.57201     host ceph-sn865
1818   hdd     5.45799         osd.1818     destroyed        0 1.00000
1828   hdd     5.45799         osd.1828     destroyed        0 1.00000
1835   hdd     5.45799         osd.1835     destroyed        0 1.00000
 -13         191.03000     host ceph-sn867
1058   hdd     5.45799         osd.1058     destroyed        0 1.00000
  -8         191.03000     host ceph-sn868
 512   hdd     5.45799         osd.512      destroyed        0 1.00000
 -10         191.03000     host ceph-sn869
 773   hdd     5.45799         osd.773      destroyed        0 1.00000
 854   hdd     5.45799         osd.854      destroyed        0 1.00000
 -54         180.11400     host ceph-sn870
1892   hdd     5.45799         osd.1892     destroyed        0 1.00000
1900   hdd     5.45799         osd.1900     destroyed        0 1.00000
1905   hdd     5.45799         osd.1905     destroyed        0 1.00000
1907   hdd     5.45799         osd.1907     destroyed        0 1.00000
 -34         196.48801     host ceph-sn871
1122   hdd     5.45799         osd.1122     destroyed        0 1.00000
1153   hdd     5.45799         osd.1153     destroyed        0 1.00000
 -21         196.48801     host ceph-sn873
  50   hdd     5.45799         osd.50       destroyed        0 1.00000
 -56         147.36600     host ceph-sn874
1953   hdd     5.45799         osd.1953     destroyed        0 1.00000
1958   hdd     5.45799         osd.1958     destroyed        0 1.00000
1975   hdd     5.45799         osd.1975     destroyed        0 1.00000
 -29         174.65601     host ceph-sn875
 845   hdd     5.45799         osd.845      destroyed        0 1.00000
 957   hdd     5.45799         osd.957           down        0 1.00000
 -35         191.03000     host ceph-sn876
1140   hdd     5.45799         osd.1140     destroyed        0 1.00000
1214   hdd     5.45799         osd.1214     destroyed        0 1.00000
1234   hdd     5.45799         osd.1234     destroyed        0 1.00000
 -31         196.48801     host ceph-sn877
 231   hdd     5.45799         osd.231      destroyed        0 1.00000
 315   hdd     5.45799         osd.315      destroyed        0 1.00000
 903   hdd     5.45799         osd.903      destroyed        0 1.00000
 -57         169.19800     host ceph-sn879
1980   hdd     5.45799         osd.1980     destroyed        0 1.00000
 -36         185.57201     host ceph-sn883
1155   hdd     5.45799         osd.1155     destroyed        0 1.00000
 -37         196.48801     host ceph-sn884
1268   hdd     5.45799         osd.1268     destroyed        0 1.00000
1409   hdd     5.45799         osd.1409     destroyed        0 1.00000
4798   hdd     5.45799         osd.4798     destroyed        0 1.00000
 -38         196.48801     host ceph-sn885
1329   hdd     5.45799         osd.1329     destroyed        0 1.00000
1420   hdd     5.45799         osd.1420     destroyed        0 1.00000
1425   hdd     5.45799         osd.1425     destroyed        0 1.00000
 -39         196.48801     host ceph-sn886
1322   hdd     5.45799         osd.1322     destroyed        0 1.00000
 -40         191.03000     host ceph-sn887
1297   hdd     5.45799         osd.1297     destroyed        0 1.00000
1364   hdd     5.45799         osd.1364     destroyed        0 1.00000
 -41         196.48801     host ceph-sn888
1292   hdd     5.45799         osd.1292     destroyed        0 1.00000
 -60         174.65601     host ceph-sn889
2099   hdd     5.45799         osd.2099          down        0 1.00000
 -61         163.74001     host ceph-sn890
2130   hdd     5.45799         osd.2130     destroyed        0 1.00000
2139   hdd     5.45799         osd.2139     destroyed        0 1.00000
2141   hdd     5.45799         osd.2141     destroyed        0 1.00000
 -63         169.19800     host ceph-sn892
2205   hdd     5.45799         osd.2205     destroyed        0 1.00000
2214   hdd     5.45799         osd.2214     destroyed        0 1.00000
2216   hdd     5.45799         osd.2216     destroyed        0 1.00000
2223   hdd     5.45799         osd.2223     destroyed        0 1.00000
2229   hdd     5.45799         osd.2229     destroyed        0 1.00000
-167         174.64799     host ceph-sn907
2525   hdd     7.27699         osd.2525     destroyed        0 1.00000
-369         174.64799     host ceph-sn916
4875   hdd     7.27699         osd.4875          down        0 1.00000
-135         174.64783     host ceph-sn922
1945   hdd     7.27699         osd.1945     destroyed        0 1.00000
-371         174.64799     host ceph-sn923
4884   hdd     7.27699         osd.4884     destroyed        0 1.00000
4891   hdd     7.27699         osd.4891     destroyed        0 1.00000
4897   hdd     7.27699         osd.4897     destroyed        0 1.00000
-139         174.64799     host ceph-sn925
2162   hdd     7.27699         osd.2162          down        0 1.00000
-155         174.64799     host ceph-sn934
2402   hdd     7.27699         osd.2402     destroyed        0 1.00000
-157         174.64799     host ceph-sn936
2428   hdd     7.27699         osd.2428     destroyed        0 1.00000
-313         260.78394     host ceph-sn969
4224   hdd    10.86600         osd.4224     destroyed        0 1.00000
-333         260.78394     host ceph-sn979
4460   hdd    10.86600         osd.4460     destroyed        0 1.00000
-349         239.05200     host ceph-sn987
4637   hdd    10.86600         osd.4637     destroyed        0 1.00000
-351         260.78394     host ceph-sn988
4656   hdd    10.86600         osd.4656     destroyed        0 1.00000
-363         260.78394     host ceph-sn994
4789   hdd    10.86600         osd.4789     destroyed        0 1.00000
5002   hdd    10.86600         osd.5002     destroyed        0 1.00000
-231         239.05200     host ceph-sn999
3332   hdd    10.86600         osd.3332     destroyed        0 1.00000
3338   hdd    10.86600         osd.3338     destroyed        0 1.00000