- slow reqs on gabe seem to have increased. My theory is this is caused by large omap objects (bucket indices) due to a few large buckets. We can / should reshard, then compact the rocksdbs. CEPH-950
- so far tested the reshard on a few large cvmfs buckets: reads are not affected, but writes block for the ~10 minutes of the reshard.
2020-08-31 10:41:10.868622 7fc4d28b5dc0 1 execute INFO: reshard of bucket "cvmfs-sft-test0" from "cvmfs-sf
t-test0:61c59385-085d-4caa-9070-63a3868dccb6.205147250.6" to "cvmfs-sft-test0:61c59385-085d-4caa-9070-63a3868dccb6.271824312.1" completed successfully
# rados ls -p default.rgw.buckets.index | grep 61c59385-085d-4caa-9070-63a3868dccb6.271824312.1
# rados ls -p default.rgw.buckets.index | grep 61c59385-085d-4caa-9070-63a3868dccb6.205147250.6
In some cases like ^^ we need to remove the old bucket index objects still. procedure TBD.
[11:04][root@cephgabe0 (production:ceph/gabe/mon*2:leader) ~]# radosgw-admin reshard stale-instances list
In any case, we still need to reshard the following buckets:
- S3 account cleaning campaign
- Lots of personal account previously flagged as "illegitimate" have been disabled
- Most of which have 0 byte used
- They will be deleted after a couple of weeks without complaints
- To see with Jose if we email those who have some data left