AFS weekly meeting

Monday 7 Aug 2017, 14:00 → 15:00 Europe/Zurich

31/1-012 (CERN)

AFS service/operation meeting

AFS operations meeting 2017-08-07

present: Kuba, Cristi, Massimo, Jan

Issues:

• new AFS restore server (afs900) is slow (corridor discussion , with Dan). A 50GB user restore took 7h real time, general feeling that before (afs252/ck, phys hardware) was "much" faster.
  • Dan ran some tests, limitations look like the 100 IOops / 80 MB/s quota kick in repeatedly:
  • I see contention between reading the image in abs_restore, and writing with vos into the afs partition. Mostly its just reading: Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vdb 0.00 0.00 101.00 0.00 11.92 0.00 241.74 3.83 38.67 38.67 0.00 9.90 100.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 The contention is the IOPS and BW throttles on the standard volumes. A few things I suggest:
    1. Move abs_restore to its own volume, independent from the partition where the volume is restored to.
    2. Use a bigger memory VM so that the page cache holds more of the data just written to disk.
    3. tweak your xfs and vdb block sizes so xfs does 4MB reads instead of the apparently 256kB per read we have now. [1]
    4. Use io1 volumes for this.

• reboot campaing ongoing (now with the user homedirectory servers, afs260+), with occasional manual prodding (procedure bombs out easily..)

• 1 auto-restart of a boinced fileserver last week (still get a SNOW ticket in parallel; often is triggered by AFS backup kicking in (and increasing the number of "vos" commands))

• ABS: should redo a mass-restore test. Ideally could have an empty machine do all the restore work that a (production) server would have to do, but without registering the newly-restored volumes in VLDB (and scrapping them afterwards)

• one CBOX ticket: apparent confusion about files owned by different groups by same user (z2, zh) - policy? Policy is "purely cosmetic for AFS, no cleanup done on group change; owner can run "afs_admin set_owner" if really annoyed by this)