AFS weekly meeting

Tuesday 5 Apr 2016, 16:30 → 16:45 Europe/Zurich

31/1-012 (CERN)

AFS service/operation meeting

AFS operations meeting 2016-04-05

present: Dan, Kuba, Jan

Issues:

• UID migration: ongoing but few users left (Kuba, Frederic + BE users that were delayed to June)
• ABS: still tries to clean up the "wrong" restore volume mountpoints?
• Instability:
  • last week was bad, with 4..5 server restarts across all use cases. Probably linked to re-UID script (but INSPIRE volume blocked by user-run "fs setacl", and work.boinc also two times ("fs setacl" stuck until server restart, then continued, then stuck again..)
  • Kuba got negative feedback; users assume this is part of our plan to phase out AFS.. (ask to get in touch)
  • List of known symptoms (these (hopefully) are linked) - should go to JIRA
    • recursive "fs setacl" gets stuck, affects volume, evtl whole server
    • "rescheduling" patch: stays at max. allowed threads, even when "/p/etc/trace_fs" shows no ongoing activity
    • dafileserver gets stuck in shutdown while detaching volumes. Logs shows increasing counter, no progress
    • "vos" commands get stuck against a server that has one bad volume (perhaps fro "fs setacl"
    • client gets 'No buffer space' at application level. Seems mistranslated "server is busy"-message that should cause the client to wait. Upstream has proposed patches around the area in master.
    • CC7 client starts afsd several times (systemd just keeps starting them, perhaps after a timeout on the first process). Causes afsd pileup and error message, evtl system instability. Workaround is to tweak systemd service file.
  • Discussion.. conclusions:
    • should build synthetic 'fs setacl" reproducer, reproduce on upstream (work.bpinc managed to), then submit bug report
    • should bump CC7 to latest upstream (unclear procedure - Koji? ask Thomas)
    • should bump SLC6 to latest upstream+CERN patches (brokenness is on server side)
    • Should submit systemd tweaks to upstream