AFS weekly meeting

Monday 17 Jul 2017, 14:00 → 15:00 Europe/Zurich

31/1-012 (CERN)

AFS service/operation meeting

("virtual" meeting = summary by mail):

Issues:

• rundeck drained+rebooted one production server (afs250), now draining the second. Took several restarts of the job, procedure is (intentionally) fragile; got one false alarm from "monitor" ("AFS prob"). 
  • https://afsrundeck.cern.ch/project/AFS/execution/show/42
  • all scripts shipped via puppet (properly tracked, but slow to edit)
• new upstream 1.6.21 (http://openafs.org/dl/openafs/1.6.21/RELNOTES-1.6.21) not yet on CentOS build server, will wait (1.6.21.1 was built from their SRPM). Contains a fix for "100ms transmit delays" seen in certain RX states (this might explain the much increased latency when probing from outside of the fileserver...).
• 1 dead disk on afs501 (Wigner), hardware repair team asked for 2 more "sudo" commands (done).
• "afs" client puppet module - several pending things pushed to production - https://its.cern.ch/jira/browse/CRM-2331
• Looking at alert for un-alarmed machines (raise SNOW ticket if alarms are off for "too long")
  • have got direct mail feed to AFS 3rd-level (afs3-support@cern.ch)
• BOINC auto-restart triggered 3x on same machine (was OK = stuck = several concurrent "fs sa" per INC1409140)
  • tweaked the thresholds somewhat
  • most likely to fire during backup (either the "vos backup" processes cause some further lockup, or they increase the "vos" count above the limit..