watchdog

Monday 7 Apr 2008, 14:00 → 15:00 Europe/Zurich

513-1-021 (CERN)

At the FIO Working Day, we talked about the need to have a watchdog mechanism on Quattor-Managed machines. We need a brainstorming on that subject to understand - what is really needed (Ulrich, Ricardo, can you prepare inputs ?) - what is the implication on the ELFms tools: -what needs to be enhanced or modified ? -who will do the work ? -what is the timescale ?

Slides from Uli watchdog.pdf

Present : Ulrich Schwickerath, Ricardo Silva, Ignacio Reguero, Jan Iven, Veronique Lefebure (minutes)

Presentation:

Ulrich presented the reasons for the need for a watchdog. See slides attached to this meeting.
Main points:

• Heavy maintenance for kernel upgrades (reboots)
• machines often forgotten on "maintenance" state after an intervention
• Needs:
  • way to enforce reboot of boxes, but
    • allow Application managers to do it themselves before the deadline
    • take into account meta-information such as load-balanced systems, small-size batch subclusters (keep a minimum of capacity alive at all moments)
    • avoid alarms when reboot
      • machine could send a "last hartbeat" before reboot, hoping it comes back within "15 minutes" (or whatever is configured): uggly option
      • machine to put itself into maintenance, for a given duration: this allows both no false alarms and machines not being forgotten in maintenance state, else SMS would clear the state itself after the defined duration

 Discussion:

 Inputs from Jan Iven:

• The security team (Romain Wartel) is pushing for not only kernels to be up-to-date, but any sw package containing security holes. So this should be monitored
• a watchdog mechanism is also needed for desktops. Let's try to use the same system as for quattor-managed machines as much as possible

All:

• We need a backend and a frontend mechanism:
  • BACKEND:
    • runs locally on each box
    • configurable
    • receives the deadline for the reboot (locally and/or from the frontend system)
    • reboots the machine if needed when deadline is reached:
      • run SMS if available
      • need to support double reboot
  • FRONTEND:
    • central source of information (web-page for ex.)
    • has access to high-level knowledge (load-balanced systems, queue length, etc)
    • allows Application manager to define a reboot time (before the deadline)

ACTION LIST:

• SMS duration option:
  • Veronique to talk to Manan for the implementation of a prototype.
  • timeline: 21st of April, to be tested by Uli
• SMS to make use of /etc/shutdowntime: see SetToDesiredState: responsibility up to the Service Manager
• watchdog mechanism: implementation by Uli
• SMS to be run by the box itself: check with Manana and Jan Iven how to have that implemented correctly with no overhead of maintenance of list of hosts etc.
  

Feedback from Vlado:

According to Veronique Lefebure:
> Please find the minutes of yesterday meeting on
> http://indico.cern.ch/conferenceDisplay.py?confId=31864

Vero,

this is nice.

Like this I can know what is going on in project ELFms even if I am not
directly concerned.

I would suggest:

 - do not call it watchdog (smells like monitoring) - but something like
   automatic-reboot

 - have an option to opt-out for a given cluster


Cheers,

                        Vlado