Sep 21 – 25, 2020
(teleconference only)
Europe/Paris timezone

Archival, anonymization and presentation of HTCondor logs with GlideinMonitor

Sep 22, 2020, 4:40 PM
20m
https://cern.zoom.us/j/97987309455

https://cern.zoom.us/j/97987309455

HTCondor user presentations Workshop session

Speaker

Marco Mambelli (University of Chicago (US))

Description

GlideinWMS is a pilot framework to provide uniform and reliable HTCondor clusters using heterogeneous and unreliable resources. The Glideins are pilot jobs that are sent to the selected nodes, test them, set them up as desired by the user jobs, and ultimately start an HTCondor schedd to join an elastic pool. These Glideins collect information that is very useful to evaluate the health and efficiency of the worker nodes and invaluable to troubleshoot when something goes wrong. This includes local stats, the results of all the tests, and the HTCondor log files, and it is packed and sent to the GlideinWMS Factory.
Access to these logs for developers takes long back and forth with Factory operators and manual digging into files. Furthermore, these files contain information like user IDs and email and IP addresses, that we want to protect and limit access to.
GlideinMonitor is a Web application to make these logs more accessible and useful:
- it organizes the logs in an efficient compressed archive
- it allows to search, unpack, and inspect them, all in a convenient and secure Web interface
- via plugins like the log anonymizer, it can redact protected information preserving the parts useful for troubleshooting

Speaker release Yes

Primary author

Marco Mambelli (University of Chicago (US))

Co-authors

Presentation materials