JCOP FWWG 6 July 2021: fwNextGenAlarmScreen
JCOP FWWG Meeting 6 July 2021
INDICO: https://indico.cern.ch/event/1053341/
Topic: the NextGen Alarm Screen, presented by Rafal and Leonid
Participants: Christophe Pires, Wassef Karimeh, Piotr Golonka, Leonid Kopylov, Raul Jimenez, Ombretta Pinazza, Frank Glege, Rafal Kulaga, Clara Gaspar, Yash Kataria, Stefan Schlenker, Ioannis Papakrivopoulos, Juan Manuel Pablo Rodriguez, Mateusz Lechman, Jean-Charles Tournier, Cristina Vazquez Velez, Ben Farnham, Valeri Falaleev, Laura Sargsyan, LuisGC, Fernando Varela, Giovanna Lehmann, Riku-Pekka Silvola
Discussion:
-
is the history of DPE ID changes taken into account in the history mode? Consider use case of DP being re-created (with different ID) in the middle of queried time-range.
-
yes, the data stored in the "meta-data history" tables are taken into account
-
however, certain information may not be available, e.g. the device comment, which needs to come from the parent "device" datapoint's "." (root) element, not from the DPElement - this is not always available, etc.
-
concluded that it would be good to prepare technical documentation for experts explaining what and how is actually archived and which columns need to be calculated from other (live) values in the system.
-
-
in history mode, the colors of alarms are "strange": a lot of them are green...
-
the "green" color is the one FwOkStatePhysics, ie. used when the alarm is gone.
-
what you see is the "final" state of alarm, which is usually already "gone", not the color when the alarm was active...
-
this may be misleading and very hard to explain to shifters
-
Is it that the information is actually "lost" or "incomplete" while archiving? Or could we recover it somehow to reconstruct the more "correct" colour, e.g. the one of the maximum severity reached
-
actually this is what ATLAS does in their web alarm screen: they "override" the alarm color remapping it to the colour that is mapped to the corresponding alarm class. This is based on the framework convention and assumes that standard alert classes are used - approach proves itself to be good enough.
-
LHCb acks that they would also prefer this way of colouring
-
Rafal confimed that this should be possible and quite straightforward to extend the NGAES to support such mode of colouring
-
How about colouring the complete row of the table? It is already possible, yet obviously with the current colouring algorithm. This is particularly useful when looking up important alarms.
-
-
Do you plan to add pagination in history mode to get alarms for the long time range?
-
technically it should be possible - data is already retrieved/processed in batches hence something could be implemented. However, depending how the paging is implemented, the actual gain in memory consumption (by use of paging) may not be achieved as all the data returned by a query needs to be processed and filtered in some way.
-
Nobody asked for this feature in the original round of requirement gathering. We do not plan to implement it unless someone asks explicitly for this feature
-
-
Is it possible to investigate a history of one particular alarm (or device)
-
Yes, should be possible by applying filters per-device;
-
in particular there are separate column to display the times of "came", "went" and "acknowledgments"
-
-
When switching to history mode in the filter, but the table still shows the live alarms... this may be quite misleading to shifters/operators
-
there is a small "filter not applied" message - far from being obvious, as switching to historical mode is not really a filter!
-
how about a "history" for the filters' fields, allowing to revert the edited content to the previous state (ie. the one that is currently displayed in the table)
-
should we have a separate "TAB" for history mode? Not really, as tabs are used to store pieces of the filter...
-
Suggestion: we should probably consider having separate windows for live and historical mode. This would allow for much better customizations, e.g. set of columns to be displayed (see also below)
-
-
Could we have separate column-layouts for live and historical mode?
-
As discussed above: maybe we should have separate screens for live and historical alarms, with dedicated configurations, and then we may hide the "switch" that activates historical or live mode. It would also help to avoid confusion.
-
the idea generally supported by everybody
[AMMENDED after discussion with CG:] this should be somehow configurable; the panel allowing to switch between the live and historical mode may still be very useful for experts -
actually in many cases access to history mode should NOT be possible to shifters. Notably, to avoid that inadvertently they start a heacy query that impacts the Oracle DB.
-
for this reason, in the ATLAS alarm screen, the historical mode has been disabled completely.
-
-
Beware of Oracle's optimizer which sometimes changes from one patch or another...
-
The warning message that is popped up when case-sensitive match is selected may... become misleading if at some moment the optimizer will chabge the execution plans and then again, case-sensitive will be faster. Consider making the message more generic, simply indicating that the query may take long (not only when case-sensitive match is selected).
-
-
Once you start to modify the existing filter, and before you apply it, it is not obvious which parts of it have been modified. Would it be possible to highlight these somehow
-
Rafal explained that internally this is already "tracked", hence it could be made visilble
-
what if someone removes one of the tabs (one of criteria)? How would we "highlight" this? In any case, non-expert users will unlikely use many criterias, hence the case is negligible
-
-
Would it be possible to highlight per-filter field that has not been applied? Or remember the last value?
-
Use case of flickering alarms that come/go very quickly. the shifter in CMS "heard" the audible notifications for alerts but it is already gone from the screen... For this purpose in CMS there is an additional "last 10 alarms" table in the screen, which is copied from the "main" alarm table...
-
the use case should be discussed - certainly a very interesting one
-
CMS could implement this table themselves, provided that access to the data model is available programaticaly
-
the AlarmScreenNG is delivered as an EWO (a widget) which is highly customizable, and in any case building a customized alarm screen with extended features is certainly achievable!
-
-
Pre-filling a filter from a particular alert row
-
even though completion exists, JCOP Fw devices have long DPE names (e.g. CAEN channel)
-
for instance, drag and drop an alarm row into one of the filter fields and then it is easy to, eg. edit the part of the CAEN channel name or add a wildcard
-
drag-and-drop works already to some degree: it copies the complete content of the row though, hence currently one would need to drop it into a text editor and then copy-paste the relevant fragment. Would be nice if we could get this automatized in a smart way!
-
-
This feature would be particularly interesting to analyze one alert or one device.
-
One should consider to open a new alarm screen window with such filter
-
-
-
After some time alert screen should switch to the live mode?
-
feature discussed in the past for the standard fwAlarmScreen and implemented...
-
as discussed above, it is perfectly feasible to come up with customized alarm screen derived from what fwAlarmScreenNG proposes - all these are configurable EWOs.
-
-
Will the AlarmScreen work if Oracle client is not installed?
-
Live mode will still work, as the DB backends are loaded as plugins - the same mechanism (and plugins) as for CtrlRDBAccess
-
if we wanted to migrate to NGA with InfluxDB, or Postgress backends, we will need to extend the AES to implement the queries
-
as direct-queries are usually used to query historical data for trends anyway, Oracle client is usually installed also in the consoles where the UI manager runs
-
-
Save filter -> Save filter"s" , as it saves all the tabs to make it clear
-
Connected systems rectangles: what if we have a large system
-
will scale-them down to have smaller rectangles; tried with 80 looked OK
-
-
Historical queries: is it that we always query all alarms in the time-range?
-
system names, device names, alarm names are pushed into SQL queries as predicates; it is not that everything is queried every time.
-
-
The "old" alarm screen has a popup panel in which alarm details could be seen. Notably things such as ACK time, or ACK user...
-
shifters should really not tinker with adding/removing columns, as they will likely not restore the alarm screen layout to the original state after this... They need to have an easy way to find the complete information about the alarm
-
-
Is there a column for the user who ACKed an alarm?
-
No. But the data seems to be archived (as user I which needs to be remapped to name - ATLAS is already doing this)
-
The use case for this actually do exist, with experts often asking to have an information about who ack'ed a particular alarm.
-
This could probably be displayed in the "alarm details" panel
-
-
Should we kick off the discussion with ETM to make sure that all the necessary data related to alarms is stored in the DB (and hence avoid the need to "recalculate" from live values, or "guess" alarm colors from archive classes, etc?
-
We should restart this topic with ETM. To be discussed with CMS (and others) to gather the requirements.
-
This has a significant overlap with NextGenArchiver too, and needs to be followed up in this scope quite likely.
-
No AOB due to extended time taken by (very lively) discussion.
Summer holidays with lots of absences envisaged - next JCOP FWWG meeting expected in the beginning of September.