JCOP FWWG 18 May 2021 - fwSystemIntegity
JCOP Framework Working Group Meeting
18 May 2021
INDICO Link: https://indico.cern.ch/event/1038886/
Participants: Ivan Atanassov, Geraldine Thomas, Luigi Marchese, Wassef Karimeh, Valeri Falaleev, Riku-Pekka Silvola, Piotr Golonka, Christophe Menezes Pires, Stefan Schlenker, Frank Glege, Ben Farnham, Michael Ludwig,
Clara Gaspar, Mariusz Suder, Mateusz Lechman, Cristina Vazquez, Jean-Charles Tournier, Piotr Nikiel, Ombretta Pinazza, Andreas Sochopoulos, Luís GC, Viatcheslav Filimonov, Ivan Shvetsov, Anton Dimitrov, Juan Manuel
News about WinCC OA and the JCOP Framework
- WinCC OA 3.18 officially released. We already have initial rebuilds containing CERN patches, which may be made available to interested developers - get in touch with Piotr to arrange for a distribution package and licenses. Note that the framework distribution is NOT YET available.
- WinCC OA 3.16: new patch (P026) from ETM and a new version of CtrlRDBAccess addressing issues with "long" data-types (used e.g. by recent CAEN devices). Distribution packages already available at the Downloads page
- JCOP Framework for WinCC OA 3.18? The JCOP CB agreed that some resources are committed to complete the already-started efforts on porting the framework to WinCC OA 3.18. This should result in a "test" version available towards the end of 2021. This version should NOT be considered for production for the start of the RUN3, but rather for development of new systems. For the startup we stay with WinCC OA 3.16. The port of the framework should allow us to prepare for the upcoming opportunities for WinCC OA upgrades in the following years.
- CANx: progress with testing on Windows and builds on Centos8/Windows2019Server. Some combinations h/w work better than others - more on this will be presented once the tests are finished
- FwFSM: new version available, yet seems to have issues in very large systems (detected in CMS); Clara looking into this
- OPC-UA servers for LXI and PDUs: available/testing. Contact Michael for details
Changes in the Wiener HAL: feedback needed
Ben requests reflection and feedback concerning the following changes to the HAL layer proposed by Wiener, having possibly serious impact - see the description below and then
please feed back to Ben, as Wiener is putting on hold the development awaiting CERNs decision.
This proposal comes in the context of JCOP request ENS-29156 to add a property to the OPC-UA namespace for Wiener MPOD devices. Please note this proposal affects all Wiener users.
The background to this is that the new MPOD per-channel property to be added (outputUserConfig) is already present in other Wiener PL5XX devices (e.g PL512), but, in the process of investigating the case Wiener noticed that property RegulationMode is in the wrong place in the property tree. An earlier mistake basically.
Currently (wrong place) à Output.Channel.5:RegulationMode
Proposed (right place) à Output.Channel.5.Config:RegulationMode
The OPC-UA layer handles wiener HAL property changes transparently (only the full configuration would need to be re-generated with option --hardwareDiscovery). However, reflecting wienerHAL property tree changes faithfully means changing the OPC-UA namespace, and that means breaking backwards compatibility between fwWiener versions and OPC-UA server versions (since fwWiener maps dpe periph addr configs to specific addresses in the OPC namespace).
The benefit of moving the property to the right place is a better interface; a more intuitive namespace – RegulationMode is used to configure the channel; therefore should be a child of node Channel.X.Config.
Options are:
- Introduce the new MPOD RegulationMode property in the same wrong place as-per PL5XX RegulationModule property. Not great - In this case the original mistake is compounded, rather than fixed.
- Have MPOD RegulationMode property in a different place than PL5XX RegulationMode property. Terrible – introduces a special case to live with forever more.
- Introduce MPOD RegulationMode in new place and change PL5XX RegulationMode to new place. Mistake fixed, good – backwards compatibility broken; users must update fwWiener (& periph addr configs) together with OPC-UA server.
- Take this to a wider scope: fix all identified mistakes and proposed improvements in the entire property tree structure. Same pain as #3, but a better property structure going forward.
I already asked Wiener for what fixes/improvements they could foresee making to the structure if we opt for option #4, they responded with the proposal below. Note that this is an opportunity for CERN Wiener users to propose structure changes also. I’ve pasted Wiener’s proposal below, changes in red, for comment.
Questions needing answered by the JCOP-CB
- Which option above best suits, in terms of long term operational needs ?
- If option #4 is best, please either (i) agree with Wiener’s changes, or, (ii) propose enhancements/alternatives.
Monitoring of hardware statuses and readout chains
- Ben introduced the topis of monitoring the "integrity" of communication with standard JCOP devices interfaced by standard OPC-UA - see the presentation attached.
The proposal is about sharing the common status computation logic across users. - The notes from the BE-ICS-FT internal "brainstorming" session are available at https://readthedocs.web.cern.ch/display/ICKB/FwRFC+-+SystemIntegrity+for+JCOP . Note that they may be imprecise/incomplete.
- CMS presented their requirements in terms of monitoring of CAN interfaces and having watchdogs for CAEN equipment. Presentation attached in indico.
- Discussion and questions turned aroud two topics: detailed statistics for CANbuses being exposed by the OPC Servers and the reusable and abstract mechanism for treating "heartbeat"-style of monitoring the hardware such as CAEN, allowing for deviceType-specific implementations that could be shared by the whole JCOP community.
- should there be only a boolean output from the monitoring or could we imagine some more complete diagnostics? Obviously, a boolean is a minimum - we may aim for something much more complete
- important to distinguish the state of device and the state of communication - important from the very beginning!
- is Ben's propoosal of hierarchical monitoring somehow a side-structure to the already existing hardware-device in the framework (a duplication of it?). Not really: the hardware hierarchy would simply present the data exposed by the OPC-UA server items, and as explained in the presentation the OPC-UA server is not going to take any "overall decision" about a health status of a piece of hardware. What is proposed as monitoring may require some activity ("watch dogs") or extra logic that takes into account the "overall" structure of devices as known completely only by the SCADA. Hence it is not really redundant or parallel
- in ATLAS there is already a structure like this one in particular for ELMB devices. However, there is indeed a need to monitor CAN - probably to be added to OPC-UA servers
- ATLAS: What does it mean that the "integrity" layer delivers the "OK/NOT OK"? It is usually FSM that is used to model devices and then more precise "states" for hardware may be modelled.
- CMS: acks that FSM is used and the behaviour is in the Device Unit. From the point of view of an expert you start with an alert and proceed to get more details about the device using panels, etc. The complete hardware structure is NOT in the OPC-UA - it is in the FSM! Modelling a second abstraction would have consequences (problems with majority, etc). FSM device units could therefore be more adequate...
- CMS: the origin of the discussion was about some CAN stats that were present in OPC-DA not being available in OPC-UA. The conclusion was that the CAN monitoring information is very meaningful for diagnostics and many problems do have origins in CAN. Ideally all information from the CAN interface should be made available... Then obviously to be seen what parts would be interesting and how you attach it to a particular device (and then used to derive the hw state). There is a need for much more detail in low-level information so that one could more easily troubleshoot CAN buses.
- CMS: CANbus stats would already be sufficient... Should we invent yet another CAN analyzer? Work that needs to be put into it? We already have existing CAN analysers... Just expose all possible statistics that are available.
- ATLAS: CAN statuses are actuall well standardized. We should have a look at them. ... yet this is only if the CAN bridge vendor exposes it... If yes, then certainly to be exposed
- ATLAS: they do not have problems with Maratons, because they use SNMP control; for CAN-open there is quite a lot of stats available already; for the VME crates, ATLAS still uses the ATLAS-own OPC server where this is reported. For the time being they are OK. For CAEN, they use CPU/Memory readouts to get the heartbeat - good to standardize. A standardization how CANmodule reports the CAN statuses could be uniformized across OPC Servers.
- Clarification from Michael: CANmodule latest version (2.0.8 in the quasar) already reports on the status of the bridge, but it depends on the individual can-bridge. Some have it very complete and some others very poor... For some we have statistics, for others we may have very few things to monitor...
- Further clarifications from Michael added after initial version of minutes:
- We have statistics for all implementations [sock, peak, systec, anagate] available, but status of bridge and CAN bus depends on the bridge vendor and the implementation: everything which is there is already published in the API.
- I think that the "unified canmodule status" and the statistics should be added to the servers which use CanModule.
- needs validation, but should be bw compatible and not too bad to do
- Ben should alrteady have all info, if not, ask me again
- Further clarifications from Michael added after initial version of minutes:
- Acked by Ben: certainly the CAN status information may be exposed by the OPC Servers - not a massive reengineering
- CMS: "Device watch dog" : looking at regularly-updating values such as fan speeds. CMS has a common component used in all subdetectors... Is this something that we could all reuse ?
- CMS subdetectors: actually looking for an abstract interface to implement such heartbeat mechanisms, as there are many types of monitoring for various types of devices. CMS implementation does not feature a tree, but rather a flat architecture, and then a summary of heartbeats. A tree structure would actually be interesting - in some sense it is already there by the FSM. This is mainly for the CAEN devices.
- Clarification from Ben: for the newer CAEN SY4527 there is an internal register that could be used to monitor the firmware of the mainframe itself. However if you have older ones, or a CAEN VME, this would not work... An abstract interface could allow to treat various types of devices with various mechanism available.
- After the meeting, CMS Tracker have sent a link to the gitlab repo with the current implementation of their mechansim and a presentation (attached to the "Discussion" part in indico), offering to further discuss/review the implementation
AOB and next meeting
- there was no "any other business"
- next meeting likely on 1st of May (clarified after the meeting: ATLAS will present the ATCA shelf manager integration)