JCOP FWWG 9 Mar 2021

→ Europe/Zurich

JCOP Framework Working Group Meeting

9 March 2021

INDICO: https://indico.cern.ch/event/1009822/

Participants: P.Golonka, A.Hennessey, R-P.Silvola, A.Kurepin, C.Gaspar, C.Pires, F.Varela Rodriguez, F.Glege, G. Thomas, JM Pabol Rodriguez, LuisGC, L.Goralczyk, M.Suder, M.Lechman, O.Pinazza, P.Moschovakos, P.Chochula, PGV, P.Nikiel, R.Kulaga, S.Schlenker, T.Angelova, W.Karimeh, S.Khomoutnikov

NEWS:

  • WinCC OA 3.16 patch 20210112 available, containing latest ETM patches P024 and P025; more details on the downloads page; no recommendation yet
  • No particular news on the JCOP Framework
  • WinCC OA 3.18 evaluation and framework porting: put on hold in spite of good advancement (major part of the JCOP Framework ported); awaiting an update of the LHC Schedule expected to be released in March, as well as more news about Centos8 successor

NGA

  • Presentation given by Anthony, Mariusz and Rafal is available on the INDICO page
  • question to have the topic of meta-data sync expanded; what does "long" mean - large systems were taking hours to synchronize for the first time; it is as long as it is for the RDB Archiver anyway - the slowness is due to the DB Schema architecture. It does not take longer than RDB - covered more further in the presentation.
  • Suggestion for test: destabilizing, breaking connection, etc and see how the system recovers: a lot of this is already being done, and in automatic ways
  • Surprising that the max rate limits for RDB and NGA/Oracle is the same while the CPU resource consumptions differ; explained that it is clealy the DB Schema in the PL/SQL code which is very complex;
  • Did you try to change the DB instance (different server) - no, but we used a production-grade DB. 
  • A performance test with "test backend" (that discards all data) is planned to stress test the messaging.
  • One may change the IO/CPU capabilities on the client too; is it 100% of the performance of CPU by the NGA process causing the problem? Would be nice to see/confirm the origin of the performance bottleneck.
  • Note that the NGA is multi-threaded, hence no cap on 100% of load. Would be nice to see how multi-threading works and see the 200%, etc.
  • CMS: Does it mean that the standard RDB schema is used by the NGA? Yes; and we do ack that it is the frustrating limiting factor.
    • how about ignoring the temp table and write directly to the data tables? Once the current stage of work is finished, we plan to have a look at this.
    • in particular for ALARMS it is very penalizing, but it may require very deep changes 
    • we focus on backward-compatibility as the most important requirement for the time being; the problem is not really in writing (even if it could be done better). The real problem is in the performance of QUERIES. 
  • Acceptable performance means to be judged as the whole of the system: use the performance of the WinCC OA servers which are much cheaper than the resources of the DB... We need to use the resoures in reasonable way: the goal is to do the maximum we could get to!
  • Exploring the performance of schema will require quite significant resources, esp. if we want to keep backwards-compatibility. Acked that it is worth investing the effort!
  • Peter Chochula: ALICE is running NGA in production sucessfully, with all issues handled very smoothly and is grateful for the help and responsiveness of the NGA team
  • What is the main selling point of NGA? The possibility to store data in other databases, with varying retention/granularity plus possibility to stream data (via custom backends); this, combined with the functionality of queries having the selection
  • How the performance will evolve with multi-backend? Writing with 3 backends should be still good to go, accorrding to initial tests. Serving "all events" backends at peak to be looked at. The writing scenario has decoupling into separate processes
  • How many additional production users for this year? Shortage of resources expected: preferred to limit the effort in supporting for this year still...
  • Not sure if  integration of grafana to WinCC OA UI EWO is well understood... Anyway, it is of 2nd priority and considered a "nice to have" feature. People would usually use the web-version for any non-trivial analysis - not sure if embedding in WinCC OA ui relevant.

 

AOB

There are minutes attached to this event. Show them.