JCOP FWWG 27 June 2023
JCOP FWWG Meeting, 27 June 2023
https://indico.cern.ch/event/1286701/
Present: Cristina V, Wasef K, Fernando V, Frank G, Rafal K, Antonin K, Piotr G, Geraldine T, Juan Manuel, Luis GC, Mariusz S, Mateusz L, Peter B, Piero Giorgio V, Raul J, Sergio G. , Valeri F, Stefan S, Michael L, Lukasz G, Paris M
Presentation
Rafal/Antonin presented an update of the status of the NextGen Archiver - slides are attached in INDICO.
Discussion
- FVR: Why compressed is so much faster?
Rafal: it is not a "zip" compression that it is about - it is the pattern of I/O operation and the organization of data
Frank: it would be good to see this tested also on the machines with no h/w limits
Rafal: this is what is planned for tests being arranged for with dedicated servers prepared by IT/DA
Frank: What about the High-Availability Features? It is extremely important (and affects the cost of the DB Service...)
Rafal: Havent't looked at it just yet - planned to be done as a part of the IT/DA tests. The aim is to have a service that runs in parallel to Oracle, and only if one is convinced about the reliability (being at least as good as current Oracle service), one would think of switching
FVR: Summary of discussions with IT-DA?
Rafal: they also recognize issues with Influx in number of deployments, and hence they are very much interested in taking part in these evaluations, provided this is approved by the IT management (need to be endorsed by the management). They work with Legal Service to even try to get the TimeScaleDB "full" license from the company - anyway the license situation is much better with many other DBs
Frank: Postgres is definitely one of the best choices that could be made.
Frank: WinCC OA is not the only client of the DB Service - it shares it with others (past experience showed it). Performance does not need to be "rocket fast" anymore these days.
Rafal: to offload one could also use the replication service of Postgress to spread various workloads to different databases, which could help a lot.
Frank: distribution of load - this is where HA is important, with proper distribution of schemas etc on centralized instance. Oracle brings extremely high-comfort standards with it.
Rafal: absolutely! We do not want to have as many databases as systems!
There is no point in comparing direct numbers of performance measurements e.g. with Oracle (running on dedicated hardware).
Frank:Is it that Postgress does not provide time-partitioning natively (hence need for TimeScaleDB)?
Antonin: there is one, but it is rather poor. One could surely implement partition-pruning on one's own, obviously, but this is what TimeScaleDB gives us out of the box. In addition, with growing amount of data TimeScaleDB's write performance remains stable (unlike with some non-partitioned large and ever-growing table in Postgress)
Stefan S: looks very nice; details on testing would be nice to know...What are the options for running/configuring this in parallel to Oracle?
Rafal: parallel archiving for NGA is fully supported now in the archive config - you specify multiple archive groups with different smoothing, etc (similar to alarm detail). Very well tested by many ETM customers (storage with different precisions).
Piotr G: It is also covered by fwCore and fwConfigurationDB - visible if you enable NGA
Antonin: for the tests: 7.6. GB this was with a couple of hundreds of thousands with various frequencies, and the query workloads were adapted not to overlap, etc. (we may send exact numbers if interested)
Rafal: insert performance was measured to be constant, with chunks of 10.000 events.
Frank: Schema design: would be interesting to compare with the CMS schema approach with dedicated table per DP (type), etc.
Rafal: Certainly an interesting point to compare. Currently TimeScaleDB employs one table per archive groups (storing all DPEs there). If one would employ dedicated tables then this requires to join data from them when querying many DPs (or all of them) in the same query...
Frank: Oracle partitioning is fast because it corresponds to one file. Is it the same for timescale?
Rafal: we haven't looked at this - question to be addressed by database admin probably...
Stefan: Bonus values - with the current RDB Schema this does not work for mixtures of fast- and slow- changing values... How TimescaleDB would deal with it?
Rafal: it depends on how this is implemented in Oracle - it could be done better or worse. These days you would probably not get a full table-scan. TimescaleDB uses BTree indexes, and there are no indications that it should be wrong; if done in efficient way it should be OK. And in addition compression helps a lot. In the current TimescaleDB schema approach it works very well (done in PgSql) - need to be seen with the hardware.
AOB and WinCC OA 3.19 news
Patch P004 has just been released - repackaged and already available on the downloads page. The same distribution package contains much-improved functionality for the CERN License Tunnel (new button added to the Project Administrator).
It is advised to check with the Central DCS teams first, but if users are keen on trying WinCC OA 3.19 now, it is ready for testing: Framework 9.0.0 release could be used to test the upgrades, license server for developers at CERN seem to be working fine and if necessary we could assign license tickets too.
Validation of SQLite-based data manager: test ongoing; an issue detected (with"configs" database not being cleaned up when a DP is deleted/recreated) already fixed in patch P004. Long-term/scalability tests ongoing
Next JCOP FWWG most likely in September, after the Summer break.