ServiceX working meeting
Attendence:
Alexx P, Ben G, Chris W, Ilija V, Jim P, Gordon W, Kevin P, Kyungeon C, Marc W, Mark N, Mason P, Peter O, Rob G,
Ben: The plan for the v1.0 release date is May 26th.
THere are some references to 'ATLAS' in the transfrmer infrastructure which need to go.
Making autoscaling work fully requires switching transformers from Kubernetes jobs (which are not supported by the autoscaler) to pods.
There's a backlog item to provide better error reporting to users.
There is a ticket to check the intergrity of data sent to Kafka, but Ben noted that this is tied to using Kafka itself.
All current Service X users are using an Object Store (MinIO); there is no obvious use for Kafka, so Ben proposed dropping it a supported component of v1.0.
Alexx asked whether it would make sense to still do some integrity checking for data sent to the Object Store.
Ben thought that this is not needed, since instead of streaming data incrementatlly as it is produced, the transformer sends whole, comlpete files to the Object Store only when it is done processing.
There was discussion that files could potentially be truncated during transfer to the Object Store, and that some file formats (like ROOT files) have built-in integrity checking, so users will find out
Mark noted that he would not be looking for any sort of sophisticated recover, just simple detection so that user don't have to sort through which files are good and bad themselves.
Rob asked, when using MinIO as the output buffer space, how much space will be needed to support a practical number of users.
Marc: 10 TB, 100 columns becomes ~300 GB in the Object Store. So, for ten users 3-5 TB might be a good idea.
Kyungeon's datasets are much smaller, essentially negligible by comparison.
Rob asked whether the object store is shared by all users, or a separate one is needed for each user. Ben answered that it is shared, although each request's data is put in a separate bucket.
Rob asked about where MinIO is running and what backing storage it uses. Ben said that MinIO is just running on River, using the local volume provisioner, so it is limited to the storage of one node.
Rob noted that Judith is working on deploying Rook on River, which opens up possibilities. Chris noted that it was his understanding with the configuration of Rook currently in progress, there will probably be in the range of 7-8 TB of usable space available, so 3-5 TB is possible
Rob asked for some tests to ensure that the whole system whorks end-to-end, esentially integration tests. Ben said that these isn't quite in place right now, but would be a natural thing to set up. Mark suggested he would like to see these run regularly, as a point of comparison when users run into problems.
Mark asked whether v1.0 applies to just the software, or to the service itself. Ben was of the opinion that it would refer to the software, and the service itself would be considered an 'alpha' for the community to try out, but with the expectation that users might encounter issues, and should send feedback to the developers.
Ben noted that garbage collection of output written to the Object Store is an area which will need to be addressed.
Gordon asked whether this is needed for the alpha release, or whether a simplistic solution like dumping the contents of the obect store daily would be sufficient. Ben thought that the simple solution could be used in the short term.
There was some concern expressed about trying to implement anything more complex, which would open up possibilities for bugs, and be harder for users to understand and interact with.
Kevin noted that there's a major differnce between allowing users to do their analysis 'out of' the object store, as opposed to treating it just as a temporary buffer.
Ben proposed moving the transactional integrity ticket to not be part of v1.0. He thought that Grodon's front end library, and resolving Kyungeon's performance concerns should be parts of v1.0.
Rob asked about how the components of the service can be monitored to ensure that they are all running normally. Ben thinks that a combination of Prometheus and built-in Kubernetes features might address this. Rob expressed a preference for check-mk, but agreed that he is more interested in the end result of detecting and debugging problems quickly is more important to him than the particular technology choice.
Gordon reported that Mason has made a new version of the Uproot transformer, which he thinks should be in 1.0. Ben said that he does plan to iinclude this.
Gordon asked whether there is a plan to allow end user access, Ben does have a story for this, which to centered on the work Neha is doing for authentication.
Chris reported that unfortunately Neha does not quite have her changes ready to send as a PR, although she says she hopes to have it ready by later today. In general, Chris does not think that this work is suck, but it is liited by how much time Neha has (competing with classes, etc.)
Kyungeon reported on the performance concerns he has been working on. His motivation was that he saw a long tail in time to transform files, which seems to correlate highly with file size. This is not surprising, but can be a problem because a file is the smallest unit of input the transformer will work with.
To measure, Kyungeon deliberately omits time taken by the DID Finder, etc. so only the transformation time should be included.
The uproot transformer has four major steps: Get an Awkward table frm the Func-ADL generated code, apply a selection to it, then convert toa Pandas Data frame, write an output Parquet file.
Testning outside of Kubernetes suggested that Disk I/O and decompression time dominate the time taken. There was a very large difference in time taken depending on whether the input file was cached.
Gordon asked why there seemed to be a factor of ~2 differnce between the motivating plot and the time to read an un-cached file from a spinning disk attatched to Kyungeon's laptop.
Ilija expressed confusion about the 23 GB inputfile being able to be cached on a laptop with 8 GB of RAM. Others suggested that since Kyungeon does not use all branches, the whole file does not need to fit in RAM.
Most recently, Kyungeon tested using an uproot arrays, instead of lazy arrays (based on a recommendation by Jim). Kyungeon noted that the total amount of data read by uproot is much larger than his comparison 'pure ROOT' code.
Jim suggested that the uproot is somehow touching more branches than it means to, and he think that the sequence of transformations, via a DataFrame, is but unnecessarily complex, and might be contributing to this.
Jim asked whether the test uses local or remote reads. It is fully local.
Gordon wondered if TTree::Draw is managing to do some short-circuit evaluation, which is how it avoids reading as much data. Jim thought this was not the case, based on the total data read, which roughly matches the expectation. It seemed instead that Uproot may be reading data repeatedly. The discussion of what micht be happening became highly technical; Jim speculated on the interactions of chunk sizes usd when reading, local and remote file access, and mmap or file-descriptor based file I/O. ROOT uses traditional file-handles (fopen, fread); Uproot does some different things, including trying to wrap file handle reading to look more like mmap.
Gordon noted that Mason has observed markedly different read times for TTrees of the same size when they are embedded in files of different sizes. Jim suspected that this could be due to the data of individual trees being more interleved and scattered in the larger files. Aftger some confusion it wa determined that Mason and Kyungeon are running the same code, yet they get opposite results on their different machines.