ServiceX working meeting
Service X Call May 15, 2020
----------------------------------------
Attendance:
Alexx P, Ben G, Chris W, Gordon W, Ilija V, Kyungeon C, Marc W, Mark N, Peter O, Rob G
----------------------------------------
https://indico.cern.ch/event/901359/contributions/3867217/attachments/2039975/3416188/2020_05_15-TreeMaker.pdf
Alexx presented on the status of the (CMS) TreeMaker Transformer.
TreeMaker is a CMS tool which writes flat ROOT trees from miniAOD data files. Analyzers use it when they don't want to use the official nanoAOD files; the output it produces is fairly comparable.
Turning TreeMaker into a ServiceX transform required making a Docker image containing the CMS software distribution, which is normally obtained through CVMFS. Fr the prototype the input file is embedded in the container image, to avoid needing to handle grid authentication. The ServiceX code for the transformer currently requires the container to have a directory /home/atlas, which is a bit odd to require for a CMS transformer. Ben mentioned that he has a ticket open to clean this up.
Alexx set up automated builds for this container image on CERN GitLab. Some unusual wiring was required to interface between GitLab where the main CMSSW code is located, and GitHub where the TreeMaker code is.
Currently the image has some issues initializing the CMSSW environment, but Alexx has a plan for resolving this, so he hopes to have this fully working soon.
Gordon asked what fraction of the 'normal' CMS analysis is run in containers. He was curious to what extent Alexx and Co are having to break new trails. Alexx said that this is essentially new territory for CMS, almost no one does this sort of thing otherwise. However, there has been an effort for data preservation which gave them a good starting point for their container. Central production uses official containers, but this is new for individual analysis.
There was some discussion of how much this is done in ATLAS. Apparently it varies quite a bit across the collaboration, but there is a concerted effort toward it.
Alexx hopes to also create more generic tools for putting analysis code into containers.
Mark asked, of the challenges Alexx encountered, what fraction were inherent to (the current state of) Service X, and what were from the CMS side. Alexx estimated it at 90% CMS, although the remaining blocker is a ServiceX issue.
Ben noted that all of the existing transformers were relatively quick-and-dirty efforts to get ATLAS data flowing, which is the source of historic baggage.
----------------------------------------
Ben asked Kyungeon about the status of the Uproot transformer updates.
Kyungeon summarized that the conversion of the Awkward table to Arrow is the core operation that he is working to optimize. He has been collaborating with Jim P to improve this. The ServiceX Docker image includes their latest fixes.
He is also trying to make a new transformer which produces flat ROOT 'n-tuples', which can generate new data columns.
Gordon noted that such operations are in many cases already possible in the current Uproot transformer. He already does this in his own usage. Since each column is defined as an arbitrary expression, they need not take a single existing column as input; every column is a 'new' computed column. He noted, however, that ServiceX is not supposed to do heavy calculation, so complex calculations have been treated as out of scope for FuncADL.
Ben noted that issue #131 on the table conversion has been closed, to reflect that task being done. There is another, more open-ended ticket, #124, about the speed of the Uproot transformer, and he wanted to check on its status.
Gordon suggested that after the fixes for #131 performance should be retested, to determine whether it is now acceptable and #124 can be closed.
Kyungeon thought that the status is probably good enough for v1.0, but he would like to continue working on this topic.
Ben asked Kyungeon to add a comment to the ticket with his recommendations for future work, which might be the basis of another ticket later.
Peter would like to resolve the issue where the Uproot transformer seems to read more data than necessary, since he feels that may be a long-term problem.
Ben suggested that if the issue exists in Uproot generally, it should be addressed upstream.
Gordon suggested constructing a test case which Jim can work with while he is rewriting Uproot.
----------------------------------------
https://indico.cern.ch/event/901359/contributions/3795354/attachments/2039972/3416182/weinbergServiceX200515.pdf
Marc showed a summary of concerns he had looking around the ZenHub board and thinking about reaching the v1.0 release.
He had two broad categories of open issues: What is required for outside users to access the service, and to 'do good physics'.
For general usability, Marc sees a need to have a stable version of the service that the team will commit to maintaining, including preparedness to address issues on ~day timescales. A sub-concern is how much storage will be needed to support a nominal set of users. He observes the extracting 100 columns from 10 TB of input produces 300 MB of output, so that may suggest the scale of what is needed.
He is also concerned about the Ingress and Auth work, and the Frontend library.
Ben reported that Neha's auth patch is in, but currently anyone can request and get an account, so Neha is now working to add user approval. The ingress is also in, protecting the internal parts of the API and bundling MinIO together with the rest of the interface.
Gordon reported that to support Kyungeon he has a checklist of tasks, on which he is making steady progress. Another aspect being developed is logging/status reporting. Once those things are ready he would like to do an alpha release and have Kyungeon test it out. He expects this to be ready very soon.
Another feature that he is working on is local caching, which will make repeated requests for the same data efficient. He has been testing this himself for a week and a half, and is revising based on his experiences.
Ben has requested some refactoring of the interface.
There are some things which still need to be started: Primary among these is integrating with the new authentication scheme.
Mark N asked who validates user accounts, and how data access is managed.
Ben explained that until now accessing the service required having direct access to the Kubernetes cluster. The new auth system issues JWTs which are valid for 6 hours, and the API requires this token to be sent in the header each time. A missing piece is that currently accounts are not validated, so adding a workflow for doing this is a high priority. Another known limitation is that MinIO has one set of credentials, so any user can read any data from it.
mark asked at what stage data access is authentication; can any user access any dataset to which Service X has access? Ben answered that currently ServiceX holds a single set of credentials (currently his own), and uses those for all data access. There will be separate ATLAS and CMS instances of Service X, which keeps the data segregated. He characterized this as using a minimal amount of technology and otherwise resting on process controls.
Mark felt satisfied that for now CMS and ATLAS data would not accidentally be leaked across the experiment boundary, although he thinks further integration with other auth mechanisms will be desirable in the longer term.
Chris confirmed that Neha is working on adding the account approval support, and should be have that ready for v1.0. Integrating with external identity providers is somewhat orthogonal, and has not been started yet, but should definitely make it possible to give users access with credentials they already have.
In terms of supporting 'good physics', Marc thinks that being able to apply Jet energy corrections, filter on triggers, and other such typical tasks.
Gordon described that he has an open ticket to implement the corrections, but there are technical hurdles in terms of how slow correction application code is (in its default mode). He is talking to an expert on how to run this efficiently.
He also noted that filtering on triggers is not a special type of operation as far as the Uproot transformer is concerned, since it already supports user-defined filters as long as the input data contains the properties needed for the filter decision. He has a student that he hopes to assign to creating examples to show users how to do this sort of thing.
----------------------------------------
Ben noted that autoscaling, and MinIO data retention policy are some of the other main tasks open for v1.0. He and Marc thought that the end of the month still seems like a realistic target.
Gordon opened an issue for implementing auth support in the frontend, and asked Ben to add some details to it of the protocol which must be implemented.