ServiceX working meeting

→ US/Central
Benjamin Galewsky (Univ. Illinois at Urbana Champaign (US)), Marc Gabriel Weinberg (University of Chicago (US))
    • 12:00 → 12:20
      Action items 20m
      Speakers: Benjamin Galewsky (Univ. Illinois at Urbana Champaign (US)), Marc Gabriel Weinberg (University of Chicago (US))

      Service X Call March 20, 2020

      ------------------------------------
      Attendance:
      Alexx P, Ben G, Chris W, Ilija V, Kevin P, Kyungeon C, Marc W, Peter O

      ------------------------------------
      Neha, the student at U. Chicago who is working on this has had her schedule badly disrupted by being kicked out of her dorm during finals. Her last final is today, after which she has two weeks of spring break, which should allow her to make some more progress. 
      Ben asked if she could give a presentation on this call in two weeks. He noted that a number of his other tasks are now cleared up, so if the authentication work is stalling he could help. 
      Chris agreed that he would ask Neha to work toward giving a talk about her design for this work and hopefully some amount of the implementation. He will also ask her to work in the direction of splitting up tasks so that more than one person could work on them. 

      Kevin et al. are trying to do a project to use ServiceX to deliver CMS data they are using. Their immediate goal is just to wire it into some existing software, to provide the same n-tuple files that they are using now. Once they have that working they will try to refine the approach to select just the necessary data columns. Right now CMS has a Tool called Crab which isn't very well maintained, so the may be a good opportunity for ServiceX to step in and provide a shared and better maintained tool. Kevin is waiting on an email from Marc with instructions on how to use the accounts they have been given. Marc promised to write them a small tutorial to get started. 

      After recent improvements Marc has begun doing some more timing tests. He is testing with the 10 TB dataset, running with 400, 1000, and 3000 workers, requesting either 10 or 100 columns, and writing output either as ROOT files to MinIO or Arrow arrays to Kafka. 
      Ilija raised a possible issue that this dataset is no longer registered at U. Chicago. No one knew why this might be; Marc will investigate. 
      With 400 transformers, Marc sees best-case times of ~5 minutes for 10 columns, and ~20 minutes for 100 columns. He did not see much advantage to using 1000 transformers, he thinks because there were large, unrelated loads on dCache. With 3000 transformers he could not get all transformers started for the 10 column request before the processing was completed. For the 10 column request he found that it took about 13 CPU-seconds to process each file, ~1 second to write the output, and ~3 MB of output produced. For the 100 column request the processing time was ~30 CPU-second/file, and the output is closer to 35 MB. There appears to be some unknown slowdown when writing the larger request to Kafka, compared to MinIO. 
      Files are now requested from a preferred site (MWT2) when possible, Rucio access times are no longer reported as this seemed to cause some slowdown. Running multiple requests at the same time puts the greatest strain on the input source. Help from Judith has optimized the use of dCache substantially. 
      Ben noted that there are as-yet no users of Kafka. It has built-in replication, but this is associated with a 15 MD message size limit. At the moment the team has the replication turned off, since it seems unnecessary, so it should be possible to raise the maximum message size. Ben suspects that this may help efficiency. 
      Ilija would like to know data rates (MB/s) as the most relevant way to know whether available bandwidth is being fully utilized. 
      Kyungeon asked who chooses the preferred site for reading data. Marc said that this is currently selected by the ServiceX administrator. Ben thought that this makes the most sense, since the admin knows where the processing work is physically located. Kyungeon was still interested in being able to control this for performance testing. 

      Kyungeon is working to interface ServiceX with the ATLAS TRExFitter. Typical inputs sizes may exceed several TB. Right now, Kyungeon has built a shim which reads the same configuration files as the existing TREx input code to determine what input data to provide. This has required implementing a compiler for TCut to FuncADL and Qastle. It is notable that a typical selection requires reading 47 branches, but produces only one as output. Kyungeon has been testing to ensure that the old and new methods of obtaining input produce the same output; so far all results match correctly. In the datasets being tested, the sizes of the input files can vary greatly, by up to three orders of magnitude, for example. It is the few large files which dominate the total time to provide the data. Using ServiceX on ~0.5 TB of data from 5 DIDs at MWT2 is about 4 times slower than running the fitter at CERN on lxplus, but about 4 times faster than running locally on a machine at UT. 
      Peter was surprised at the speed of processing on lxplus. Kyungeon noted that he is running serially on lxplus, and it is quite slow the first time, but becomes much faster, so it appears that some type of caching is involved. Peter thought that this indicates some inefficiency in the ServiceX transformation if lxplus can really load and process the data so much faster. 
      There was some question exactly what TRExFitter is being used to do. Kyungeon described that at the moment he is just using it to load the data and produce a histogram. He could do a fit, but that is a separate concern from ServiceX performance. 

      Alexx has a first version of a TreeMaker container, but it is still much larger than he would like, so he is still working to slim it down. He isn't familiar with how containers are used by ServiceX, but he is writing up documentation of how his container is designed to be used which the ServiceX team can hopefully use to learn to adapt it. Alexx was also unsure how to handle CVMFS. Currently its cache is taking up ~1 GB of his image, so he would like to clean it out. He has designed his container to be able to mount CVMFS from the host system. Chris noted that on River CVMFS is provided by a Kubernetes volume provider, however since Alexx already has CVMFS installed in his container, another option might be to point it to the squid cache otherwise being used for CVMFS, to retain at least some of the benefit of cluster-wide caching. Ben did not like the idea of the software in CVMFS being able to change, independently of the image, and ServiceX currently has no way to support doing volume mounts for transformer pods. He directed Alexx to bake all necessary contents of CVMFS into the image. Alexx will try to do this by running test which will probably touch every necessary file. 

    • 12:20 → 12:40
      ServiceX for TRExFitter 20m
      Speaker: Kyungeon Choi (University of Texas at Austin (US))
    • 12:40 → 13:00
      Timing tests 20m
      Speaker: Marc Gabriel Weinberg (University of Chicago (US))