AF Pilot - Informative session for early testers

Thursday 18 Apr 2024, 16:00 → 17:00 Europe/Zurich

31/S-027 (CERN)

Present in the room:

Enric, Markus

Zoom:

Ben, Jamie Gooding, Davide Vasecchi

 

Only a few users could connect because there was a collision with other events…. 

Thomaso Tedeski missed the call. 

The meeting started with the presentation by Enric, which is attached to the agenda.

It covers the general principles, the interactive layer and the technology under the hood. Coffea and RDataFrame, both support DASK, which can scale out to Condor.

The presentation contains a walk through of how to get started, where to find the documentation. 

It was also shown how you can reconnect to your session and how the migration to batch works.

 

There was the general impression that this should be good enough to get started, given that you have prior experience with  RDataFrame.

The users have been encouraged to report back with their experience. Issues and general impressions. 

 

There was some discussion on how extra packages could be installed. pip install works but it isn’t clear  how they are propagated. The same software environment on both sides is needed. 

Enric pointed out that we have curated packages only.  Right now it  is isn’t implemented to support your own packages. If custom packages are needed we have to look into it. 

This was followed by a discussion on how this can be done. CERNBox pip install. Then the python path needs to be monitored. On the condor side we would need the EOS user space to be there too. 

Ben reassured the audience that this is there, but the path might be different between batch and in SWAN. The namespace could be translated. (this needs to be verified)

Best solution would be an option on SWAN to send the packages on the worker nodes. 

 

One of the early users (I didn’t get the name) pointed out that they can start to run on top of the LCG stack, but that they need to be able to add. This inludes also the coffea version. LCG brings ONE version, they might prefer a better one. 

 Enric remarked that a stable LCG release has the advantage that we know that the versions work together. With flexibility comes the potential for incompatibilities. Maybe they can  start on top of the stable release and then report their experience, either by mail or by the mattermost channel on which all can get support.

 Jamie, LHCb, wanted to know what feedback is required, the user experience, or a more formal benchmark result? The latter would be much harder.

 Enric clarified that we take what you can provide. If you have some ready analysis please let us know how it worked. If you have something that scales out to 300 cores let us know too. Ben can make sure that the resources are there at the necessary scale. 

 Jamie: We can do a quick test, anything more would take a while.

 Enric addressed the users:

 Are you all on the mattermost channel?

We will make sure that you all are on the channel. If you have questions later that is the best place to post them. 

We are very interested in the feel and the time behaviour that you experience. 

Please start a trivial test and then take it from there.

If you have experience with other analysis facilities let us know how it compares. 

We really need the feedback from early testers!

Let us know if you need anything!

There was a brief discussion at the end of the level of backwards compatibility of different versions of coffea. It is still under rapid development and therefore backward compatibility is poor.  

16:00 → 17:00 Discussion

In this session we will explain to the early testers of the CERN AF pilot what is expected from them and answer their questions.

AF@CERN Pilot - Slides for early testers.pdf