Choose timezone
Your profile timezone:
Madgraph5 GPU development
→
Europe/Zurich
Virtual (Zoom)
Virtual (Zoom)
Stefan Roiser
# Meeting Monday 07.06.2021
Present: SR, OM, AV (notes), Josh, Smita, Walter, Carl, Andy, Laurence, Taylor, Stephan
## Round table (1/2 while waiting for Taylor)
Andrea
- working on vCHEP paper
- did some tests of Xeon Silver vs Xeon Gold for AVX512 following a discussion with Hadrien Grasland... no improvement yet, to be followed up
- WIP on separating AOSOA neppM from SIMD neppM layouts: in eemumu memory acces has more overhead than expected (expect less in ggttgg?...)
## Taylor's presentation
A few general comparisons of CUDA and Kokkos code (eg memory layouts)
Ported ggttgg epoch2, working on gitlab CI at Argonne JLSE (Nvidia A100/V100, AMD MI50, Skylake, Intel Aurora GPUs)
Run every night Nvidia CUDA and also kokkos.
Throughput: kokkos is ~0.40 for eemumu, but ~1.00 for ggttgg for both V100 and A100.
He imagines that for eemumu the oevrhead is from kokkos itself in launching kernels.
Q/SR: same architecture as our CUDA, eg 1 big kernel for sigmakin?
A/TC: yes tried to do exactly the same
Q/SR: what if we move to CUDA graphs to have smaller kernels?
TC: not happy about multithreading performance on skylake c++ in eemumu, should see 28x (28 threads) but only see a bit more than 1
Q/AV: do you specify pragma parallel for somewhere?
A/TC: not as pragma, but specify a "parallel for" in the sigmakin kernel, which is translated either on cuda kernel or openmp pragma
Comments/AV
- C/AV: slide 8 find left plot more interesting; the right plot compares to fortran, may depend on fast math, O3 etc
- Q/AV: chose eemumu to minimise handwritten code, did you have to modify modified oxx/ixx or ffv?
A/TC: had to modify the big momentum array, but it was enough to do a simple grep replace
- Q/AV: any vectorization in this? A/TC: no not yet, looking at this now
Some profiling plots
- slide 9 nsys, eemumu clearly is not compuationally intensive
- slide 10/11/12 ncu
C/AV: about memory, look at sectors vs requests/transaction
C/OM: also look at occupancy and #registers (AV: smaller kernels would achieve this)
A/TC: nvidia expert was suggesting to paralleize ME and split it in several kernels
TC: note that kokkos implementation does not allow switching from double to float
OM: do not care about FP128 on GPU, will do openloops on the CPU
TC/slide 14: note that in eemumu consider throughput as only sigmaking (not the memory copies)
Q/SR: does kokkos give handles for shared memory?
A/TC: yes it does but still looking at that
Q/AV: does kokkos allow you to do cuda graphs?
A/TC: maybe
C/OM: we could also try helicity parallelism and not only event parallelism
problem is that it is not always a multiple of 32, not even a multiple of 2!
In that case however there is a lot you can put in shared memory
Discussion
- TC: could kokkos become another backend?
OM: yes could be, poroblem is that I am the only one doing the backports
OM: if someone wants to learn (eg TC for kokkos and AV for cuda) I am happy to teach you!
JMF: dont forget that we have traditional resources and eg we'll use
AV: at some point it would be nice to go to "epochX" where we continuosly backport and regenerate
SR: yes agree would be nice to go to self sustained mode?
SH/AV: Taylor please upload your slides in indico
## Carl's presentation
Results on CPU and GPU for both eemumu and ggttgg epoch2
Slide 3
SH/AV/OM: very surprised by the <0.01% spent in sigmakin ggttgg for GPU, this cannot be...
Slide 4
AV: is IPC CPU side or GPU side? would expect IPC so low on CPU while GPU is busy..
CV: this comes from nsight compute
SH: if it is from ncu, it is probably GPU side, agree it would be nice to investogate
AV: agree, maybe this is eemumu?
OM: for complex processes, IPC should be much higher
Slide 6
Again, IPC very low, scheduler only issues 1 instruction per 60 cycles!
Slide 7
Varied max registers, best is with the highest, 160 registers
SH: did you try higher than 160? it would be useful
AV: yes, unlimited probably means 255, whoulc try that too
Discussion
- SH: have a look at how many FP64 units you have, maybe you are using a card with few FP64?
CV: it is a tesla T4
SR: well spotted Stephan, this has few units
AV: yes 1/32 of FP32, maybe try also to compile in float
## Round table (2/2)
OM: working on next step, integrate gpu with madevent to have full phase space integration
AR: still working on plots
SR: Andy will defend his thesis on July 28, everyone is welcome!
JMF: was on holiday last week, previous week was at ATLAS gpu workshop with some nvidia people, was very useful
SH: also holiday last week, nothing to say
WH: not much, but would be intersted in attending the meeting on code generation!
SR: still working on integration of fortran with c++/cuda
would propose to olivier a memory layout in fortran so that it fits with andrea's AOSOA?
OM: this would not be convenient at all... have a lot of memory to handle in madevent
SR: ok then maybe we'll create a kernel on the GPU for doing the transpose
AV: or another option is using the pIpar functions that feed data to the ixxx/oxxx functions?
LF: used VTune to investigate AVX512 performance differences, will discuss with Intel at the next meeting.
## AOB
Agree to have weekly meetings sometimes if/when necessary
Next meeting next week, in particular
AV: may be unable to attend on Monday 21/6
TC: on holiday next week
OM: will set up a doodle for another special meeting on python code generation
Will try to set this up as a tutorial where people work on doing things!
Interested: TC, AV, SR, WH, JMF
There are minutes attached to this event.
Show them.