Choose timezone
Your profile timezone:
Developer's meeting
→
Europe/Zurich
Present: Brett, Phil, Enrico, Roland, Giovanna, Milo, Frank, Thijs, Eric
EHN1 (Enrico, Roland, Giovanna):
- hardware:
- now have the Nikhef server in place to host the VC709 card (connected to BR host using copper cable; Mellanox cards are used for the BNL711 host)
- new quad server has just arrived at CERN
- firmware: need to get this back in a state where at least 8 BNL711 links can be made to work properly (this was working at some earlier point)
- tests yesterday indicated that only 6 links could work; but Roland established today that 8 links should indeed be feasible
- present status: 8 links (should be) possible when using superchunks only; 6 links when using superchunks + jumbo blocks
- VC709 firmware: this was built under Vivado 2015, and Vivado 2018 is known to improve significantly over this
- at present can only use 2 out of 4 links; need a rebuild to see if this improves the situation
- Andrea will be at CERN next week to work with Enrico and hopefully fix remaining issues
- Milo is using the existing FELIX data (taken with 6 links?) to be able to provide quick feedback on noise etc.
AVX (Thijs):
Presentation (see slides for details):
- studied implementation of re-ordering code using both AVX2 and AVX512 instruction sets for parallellisation
- notes on slides:
- plot legends: "i" prefix (e.g. "iavx2") indicates use of Intel compiler (icc) rather than gcc
- re-ordering rate/core (main performance measure) means number of 10k frame re-orderings (so 300 Hz means 3M frames/sec)
- AVX512 also offers scatter/gather functionality; consider groups of 4 frames to exploit scatter
- profiling tests indicate that when using AVX, most time is spent filling registers
- main observations:
- Broadwell CPU (no AVX512 capability): 80 Hz w/o AVX → 160 Hz w/ AVX2; but drop-off when > 2 cores used
- slightly better performance using icc than using gcc (icc actually uses AVX2 internally)
- Skylake CPU (w/ AVX512): 150 Hz w/o AVX → 300 Hz w/ AVX; no drop-off
- somewhat better performance using AVX512 than using AVX2 (conclude that icc does instruct for AVX512 behind the scenes even when asking for AVX2)
- overall factor of ~ 2.5 in speed possible; this seems compatible with the 300 Hz ⇒ 3.3 msec/event (Milo observed 6 to 8 msec/event previously)
- Broadwell CPU (no AVX512 capability): 80 Hz w/o AVX → 160 Hz w/ AVX2; but drop-off when > 2 cores used
Discussion:
- Frank: readiness of code?
- validity of output has always been checked against Milo's outputs
- intend to integrate code in FELIX BoardReader code and with compression tomorrow; it should be possible to make it easy to switch between different compression schemes
- Brett: validity of error codes and checksums?
- frame CRC is checked by FELIX firmware (but not sure what happens with the result of the check..)
- COLDATA CRCs at present are not checked; would be good to do so (as it will be very impractical to try and reconstruct the original frame later)
- Roland: will do, but not at highest priority
- Phil: any use for AVX gather functionality?
- Thijs: looked into this but could not find a practical use
- Brett: is using 300 Hz going to be good enough for DUNE?
- for continuous readout, 300 Hz * 5 msec/event = 1.5, so need ~ 66% of a core's resources; still seems quite high
- indicates that for DUNE it will be worth simplifying the frame format in such a way that bit re-ordering (and especially bit shifting) becomes unnecessary
- Giovanna: also plan to have someone working on re-ordering in firmware
- also: if for DUNE we need the collection-plane outputs only, this will reduce the re-ordering needs accordingly
- Giovanna: can we use the ProtoDUNE infrastructure to make progress on more complete tests for DUNE?
- easiest way seems to be for Phil to append his hit-finding code onto Thijs's re-ordering code (the BoardReader code is available on the redmine server)
- this will then run in the BoardReader, i.e., artdaq format, so cannot extract the resulting information directly (but for now it should still be possible to report useful summary information)
AOB:
- have the PCIe QAT card back; this can be used e.g. by Milo to complete compression studies on μBooNE data
- should also compare achieved compression using this card with the one having the FPGA on-board (there have been reports that the on-board compression is significantly worse)
Next meeting: will decide early next week whether it is useful to schedule another meeting already next week, otherwise will fall back to August 15
There are minutes attached to this event.
Show them.