Choose timezone
Your profile timezone:
Developer's meeting
→
Europe/Zurich
present: Milo, Enrico, Roland, Giovanna, Andrea, Eric, Frank
Enrico:
- using the f/w from last Thursday, checked together with Andrea, especially the failure after transmitting some given #blocks
- found (and fixed) incorrectly set FIFO thresholds, but issue persists
- still investigating, but need input from Frans/Andrea
- multi-DMA: f/w OK, but after build find that no data flowing, even with internal data emulator on the VC709
- on the BNL711, internal emulation works but not input from WIB
- Andrea: Frans made changes to link interface, this could affect issue with BNL711 reading from WIB?
- Giovanna: should prioritise for multi-DMA: there is no way to run without this achieved
Andrea:
- super-chunks: have identified faulty f/w block but still need to fix
- larger block size: affects other FIFOs, FIFO thresholds
- changing FIFO settings leads to loss of headers etc. (not yet known why)
- going beyond 2 DMA descriptors per DMA engine leads to timing issues
- multiple descriptors / DMA engine: reduced FIFO depth per descriptor
each FIFO represents a high-speed "net", with accompanying constraints on DMA to PCIe - more FIFOs => need them shallower to meet timing constraints
- can ask Frans what the limitations are
- multiple descriptors / DMA engine: reduced FIFO depth per descriptor
- main FIFO now is 128 kB, corresponding to ~ 300 frames (or 30 / link, if we used 1 DMA descriptor/link: good for 15 μsec only)
- need for deeper FIFO is PCIe downtime (unpredictable); Andrea et al. have previously seen issues with a single-descriptor 32 kB pipeline
- PCIe reliability thought to be non-trivial to tune on s/w side -- worth asking Jos
Roland: worked with Jörn on software items
(see also FELIX_perf_issues.pdf)
- managed to readout at 2 MHz of two links, using two DMA engines, but only once:
- not reproducible; lost 300 kHz, after the reprogramming of the FELIX and the reboot of the host
- there are no BIOS, systemd, NIC configuration changes that could induce the issue. (Ensured via trying out several configurations to reproduce the expected rate.)
- even with Infiniband, miss the 2 MHz by 50 kHz (that indicates NUMA locality issues)
- scatter/gather (to reduce memcopies) did not help
- analysis shows higher than expected QPI rate, so handled data is not NUMA symmetric
- with the current FULLMODE baseline firmware, not even a single link at the 2MHz is sustainable on the sw side, in a stable manner
- Giovanna: less concerned about s/w side since it can adapt easier
- need NUMA-aware driver? Have contacted Markus Joos about this
- if we experience limitations due to software, during scaling out to 10 links, hardware changes of the hosting server are easier and straightforward compared to fw development efforts
There are minutes attached to this event.
Show them.