ATLAS and CMS tested RAL with intention to fill the OPN (200Gbps). We struggled to do this, at first blaming production transfers, e.g. small files taking the I/O, tape recalls etc. But looking more closely at individual file transfer speeds, these seemed to have a distinct split between ‘good’ transfer rate and ‘bad’ transfer rate. During the test, good rate was ~30MB/s. Bad rate was <5MB/s.
Jyothish tracked down the split to be correlated to the 25G and 100G NIC gateways. The 100s did the good rates, the 25s did the bad rates. Another observation was that the 25s had a huge number of connections - Jyothish tried to correct this by changing the weighting on the decision over which gateway a new connection would be assigned to, but this had little effect.
Further testing over the following days and using the same system but without the high load of the test confirmed the differences between 25s and 100s. Files were transferred one-by-one, and although the absolute rate was higher (for both gateway types) the difference between them remained very noticeable, e.g. a factor of 5 or more. We do not believe this should be the case.
Addition of jumbo frames to one of the 25s doubled the rate there (although of course the same addition on the 100s had no effect - as perhaps expected for a short connections like CERN to RAL). So this is not the solution.
Another observation is that my (slightly dodgy) ‘per-file rate’ monitoring shows a big drop in average file rate on 28th September. However Jyothish has not seen any significant change around that date.
Only writes affected...reads seem similar across the different gateways.
Jyothish is investigating further...