IT-protoDUNE coordination (Single Phase and Double Phase)
# Assistants
Room: Geoff, Cristi, Xavi, Marc Collison, Ignacio.
Vydio: Stu, Giussepe Lo Presti, Steve Trimm
Apologies: Elisabetta, Dennis
# Round Table
## Geoff:
- 160KV, no beam. hopefully back tomorrow. Water has been found in the HV system, don't
know why. It has been taken apart and reviewed and looks fine now.
Tomorrow will be increased at 180KV.
Q(X): The argon is not yet fully purified.
## Giussepe
Regarding Castor issue:
- Yesterday afternoon there were many errors of different categories, some of them
still being debugged, causing to many parallel retries and congestion (probable
root cause xrootd)
- Activity was stopped and moved to a different pool (the one for data taking)
Throughput is ~1.2Gi which is still a bit low. EOS side is still kind of congested.
Some of the errors point at errors coming from the transmission source.
Remark(ST): FTS does not retry, they are internal retries
GLP: Our FTSs are not completely the same
Q(ST): What happens when there's a timeout during transfer?
A(GLP): In this condition a file should be already created, and there are parameters
that can be use to force a write or to delete the pre-existing file in the
destination.
Action: Follow up with Cristi and Andrea Manzi (from FTS).
Xavi: Proposal: Assess the current situation by measuring queues and conditions
Ideal conditions ~12GB/second
A few transfers always going on. Around ~40minutes for files of that size?
ST: We can't deduce which endpoint (Castor, EOS servers) are actually transfering
the files.
Request from Cristi to get some of the filenames to debug the issues
Q(X): Can we increase the Castor parallelism up from 26? (up to 50 from EOS to Castor)
to try to take out the backlog?
A(GLP, Cristi): Good for us
If the FNAL and Castor transfers are happening at the same time they can exhaust each other
resources and result in the operation being bonded by throughtput of the local disk
Q (Cristi) EOS to FNAL is fine?
A(ST): Normally is doing OK, yes
## Marc Collison
A certain router is not responding to SNMP. An incident has been raised to the vendor
(Brocade). It has been suggested to upgrade the firmware, but it comes with a full
review of the ACLs which requires changes on CS configuration management system
part that can not be done at the moment.
No impact on the user side should be seen.
G: These alarms seem to be triggered when tests with a special firmware by Bruckhaven is deployed.
Is it possible to relax the alarm conditions (at the SNMP level) to reduce the number
of alarms?
A (MC): Yes, that will probably be doable before we are able to deploy the new firmware.
Q: How many RJ45SFP+ do you need?
A: Geoff: 4
## Ignacio
WinCC ProtoDUNE (PDBR db): Invite ticket caller to next meeting + DB Expert to better understand
the issue
## Cristi
Q: Bouncing around of certain files in EOS that are being copied (with 3r-party-copy) to change path.
Could this be changed to a file move/rename?
A(ST): Needs to be seen. The use case kind of needs the two copies (due to synchronicity of both copies)
Q: What happens with the data in EOS when you copy it out?
A(ST): Some of it will cleaned.
-- Parallelism changed ~15:40PM
Q (Regarding Slow Control)
There was a EOS Web page hosted in EOS USER. Should this be moved to EOSPUBLIC? (~350GB)
Q(G): EOS doesn't seem to be ready to be used to on user space, is it ok for web pages?
A: Problems only with high iops operations (writes) on FUSE
-- Elisabetta (offline information sent)
1) event builder: Denis had some problems in installing CentOS7 via pxe, (related to the particular configuration of these machines)
A ticket has been opened--> the problems have been understood and solved.
2) batch farm: yesterday 10 chassis DELL C6220 have been delivered from Lyon.
Each chassis contains 4 computing servers (x2 processors Intel E5-2670, 96 GB RAM, 2x500GB SATA disk system)
These 40 servers have been installed and cabled. A dedicated service has been created by Marc, a switch has been installed by Giovanna, and its configuration is going on (many thanks to Giovanna and Marc)
Next meeting:
(Next week) 27th