WLCG MW Readiness WG 13th meeting Minutes - October 28th 2015

WG twiki

• MiddlewareReadinessArchive

Agenda

• https://indico.cern.ch/e/MW-Readiness_13

Summary

• The xrootd 4 monitoring plugin issue with dCache v. 2.13.8 was fixed by the developer immediately after the last MW Readiness meeting. ATLAS confirmed the plugin is important for them. CMS finds it 'nice to have', not indispensable.
• One host of the CERN FTS-3 pilot will be running CentOS7. In this way ATLAS and CMS will be able to test FTS3, via their workflows, on both OS environments.
• We need a contact at DESY to arrange the dCache on Prometheus testing.
• NDGF is the only Volunteer Site where the pakiti client installation is still due.
• The MW Readiness App will move to production real soon now. Volunteer Sites will be called to comment on its functionality.
• PIC-CERN PhEDEx transfers failing are not yet understood; they are possibly due to a bug in one of the involved MW components (EOS, dCache, FTS-3, Globus, ...) or a misconfiguration somewhere. The MW Officer and EOS and FTS-3 experts at CERN are looking into this.
• There was an ARGUS Collaboration meeting on Oct. 9th. The next one will be on Nov. 6th. The periodical sudden bursts of high load on the CERN ARGUS servers still persist and are not yet explained. A number of other issues from which CMS suffered are now understood. There is one more FTE now working in the ARGUS dev. team. Moving to CentOS7 is hoped to solve a number of issues currently due to historical dependencies.
• Less than usual participants joined this meeting from the Volunteer Sites. Suggested date for the next one is Wednesday 2nd December at 4pm CET. Objections with alternative dates should be sent to wlcg-ops-coord-wg-middleware@cernNOSPAMPLEASE.ch

Attendance

• Local: Maria Dimou (chair & notes), Alberto Peon (T0), Maarten Litmaath (ALICE & notes), Andrea Manzi (MW Officer), Andrea Sciaba (CMS), Vincent Brillault (security), David Cameron (ATLAS), Lionel Cons (MW Package Reporter developer), Alberto Aimar (CERN IT mgnt).
• Remote: Raul Lopes (Brunel Univ.), Antonio Maria Perez-Calero Yzquierdo (PIC), Vincenzo Spinoso (EGI Ops Officer). Raja Nandakumar (LHCb).
• Apologies: Jeremy Coles (GridPP), Alessandra Doria (Napoli)

Minutes of previous meeting

The minutes of the last (12th) meeting HERE were approved.

MW Officer report

ATLAS workflow Readiness Verification Status:

MW Product	version	Volunteer Site(s)	pakiti client installation status	Comment on testing on CentOS7	Other comments
DPM	1.8.10	Edinburgh	OK up-to-date	test machine setup at Glasgow for this, waiting to complete the Atlas specific configuration	upgraded , waiting for reconfiguration as per https://its.cern.ch/jira/browse/MWREADY-82
StoRM	1.11.9	QMUL & CNAF	OK up-to-date	n/a	problems running panda jobs https://its.cern.ch/jira/browse/MWREADY-61 @CNAF
dCache	2.10.42 & 2.13.8	Triumf (2.10.42) & NDGF (2.13.8)	not yet installed at NDGF	n/a	Triumf all OK. The NDGF verification cannot progress due to a blocking issue with IPv6. A.O.B. Can someone help to make progress with the DESY offer to test dCache on Prometheus as per https://its.cern.ch/jira/browse/MWREADY-36?
HT-Condor (condor-g)	8.3.2	CERN	OK up-to-date	n/a	new HTCondor version released today 8.4.1.to ask for upgrade
FTS3	3.3.3	CERN	OK up-to-date	n/a	new version 3.4.0 to be tested soon
other?

CMS workflow Readiness Verification Status:

MW Product	version	Volunteer Site	pakiti client installation status	Comment for testing on CentOS7	Other comments
DPM	1.8.10	GRIF	OK up-to-date	GRIF plans to run tests on CentOS7 in Q4 2015 as per https://its.cern.ch/jira/browse/MWREADY-71. What exactly can be done now?	problems showed-up with DPM-DSI 1.9.5-8 and gridftp redirection as per https://its.cern.ch/jira/browse/MWREADY-89
dCache	2.13.9	PIC	OK up-to-date	n/a	PIC please comment on status in https://its.cern.ch/jira/browse/MWREADY-91
EOS	0.3.129-aquamarine	CERN	OK up-to-date	n/a	Issue with PIC -> CERN PhEDEX transfers as per https://its.cern.ch/jira/browse/MWREADY-81
FTS3	3.3.3	CERN	OK up-to-date	n/a	new version 3.4.0 to be tested soon
ARC-CE	5.0.3	Brunel	OK up-to-date	n/a	Issues in the glidein factory as per https://its.cern.ch/jira/browse/MWREADY-84

WLCG MW Readiness Software Status

Decommissioning of the old MW PKG DB nodes:

• only 2 nodes still publishing to the old collectors, (one test node from LHCb and one from LAL), we have contacted the responsibles but not reply so far
• the old MW PKG DB nodes are going to be decommissioned by the end of the month

Recent and on-going development items by the MW Officer:

• https://its.cern.ch/jira/browse/MWREADY-69: Implemented view for Volunteer site managers to see products tested at their site on the https://mw-readiness-dev.cern.ch/sites/ App.( integration with GOCDB for authz)
• https://its.cern.ch/jira/browse/MWREADY-92: Changed the MW Readiness App to return a more comprehensive error message in some cases of querying the pkg db for the product version.

Next:

• Deployment of the Volunteer site view in production

Sites' feedback

• PIC report:
  • Pre-production storage using dCache 2.13.9,
  • Running with the most recent xrootd monitoring plugin (now available in the WLCG software repository).
  • Phedex transfers:
    • currently using PhEDEx: 4.1.3-comp3, moving to 4.1.7, as requested by CMS
    • agents configured to use FTS3 pilot at CERN
    • links: PIC->CERN and PIC->GRIF_LLR not working for the last weeks.
    • links: GRIF_LLR -> PIC OK, CERN -> PIC not working for the last weeks
  • HC jobs running fine

• CNAF
• Napoli
• Edinburgh
• QMUL
• Brunel: Raul reported the glidein issues experienced so far are now fixed and the test pilot jobs started appearing.
• CERN
• ... other sites

Report from the ARGUS meeting

• Argus meeting held on Oct 9th agenda with minutes. Maarten's notes on 12/10:
  • Investigations of tickets related to Argus at CERN:
    • SAM test failures due to gridmapdir corruption (GGUS:116468).
    • CMS VOBOX unexpected usage of host proxy instead of user proxies (GGUS:116092).
    • User mapping failures due to lack of lcg-expiregridmapdir cron job (also GGUS:116092).
    • CREAM wrongly remembering Argus troubles for many hours (GGUS:116791).
  • Argus meeting. Gridmapdir will become replaceable by a simple DB.

• main points for MW Readiness:
  • Argus team expansion
    • 1 FTE started working with the ARGUS dev. team. The other position is not yet covered.

Actions

Action items Done from past meetings can be found HERE.

• 20150916-01: Andrea S. & David C. to ask CMS and ATLAS whether the xrootd 4 monitoring plugin is important for them. DONE Right after the last MW Readiness meeting, Ilija Vukotic developed a fix that solved the issue with dCache.ATLAS position: the xrootd monitoring plugin is important for them. CMS position: nice to have but not indispensable.
• 20150506-03: NDGF to install the pakiti client. Updated instructions here. Pending.
• 20150318-02: Ben to set-up the ARGUS testbed at the T0. The testbed is there, the load testing is in the list but Postponed
• 20141119-03: Andrea M. to contact the GRIF site to proceed with WN testing via the CMS workflow POSTPONED

Next meeting

• Suggested date is Wed 2nd December 2015 at 4pm CET. David Cameron is away on that date. Other members unavailable, please email our e-group wlcg-ops-coord-wg-middleware@cernNOSPAMPLEASE.ch a.s.a.p.

AOB

-- MariaDimou - 2015-10-12