RAL Tier1 Experiments Liaison Meeting

Wednesday 6 May 2020, 13:30 → 14:30 Europe/London

Access Grid (RAL R89)

13:38 → 13:39 Major Incidents Changes

13:39 → 13:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20200506.docx

RT1EL-20200506.pdf

WN squids updated.

Brief partial drain whilst condor hosts were updated

Test FTS  on SL7 test by CMS . OK!

https://fts3-test.gridpp.rl.ac.uk:8449/fts3/ftsmon/#/?vo=&source_se=&dest_se=&time_window=48

 

13:40 → 13:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:41 → 13:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:42 → 13:43 Experiment Operational Issues

13:44 → 13:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

* Job Walltime Ratio:
Pilot determines CPU time and Wall time as:

The Pilot reports CPU timing information on every server update (getJob and updateJob). The measurements (system+user time for all child processes) are done during running approximately once a minute (using /prod/pid/stat) and a final measurement done immediately after the payload has finished (using os.times()).
Given an initial t0, user+system time is calculated like so:

t1 = os.times()
user_time = t1[2] - t0[2]
system_time = t1[3] - t0[3]

The instant CPU timing calculation extracts the system+user time from /proc/pid/stat for a given pid (using os.sysconf_names[‘SC_CLK_TCK’] for conversion) and loops over all child process stat files. 
For technical details, see processes::get_cpu_consumption_time() and processes::get_instant_cpu_consumption_time().

This means (?) threading is not accounted for in standard metrics.
Questions:

 * Can we define an equivalent (but meaningful) metric at the job level ?
  - (How do we monitor this at the job level)
 - Does this impact the accounting?
* How does CPU overcommit model interact with high 'IO wait' operational running (different answers between SSD and older tranches) ?

 

* DirectIO:
- Question raised again on directio at RAL;
  * If agreed, can follow-up on "activity" or gShare based job / resource targeting ?


* FTS :
  Requested to test the RAL FTS instance. Hoped I could send some 'innocuous' jobs through, but didn't work. Will try again ... 

 

13:46 → 13:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

13:48 → 13:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

LHCb

1. ECHO streaming issue (GGUS:142350) : Dedicated WNs running around only 15% of the submitted jobs even though they are empty. Rest of the jobs running elsewhere on batch farm. Waiting for fix from the team in charge before being able to progress on this.
   • http://lhcb-project-dirac.web.cern.ch/lhcb-project-dirac/RALEchoTestMonitoring.html
   • https://wiki.e-science.cclrc.ac.uk/web1/bin/view/EScienceInternal/LHCbJobFailure
   • Current status : Fairly certain that the issue does not occur as long as the file is locally cached on the WN
2. GGUS ticket : 146472 (Intermittent download of json file) : Resolved by LHCb moving over to using information from SRR
3. Not able to access lhcbdirac-cert.s3.cern.ch from RAL WNs (currently used for running tests involving pilot configuration). Very likely an issue at CERN end, keeping an eye on it.
4. RT Ticket (266522) : Request to install runit on LHCb VO-box at RAL - done and to be verified/reinstalled 
5. Upcoming activity : Move of LHCb VO-box at RAL to CC7

DUNE

1. Very quiet (291 jobs at RAL in the last week out of 1437 on the OSG)
2. ETF monitoring : https://etf-dune-preprod.cern.ch/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fview_name%3Dallhosts

13:52 → 13:53 VO Liaison Others

13:53 → 13:54 Experiment Planning

13:54 → 13:55 Dune/protoDune

13:55 → 13:56 Euclid

13:56 → 13:57 SKA

13:57 → 13:58 AOB

13:58 → 13:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))