RAL Tier1 Experiments Liaison Meeting

Wednesday 10 Jun 2020, 13:30 → 14:30 Europe/London

Access Grid (RAL R89)

13:38 → 13:39 Major Incidents Changes

13:39 → 13:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20200610.docx

RT1EL-20200610.pdf

13:40 → 13:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:41 → 13:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:42 → 13:43 Experiment Operational Issues

13:44 → 13:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

20200608_DC_v1.pdf

Next round of Data Carousel  started today:
~ 18Tb of AOD -> DAOD_PHYS and PHYSLITE formats

 

13:46 → 13:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

AAA - After looking at RAL WN hits on AAA manager last week with Alastair, I asked Chris B about it again. He confirmed that only reads from Echo should be seen in those logs. This was confusing as we thought that we had redirected requests from RAL jobs, so that it didn't try to get Echo data via AAA when a direct read failed. I examined the /etc/hosts file in the job docker containers, and found that I had never remembered to update this with the new hardware - i.e. proxy 695 and 719 were still listed instead of ceph-gw10, 11. I had a look at how many requests were appearing from RAL nodes on different days, and it is in the order of several hundreds, a small fraction of the total, but maybe still worth doing. However Chris said to chat with Alastair and decide, as clearly there is some additional complexity with what we are/were doing.

 

Weekend job failures with Offsite reads - all T1 sites doing Offsite reads were failing 100% last weekend. I did some investigation myself, and it looks to me like the 'secondary input' files were not listed properly. I passed this on to computing operations, and i hope they will confirm that this was just a silly mistake.

 

Kibana - have spent some time this week trying to figure out why my plots and searches started looking strange, and believe i have made significant progress in understanding

 

 

13:48 → 13:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

LHCb

1. Reduction of running jobs yesterday 
   • Update of pilot code - bad PR
   • Fixed and slowly recovering 
2. ECHO streaming issue
   • Xrootd variables did not seem to have any effect 
   • Testing  out memory proxy
   • WNs not running properly since last Friday 
     • Jobs stuck in IDLE state 
     • A few jobs run at midday everyday 
     • Under investigation 

DUNE

1. Low activity - only tens of production jobs on the grid
   • Glide-ins apparently fairly agnostic as to site
   • Other possibilities being checked
   • Started a push to get user jobs also on to the grid
2. Various etf / Sam updates
   • Added test for stash cache
   • monit page available. Trying to understand it before circulat

 

13:52 → 13:53 VO Liaison Others

13:53 → 13:54 Experiment Planning

13:54 → 13:55 Dune/protoDune

13:55 → 13:56 Euclid

13:56 → 13:57 SKA

13:57 → 13:58 AOB

13:58 → 13:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))