US ATLAS Computing Integration and Operations

Wednesday 19 Aug 2015, 13:00 → 15:00 US/Eastern

Other Institutes

Notes and other material available in the US ATLAS Integration Program Twiki

13:00 → 13:15 Top of the Meeting

Speakers: Michael Ernst, Robert William Gardner Jr (University of Chicago (US))

In addition to the usual operations and site updates, we’ll continue to review action items from the the US ATLAS S&C Technical Planning Meeting at CERN:

• Action items: https://goo.gl/ttjx4c 
• Technical meeting: https://indico.cern.ch/event/385987/

 

 

13:15 → 13:25 Production

Speakers: Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US))

shift-summary-8_12_15.pdf

shift-summary-8_19_15.pdf

13:25 → 13:30 Data Management

Speaker: Armen Vartapetian (University of Texas at Arlington (US))

150819_DataManagement_Armen.pdf

13:30 → 13:35 Data transfers

Speaker: Hironori Ito (Brookhaven National Laboratory (US))

• Hiro checked that only one stream for FTS at BNL on incoming data.  Needs some investigation - is this value optimal?
• Ans: Saul's study said its 1 - but there are other parameters - Need an independent look.

 

13:35 → 13:40 Networks

Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)

http://maddash.aglt2.org/madalert.html

Have to go. See throughput notes. Bob will report on AGLT2 status

13:40 → 13:45 FAX

Speakers: Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

Ilija:

• BU issues - probably fixed with a firewall issue.
• FR site problem (DPM site) - appears rarely (for other sites).  This creates a problem - caused CERN_PROD to show up red.  Issue solved by taking the site offline.
• Two other FR sites going red for downstream redirection every 2 or 3 hours - probably a config problem, hopefully fixed. 
• In general - what to do with chronically failing sites - BU was causing a user problems.  Need a way to "blacklist" a site. Issue for Andy to address in xrootd.  

 

Wei

Hi Rob,

I will miss Wed’s facility meeting. Here is a few things about FAX and WT2

FAX: 

1. TRUIMF forward proxy (proxy for their batch nodes) is fully functioning.

2. Andy, Ilija and Augustine are working on solving the problem at BU.

 

Good news about TRIUMF!  As a follow-up - is the reverse proxy at BNL reached production stability?

Not yet. Andy believes that an rpm update (to new xrootd rpms) will solve the problem. I intentionally kept BNL at a slightly older version in order to help debugging TRIUMF’s problem. I will e-mail both sites to upgrade.

 

 

13:45 → 14:45 Site Reports

13:45 BNL

Speaker: Michael Ernst

13:50 AGLT2

Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

13:55 MWT2

Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

• Site running well an full of jobs
• Full Partitionable configuration
• Converted GlideinWMS on two CEs to HTCondor-CE. Remaining two will be today
  • Fermilab and Glow are MCORE ready
  • OSG is being looked into by BrianL
• Adding a new Condor Poll "ceph" to utilize cores on Ceph Servers
• ES still in progress (pilots bug etc)
• MWT2 CMFS Server being converted to 2.1.20
• Ceph SE:
  • Successfully replicated many test datasets to MWT2_UC_CEPH last week. Low error rate (<1%), seemed to have stemmed from source rather than our endpoint.
  • Saw load issues on BeStMan server during initial tests. Cause: adler32 checksumming is done by the SRM server, which necessitates reading all files out of storage once written — thus bottlenecking on a single 10Gb. Resolved problem by adapting INFN StoRM GridFTP DSI to work with OSG stack. Checksumming is now done in situ by the GridFTP servers themselves. Installed on all 14 GridFTP doors and is operating stably. RPM and sources available for interested sites.
  • Currently working on XRootD doors.
• ATLAS Connect:
  •  NX available, currently 1 shared server backing it. Documentation forthcoming.

14:00 NET2

Speaker: Prof. Saul Youssef (Boston University (US))

14:05 SWT2-OU

Speaker: Dr Horst Severini (University of Oklahoma (US))

14:10 SWT2-UTA

Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

14:15 WT2

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

 

WT2:

Ale DiGi helped me understand the ADC reporting on site HS06 contribution. It is clear that the site average HS06 factor used for WT2 is underestimate significantly. It is derived from the site total HS06 (from OSG OIM) and site total logical core count (via BDII / OSG GIP). We can’t say that ADC uses these numbers in a wrong way. But it was unexpected to me - the total HS06 and total core count for a site can refer to different (hardware) scope. This was the case for WT2 (so I have to change the way we report logical core count), and I don’t know if other site admins know that ADC calculates the site average HS06 in such a way (remember that we (the US facility) maintains an up-to-dated spreadsheet that has a relatively accurate site average HS06). Giving the complexity, I would like to suggest that ADC sends out monthly site contribution reports so that site admins have an opportunity to verify those numbers. Both OSG and WLCG do this.

 

The ADC reporting also showed two other issues at WT2: higher percentage wall time spent on failed jobs at WT2 (compare to other US sites), and 10% discrepancy of walltime reported by WT2’s LSF and by Panda (with Panda reports 10% less). Thanks to your observation that WT2 had very high failure rate early last week, I am confident that all these are now traced down to an HTCondor-CE bug that prevents the APF from sending the renewed X509 proxy to the running jobs. This wasn’t a serious issue a few months ago when we mostly run Mcore jobs (because Mcore jobs are short, they run 1-2 hours, combine with the AGIS timefloor parameter, those jobs seldom exceed the initial X509 proxy’s lifetime). But single core jobs usually run longer than a day, which exceed the initial X509 proxy’s lifetime. This cause very large number of jobs at SLACXRD to lost heartbeat with Panda. I have reported this bug to the Condor team. Haven’t heard back from them but since the fix is trivial, I put it in at WT2.

 

regards,

Wei Yang  |  yangw@slac.stanford.edu  |  650-926-3338(O)

14:45 → 14:50 AOB

​Michael

• ATLAS is using site-specific information in REBUS to determine the delivered CPU capacity to the collaboration.
• It's a responsibility of site administrators to provide the correct numbers for pledges and number of logical cores - these are the essential parameters used by ADC to calculate the HS/core number.
• Bob has noted this is a responsibility of the site admins to make sure the sub-cluster information is set correctly, which is propagated via the OSG BDII to REBUS  
• Site admins should take a look - and update when there are changes to the hardware configuration.
• Bob: notes it does take time to propagate to REBUS.  Eventually gets there. 
• n.b. Accounting information nodes from Dave Lesny: https://docs.google.com/document/d/1dMZrxGhleJ-GGLjyAoZtHfN9l0OZfakkSVM1Y4d09Rs/edit?usp=sharing
•