US ATLAS Computing Integration and Operations
-
-
13:00
→
13:15
Top of the Meeting 15mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))
-
13:25
→
13:35
Capacity News: Procurements & Retirements 10m
-
13:35
→
13:45
Production 10mSpeaker: Mark Sosebee (University of Texas at Arlington (US))
-
13:45
→
13:50
Data Management 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
-
13:50
→
13:55
Data transfers 5mSpeaker: Hironori Ito (Brookhaven National Laboratory (US))
-
13:55
→
14:00
Networks 5mSpeaker: Dr Shawn McKee (University of Michigan ATLAS Group)
-
14:00
→
14:05
FAX and Xrootd Caching 5mSpeakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:25
→
15:25
Site Reports
-
14:25
BNL 5mSpeaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
-
14:30
AGLT2 5mSpeakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
All operations proceeding normally at AGLT2. Should bring up a second storage node and begin dCache balancing at the MSU site by the end of this week (node is active and formatted, awaiting the arrival of an SFP+ cable or 3).
Investigating slow dCache transfers to many WN. Showed as a problem with MCORE jobs when they copy in of order 100GB of files. No answers yet, just tantalizing hints concerning network connectivity and IOPS saturation.
space-usage.json file now stored in scratchdisk as per Wei's request, and the ruby file used at AGLT2 is on the Wiki page.
-----------------------------------------------------------------------------------------------------
In regard to the Linux privilege escalation bug that is under active exploit:
See OSG-SEC-2016-10-21, CVE-2016-5195 and other items below.
REFERENCES:
[1] https://security-tracker.debian.org/tracker/CVE-2016-5195
[2] http://people.canonical.com/~ubuntu-security/cve/2016/CVE-2016-5195.htmlRELATED LINKS:
- - https://access.redhat.com/security/vulnerabilities/2706661
- - https://wiki.egi.eu/wiki/SVG:Advisory-SVG-CVE-2016-5195
- - https://ticket.opensciencegrid.org/31592the vulnerability is related to copy-on-write code that allows writing to an area where user should not write. A nice description of the details are summarized here:
http://www.theregister.co.uk/2016/10/21/linux_privilege_escalation_hole/(There is a nice explanation starting with paragraph "The exploit works by racing …”)
It has been in the Linux kernel for at least 12 years. Previous attempts were made to fix it, but it was difficult to fix, and originally not so easily exposed. This latter has changed, and the fix is now critical as bugs to exploit this are now in the wild, affecting all kernels listed here.
VERSIONS IMPACTED:
- - CentOS Linux 5, 6 and 7
- - RedHat Enterprise Linux 5, 6 and 7
- - Debian Linux
- - Ubuntu Linux
- - SUSE Linux Enterprise 11 and 12Machines with user access are the ones most at risk as code can easily be compiled that takes advantage of the vulnerability, but any machine where a file can be dropped is open to the exploit.
================ WHAT CAN WE DO? ===================
We have 2 choices, apply a mitigation developed by CERN, or wait for the kernel rpm releases that are appropriate, and which now seem imminent.
Mitigation:
The mitigation does not switch off copy-on-write, it forbids ptrace() syscall (used mainly by strace, ltrace and gdb). I was not clear about this in my previous email.
"CERN IT has now deployed the mitigation on lxplus and lxbatch machines so we should see quite soon if it breaks any ATLAS workflow"
It did not break the workflow. athenaMP continues to work, which was a prime concern.Kernel rpm update:
there is now a patch for RHEL7 systems:
https://rhn.redhat.com/errata/RHSA-2016-2098.html
kernel-3.10.0-327.36.3.el7 is the one we need, and it's already
available on CentOS at least.and now there is a new kernel for el6 and derivatives too, see http://rhn.redhat.com/errata/RHSA-2016-2105.html
As of this morning there was not yet a Scientific Linux kernel in our local repository copies. AGLT2 will automatically apply this most everywhere once it is available, and once we see that has happened, we will begin to reboot all our systems to the new kernel, with the exception of Worker Nodes where we will take a more leisurely approach.
-
14:35
MWT2 5mSpeakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))
Site is running well but Illinois is currently offline
Kernel Copy on Write bug
- All UChicago and Indiana worker nodes are patched and rebooted
- Illinois worker nodes are offline and waiting on ICC for new image
- All Perfsonar nodes are now patched
OSG 3.3.17
- Update currently in progress
- New HTcondor-CE
- New CVMFS 2.3.2
FY16 Purchases
- UChicago
- Network upgrades complete (firmware, power, linecards, TOR 10Gb switches)
- 48 R630 Compute nodes online
- 2 of 7 storage servers are now online
- One is full of data and in used
- Migrating data onto the second today
- Retired very old storage at UC and IU
- Currently have 5.3PB available in dCache (will now meet pledge)
- Other 5 servers will be brought online over next couple weeks
- Indiana
- 24 R630 nodes are now online
- Illinois
- 3 of 6 c6320 chassis installed (12 out of 24 nodes)
- Waiting on Dell to ship parts for the damaged nodes
- All nodes now using 10Gb connections
- Relinquished the IB connections (GPFS is over 10Gb)
- Initially lack of sufficient power in new racks caused some issues
When all equipment is installed and online, MWT2 will provide about
- 169K HS06
- 16368 logical cores
- 9PB dCache disk
-
14:40
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
We had a very enjoyable NET2 site visit from Paolo, Eric, Kaushik and Rob in the past two days, including:
1) Visit to MGHPCC http://www.mghpcc.org
2) Meeting with the Provost and VPR.
3) NET2 meeting with BU and Harvard research computing, Mass Open Cloud.Our admins are watching for the kernel fix for CVE-2016-5195. We will act quickly as soon as there is a fix.
Major LHCONE progress: Attempt #2 to switch over to the new NoX ASR succeeded ( http://egg.bu.edu/NET2%7binf:NET2%7d/gadget:Studies/section:report/2016-10/Successful_switch_to_NoX_ASR/ ...showing that we have our full 20G/s after the switch over)! With this switch, Chuck is now in touch with Yvonne & co. at ESNet to do the last steps in joining LHCONE.We have smooth operations in the past two weeks with both sites mostly full, as Mark says. The new 1.5 PB of storage is still being testing, but will be in GPFS and in space tokens soon.
We're gearing up with Harvard to start the NESE project as well.
-
14:45
SWT2-OU 5mSpeaker: Dr Horst Severini (University of Oklahoma (US))
- all sites running well
- remaining OSCER issues with long running or hung jobs seem to be fixed as well
- still occasionally see lost heartbeat jobs at Lucille, and so far nobody has been able to track those down; jobs lose connection with panda server, but only after starting to run normally (meaning, after a lot of interaction with panda server already), so no obvious network problem on our end ...
- still waiting for new OSCER storage rack to be fully cabled, then OS installed, then can install and configure xrootd
-
14:50
SWT2-UTA 5mSpeaker: Patrick Mcguigan (University of Texas at Arlington (US))
-
14:55
WT2 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:25
-
15:25
→
15:30
AOB 5m
-
13:00
→
13:15