The main topic for today is scrubbing preparation!
- We have partial drafts ready for most WBS 2.3 L3 areas, but most have significant gaps that need covering
- We have an updated and correct effort spreadsheet (verified by Verena) that we need to match for FY26 and potential updates/changes for FY27
- If anyone needs to change FTE levels for FY27, please mark changes in bold and add a comment for the change.
- Most L3 still need to address risks and FY27 plans.
- Copy of the risk registry spreadsheet is in the 2026 scrubbing folder
- Milestone updates and additions should be modified on our WBS 2.3 copy: https://docs.google.com/spreadsheets/d/11FMFmiCHdkkyWvMmNIHiyMc0bBUQI6sS4AzE0FR5TM8/edit?usp=sharing
- If you modify a field, change it to BOLD
- If you add a new milestone, all content in the added row should be bold
- There are AI summaries from last week's face to face in the 2026 scrubbing folder that may be helpful
- ACTION ITEM: L3's should schedule a review slot with Shawn and Alexei ASAP so we can discuss content.
Note: We need to provide Paolo and Verena Tier-2 availability for the last 3 months.
Upcoming meetings:
- ATLAS S&C at CERN June 28-July 3
- USATLAS Scrubbing (WBS 2.3) at Stony Brook on Tuesday July 14
ZOOM Notes for Meeting
Quick recap
The meeting focused on scrubbing preparation for WBS 2.3, with discussions about missing forward-looking plans and priority items in various L3 areas. Shawn emphasized the need for consistency across different WBS versions (2.3, 2.4, 2.5) to avoid discrepancies in presentations, particularly regarding FY26 and FY27 plans. The team discussed updates to effort spreadsheets, milestone tracking, and the transition from WBS 2.3 to 2.5 for FY27 activities. Frederick reported on Tier 2 operations, including recent downtime at Midwest and Net2 sites, while Rui provided updates on edge pieces and parameter testing. The conversation ended with a discussion about potential changes to data transfer policies at NERSC, where Douglas raised concerns about current workarounds that may not be permitted long-term, leading to a decision to develop a backup plan for future HPC operations.
Next steps
Brian
- Register Alexei as a US Atlas VO contact as per instructions provided.
Douglas
- Develop a backup plan for data transfer at NERSC and other HPCs in case direct data movement is no longer allowed, and evaluate the required work and potential impact.
Frederick
- Transfer all milestone changes made in the central copy of the milestone spreadsheet to the correct (facility) copy, and remove bold formatting from the original as appropriate.
- Provide Tier 2 availability and reliability numbers (for March, April, May) for each Tier 2 to Shawn, including both aggregate and individual site data.
- Ping John Hobbs for a definitive answer on Midwest Tier 2 funding status in time for the next management meeting.
Gamboa
- Check with DCACHE developers/organizers about the possibility of opening the DCACHE developers meeting to Tier 2 DCACHE admins and report back to Ofer.
Qiulan
- After 30 days from notification, archive then delete inactive NFT user data according to policy, and await further guidance from Viviana regarding the 24 users who have not responded.
Rui
- Add machine names to the CPU time information in the scrubbing slides for ALCF.
Collaboration
- L3 leads: Schedule a meeting with Alexei and Shawn to review current slides, discuss what's missing/needs changes, and ensure content is ready for final review with Paolo and Verena.
- All L3s: Update the FY27 effort spreadsheet with any needed changes, marking updates in bold with comments for traceability.
- All L3s: Address risks and FY27 plans in their slide decks and update milestone information in the RWBS 2.3 copy, marking new/changed fields in bold.
- Kaushik and Zach: Meet with L3s (especially 2.3, 2.2, 2.5, and 2.4) to discuss and clarify the transition of milestones and research/operations thrusts between WBS areas, ensuring consistency across slides.
- Aidan (and Frederick): Begin testing CVMFS 2.14.0 pre-release on CIT nodes and confirm to Brian when testing starts.
- Kaushik, Raphael: Review the Tier 2 FTE sheet in the scrubbing slides and confirm accuracy, especially for Midwest Tier 2 and any other areas of concern.
- All PIs (Tier 2 leads): Check and validate that their effort numbers in presentations match the official effort spreadsheet, and update as necessary.
Summary
Scrubbing Meeting Preparation Updates
The team discussed preparation for an upcoming scrubbing meeting, with most L3 areas having partial or complete drafts but several significant gaps still needing coverage. Shawn noted that the effort spreadsheet for FY26 has been verified by Verena, and requested that any needed changes for FY27 be marked in bold with comments explaining the changes. The team also needs to address risks and FY27 plans, with Joe Boudreau's risk registry spreadsheet provided as reference material, and milestone updates should be modified in RWBS 2.3.
WBS Version Consistency Planning
The team discussed ensuring consistency across different WBS versions (2.3, 2.5, 2.2) to avoid discrepancies in presentations and documentation. Kaushik suggested that items for FY27 should initially be placed in WBS 2.3 and later moved to 2.5, with team members notified via email about items that should move to 2.5. Douglas emphasized the need to maintain consistent milestones and research thrusts across all versions to ensure coherence, particularly regarding HPC-related items and facility requirements. Kaushik acknowledged that while the transition would be challenging, they would need to have discussions about all versions including 2.2 and 2.5, as the level 3 structure for 2.5 is not yet established.
Milestone Template and Data Updates
Shawn and Frederick discussed issues with the milestone template copy, where Frederick had made changes to the wrong version. Frederick agreed to transfer his changes to the correct copy and remove bold formatting. They also discussed the need to gather availability numbers for Tier 2 systems over the last three months for future funding agency reports, with Frederick offering to provide both aggregate and individual availability and reliability data.
Project Updates and Software Releases
Shawn noted upcoming meetings including Atlas Software and Computing in late June and WBS 2.3 scrubbing on July 14th. Frederick clarified that the three-month period would cover March, April, and May due to delayed June data. Brian announced the release of IGTF 1.143 to address issues with InCommons GTF certificates and mentioned ongoing work with the ExtraD team on patches. Frederick and Aidan agreed to install CVMFS 2.14.0 on CIT nodes after Brian's reminder.
US Atlas Development Updates Meeting
Frederick confirmed he would notify the team when starting a test and updated them on US Atlas VO contacts, noting that Alexei needs to register as a contact. Brian reported that API key development for topology is still targeted for the end of the month despite Matias being on vacation next week. Gamboa discussed an upcoming stakeholders and users technical meeting about storage technologies and mentioned a new minor release of recache (11.2.05) that improves NFS components, with plans to upgrade the integration instance soon. Ofer inquired about opening the developers meeting to Tier 2 DCACHE admins, and Gamboa agreed to check with the meeting organizers and provide feedback next week.
Tier Operations Update Meeting
Frederick provided updates on Tier 1 and Tier 2 operations, noting that Midwest Tier 2 experienced a production decrease due to server relocation and power consumption issues, while Net2 took downtime for maintenance. Frederick is working on Tier 2 scrubbing slides and requested Kaushik and Raphael to review the FTE sheet for accuracy. He also mentioned plans to follow up with John Hobbs about funding for tomorrow's management meeting.
Effort Spreadsheet and Data Updates
The team discussed effort spreadsheet validation, with Frederick agreeing to review FY26 and FY27 numbers and asking PIs to verify their numbers match the official spreadsheet. Rui reported on edge pieces parameter testing and data transfer issues, while Douglas explained NERSC's policy on Globus data transfers and potential future changes that could affect their current workaround methods. The team also reviewed updates from project 234, including storage configuration changes and management node implementation, and Ivan provided comprehensive updates on various project activities including LAC Run 3 completion and data recovery efforts.