WP4 - Coordination meeting

→ Europe/Zurich
Description

Video link

 

https://fau.zoom.us/j/63454144536 (permanent for this WP)

Documents

In this Google drive folder: https://drive.google.com/drive/folders/1qwDYjVgCnWjMrKUxAIxlx_aRRMCGPsfl?usp=share_link

Agenda

  • Update of the work plan
  • Update of the workplan
    • Task 4.1 
      • Workflows: Testing difficult, jobs failing, evaluation difficult --> explore contacts at Nikhef (Belle2), otherwise offer to organize discussion
      • Integration with Rucio was underestimated
      • On workflows progress ok, no big show-stoppers
        • Detailed feedback by Paco:
        • Testing, performance, evaluation :
           -> Corsika and Run-Based-Data-Processing tests still frozen after  (failed) tests of pseudo-mass-processings in February/March. Many of the  jobs failed, depending on the destination site, and the amount of (true) consumed resources was much larger than the amount that Dirac was aware of having consumed. These two problems need to be fixed before mass-processings can be resumed. In the meantime, for efficient testing, the tests also revealed the need both for a downscaled (i.e. faster) run-based-data-processing and for better grid storage management. In the case of Corsika, no testing procedure has been defined to check whether the output files are correct, and I do not possess the knowledge to define one myself; we are attempting to do so together with Antonio Condorelli and Brían Ó Fearraigh.
        • Design & Implementation of workflows :
            -> Delayed. The amount of effort required to integrate existing code with Rucio was underestimated, as well as the need to implement this before testing could proceed; this step has now been nominally achieved (including automatic replication from tape to disk before job submission), but testing for a mass processing is still required. Additional effort to produced scaled-down versions of Run-Based-Data-Processing (to go down from ~70h per job to ~1h for testing and CI/CD purposes) are underway; this has been achieved for MCs, but not for processing of real data.
          Once workflows pass the testing phase, defined as being able to manage a mass processing, the next step is to design a production system (such as with the Dirac Transformation System), to automate as much as possible submission of mass jobs or processing of new data. Discussions are ongoing with Andrei T. (EGI) and Cedric S. (Belle2) to test an integration between the Dirac Transformation System and our Rucio server.
        • Development workflow monitoring tools : 
          -> Basic tools and displays exist thanks to the Dirac dashboard. Some additional work is required to find out the source of the discrepancy between the resources consumed according to Dirac, and the "true" amount according to the destination computing sites. Additional work will be required after the lessons from a test mass-production, and/or after implementing a production system.
    • Task 4.2
      • Dataverse in progress to be set up by Tom, will contact developers for swarm deployment
      • Jutta will improve on system in summer
    • Task 4.3
      • offering alert data on dataverse might be in coordination with Vladimir
      • GCN broker sends new alerts, work on alert sending accelerating
      • alert sending pipeline during summer, aiming for 1/month alerts for ARCA
  • Draft MoU @GoogleDrive
    • Can go the same way as scientific MoUs
    • Jutta will extend proposal with first best guess on governance structure, pass to Kay to review, then to management and IB chairs via Victoria
    • process like WP2 and SLA, asking legal experts, then to RRB --> support by Victoria
  • Organization
    • payments for first round are on their way
    • no meeting in August
There are minutes attached to this event. Show them.
The agenda of this meeting is empty