Go to the Indico Home Page Oldest event Older event Up to category Newer event Newest event Printable version
  • Compact style
  • Indico style
  • Indico style - inline minutes
  • Indico style - numbered
  • Indico style - numbered + minutes
  • Indico Weeks View
    • Switch to the management area of this event
    Login

    ATLAS UK Cloud Support

    → Europe/London
    Vidyo

    Vidyo
    Tim Adye (Science and Technology Facilities Council STFC (GB)), James William Walder (Science and Technology Facilities Council STFC (GB))
    Hide

     Open GGUS:

    •  146619 UKI-SOUTHGRID-RALPP dCache WebDAV macaroon token configuration

    (Taken from email):
    I’ve got a (rather too old) EGI Security ticket because the atlasscratchdisk area is writabe by any user of any supported VO.
    This was because when it was originally created write access was needed by both the Atlas Production and normal roles which map to different groups and either ACLs were not available in dCache or we didn’t want to use them for some reason.
    I’ve now enabled ACLs in our dCache and would like to schedule a time to change the permissions on this area. Which if things go wrong may disrupt you.

     

    •  146615 UKI-NORTHGRID-LANCS-HEP deletions

    * Details: DavPosix::unlink Authentication error, reached maximum number of attempts
    Problematic server is back up

     

    • 146596 UKI-SCOTGRID-GLASGOW  ATLAS delegated proxy with short lifetime

    In progress and updates expected soon. Suggestion to others upgrading CE, this can be a change to the default configuration.

     

    • 146588 Failovers from RAL-LCG2 to CERN CVMFS backup proxy 

    In progress, one of several Squid related activities

    •  146571 Failovers from UKI-LT2-IC-HEP to Fermilab CVMFS stratum1

    Ipv4/v6; old squid version; in progress

    •  146525 UKI-NORTHGRID-SHEF-HEP: evicted jobs

    Depends on 146374

     

    • 146523 UKI-NORTHGRID-MAN-HEP: timeouts on stage-out, file existence verification failed

    More timeouts observed; in progress.
    Timeouts are very difficult to diagnose, as Rucio mover gives no debugging information.

    • 146374 ATLAS pilot jobs idle on UKI-NORTHGRID-SHEF-HEP CE

    Rebuilding of the Arc-CE is required; sill take some time as WFH.

     

    • 146159 UKI-SCOTGRID-GLASGOW_DATADISK Unaccessiböe files

    Generally happy with the plan from Rod.
    Draining disk progressing

     

    • ## 145688 Very old version of squids at UKI-NORTHGRID-MAN-HEP

    Currently in hold, but will start to plan for upgrade soon.

     

    • 145510 RAL-LCG2: timeouts on stage-in/outs

    On hold - need more info in Rucio mover logs

     

    • 144759 High traffic from UKI-SCOTGRID-GLASGOW on RAL CVMFS Stratum1

    Can be similar to other Squid issues; however some routing configurations might still be needed.

    • 142329 CentOS7 migration UKI-SOUTHGRID-SUSX

    2GB single core jobs are excepted, but possible hard limits on memory at 2GB could cause the issues.
    Also possible that JVM for the voms-proxy-init can take significant memory, for short period.
    Ticket updated to suggest to give higher free memory requirements.

    Try to use Lancaster policies as starting point.


     CPU

    Major problem affecting upto 50% of all ATLAS slots on 22/4; traced down to problem with Rucio-clients version on CVMFS.
    New 'patch' versions of 1.21.11 patch1 rolled out; sites should be recovering.

    * Some Zombie jobs might exist; sites can kill; ATLAS looking into a central command to remove these jobs.

    RAL
     Smooth running (no observed impact from overall problems). All new WNs in production.

    NorthGrid

        Man and Lancs / zombie jobs

    UK-London-Tier2

    SouthGrid

    ScotGrid
    Glasgow may also have large number of zombie jobs; will be investigated

    # Other new issues

    ## Squid at Durham
    also serves CVMFS; Glasgow happy to for Durham to switch away from their squid; hope this will reduce the load and number of times that fallover occurs

     

    Ongoing issues

    •  Sussex SL7

    * discussed above

    • Glasgow Ceph storage
      •  - Nautilus problems are GLASGOW specific to site; RAL should be unaffected when it upgrades. Some other useful information already passed to RAL

         
    • Grand Unified queues

    Waiting for SHEF to come back online, then can conclude


    Alessandra: NTR
    Peter: NTR
    Matt:NTR
    Sam: NTR
    James: NTR
    Gareth: Microphone ok; suggestion to actively invite more people to the meeting
    VIP (left early)

    There are minutes attached to this event. Show them.
      • 10:00 → 10:20
        Status 20m
        • Outstanding tickets 10m
          Open ATLAS UK GGUS tickets

          ## 146619 UKI-SOUTHGRID-RALPP dCache WebDAV macaroon token configuration
          (Taken from email):
          I’ve got a (rather too old) EGI Security ticket because the atlasscratchdisk area is writabe by any user of any supported VO.
          This was because when it was originally created write access was needed by both the Atlas Production and normal roles which map to different groups and either ACLs were not available in dCache or we didn’t want to use them for some reason.
          I’ve now enabled ACLs in our dCache and would like to schedule a time to change the permissions on this area. Which if things go wrong may disrupt you.


          ## 146615 UKI-NORTHGRID-LANCS-HEP deletions
          * Details: DavPosix::unlink Authentication error, reached maximum number of attempts
          Problematic server is back up


          ## 146596 UKI-SCOTGRID-GLASGOW  ATLAS delegated proxy with short lifetime
          In progress and updates expected soon. Suggestion to others upgrading CE, this can be a change to the default configuration.


          ## 146588 Failovers from RAL-LCG2 to CERN CVMFS backup proxy 
          In progress, one of several Squid related activities

          ## 146571 Failovers from UKI-LT2-IC-HEP to Fermilab CVMFS stratum1
          Ipv4/v6; old squid version; in progress

          ## 146525 UKI-NORTHGRID-SHEF-HEP: evicted jobs
          Depends on 146374


          ## 146523 UKI-NORTHGRID-MAN-HEP: timeouts on stage-out, file existence verification failed
          More timeouts observed; in progress.
          Timeouts are very difficult to diagnose, as Rucio mover gives no debugging information.

          ## 146374 ATLAS pilot jobs idle on UKI-NORTHGRID-SHEF-HEP CE
          Rebuilding of the Arc-CE is required; sill take some time as WFH.


          ## 146159 UKI-SCOTGRID-GLASGOW_DATADISK Unaccessiböe files
          Generally happy with the plan from Rod.
          Draining disk progressing


          ## 145688 Very old version of squids at UKI-NORTHGRID-MAN-HEP
          Currently in hold, but will start to plan for upgrade soon.


          ## 145510 RAL-LCG2: timeouts on stage-in/outs
          On hold - need more info in Rucio mover logs


          ## 144759 High traffic from UKI-SCOTGRID-GLASGOW on RAL CVMFS Stratum1
          Can be similar to other Squid issues; however some routing configurations might still be needed.

          ## 142329 CentOS7 migration UKI-SOUTHGRID-SUSX
          2GB single core jobs are excepted, but possible hard limits on memory at 2GB could cause the issues.
          Also possible that JVM for the voms-proxy-init can take significant memory, for short period.
          Ticket updated to suggest to give higher free memory requirements.

          Try to use Lancaster policies as starting point.

        • CPU 5m
          UK Cloud jobs over last week

          Major problem affecting upto 50% of all ATLAS slots on 22/4; traced down to problem with Rucio-clients version on CVMFS.
          New 'patch' versions of 1.21.11 patch1 rolled out; sites should be recovering.

          * Some Zombie jobs might exist; sites can kill; ATLAS looking into a central command to remove these jobs.

          RAL
           Smooth running (no observed impact from overall problems). All new WNs in production.

          NorthGrid

          Man and Lancs / zombie jobs

          UK-London-Tier2

          SouthGrid

          ScotGrid
          Glasgow may also have large number of zombie jobs.

        • Other new issues 5m

          Frontier cache at Durham
          squid.dur.scotgrid.ac.uk:3128

          ## Squid at Durham
          With CVMFS; Glasgow ok to switch over; hope this will reduce the load and number of times that failover occurs

      • 10:20 → 10:40
        Ongoing issues 20m

        ## Sussex SL7
        * discussed above

        ## Glasgow Ceph storage
         - Nautilus problems are GLASGOW specific to site; RAL should be unaffected when it upgrades.

        ## Grand Unified queues
        Waiting for SHEF, then can conclude

      • 10:40 → 10:50
        News round-table 10m

        Alessandra: NTR
        Peter: NTR
        Matt:NTR
        Sam: NTR
        Gareth: Microphone much better now;
                      Suggestion to actively invite more people to the meeting

      • 10:50 → 11:00
        AOB 10m