Operations team & Sites

Tuesday 22 Sept 2015, 11:00 → 12:30 Europe/London

EVO - GridPP Operations team meeting

- This is the weekly GridPP ops & sites meeting - The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6 -- The PIN is 1234. To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers. -- The London (UK) service is on +442030510622 -- The meeting extension is 9308582. Apologies: Elena. Alessandra. Pete C.

Operations Minutes 22/9/2015
============================

Present:
Andrew Lahiff
Andrew McNab
Brain Davies
Chris Brew
Daniel Traynor
Daniela Bauer
David Colling
David Crooks
Duncan Rand
Federico Melaccio
Gang Qin
Gareth Roy
Gareth Smith
Gordon Stewart
Govind Songara
Jeremy Coles
John Bland
Kashif Mohammad
Oliver Smith
Paige Winslowe Lacesso
Raja Nandakumar
Raul Lopes
Robert Frank
Robert Fay
Samuel Skipsey
Steve Jones
Terry Froy
Tom Whyntie

1) Experiments
==============

LHCb  - Some data transfer problems but this should not have effected the UK
      - Enabling Machine job features by the end of the year, JC asks if there is a hard cut off on the date. 
    - Andrew McNab stated LHCb was looking to use it for a number of purposes as well as Atlas.
    - Raul, ran an initial test. Waiting for CMS to test the features.
    - JC asks for additional volunteers to try out the machine job features.
    - SJ asks for an outline of the purpose of Machine Job features.
      - Andrew McNab comments that for PBS it is a simple addition to the prologue script
    - JC states progress is slow due to lack of adopters.

      - Jobs not running at Lancaster
    - Raja asks for Lancaster contact details.

CMS   - Very little to report
      - Problem in Bristol with Phedex transfers being slow (but not failing)
      - All seem to be going to the same RAL FTS server (fts01).
      - A. Lahiff states they are in process of upgrading the RAL FTS to fix a memory leak.

Atlas - FTS transfers have been effected by memory leaks.
      - Problems getting files because they were pointing to the wrong VOMS server (voms vs voms2)

Other VO
========

DiRAC - Transferring lots of small files. Lydia is on holiday so it has been left to run.
      - Large number of files may have been contributing to the memory leaks see at RAL & CERN

LIGO  - Written update:
“Jeremy's summary: initial success getting statically compiled applications onto the CVMFS and running on the RAL cluster. Started to worry about supporting the dependencies and software for ALL of the pipelines used by LIGO. After some messing about settled on building the dependencies up one at a time.  Just installed one of the Python workflows and have started to run that at RAL. 

It terms of running, mostly supporting people who are running data analysis pipelines: this includes examining the data for gravitational wave signals, but also injecting artificial signals to compare any discoveries to. 

Question of how best to manage and update CVMFS. Noticed that /cvmfs/atlas.cern.ch/ is using PacMan to organise its software. Are there any recommendations on organising software and dependency installs. LIGO already has RPM and Debian packaging repositories and was hoping to make use of these if possible.”

      - Catalin states GridPP cvmfs setup at RAL is simpler than at CERN and doesn’t use PacMan.
      - Discussion ensues about software deployment. 
    - SS & SJ state that this is likely an issue for the VO as it will require understanding of the code base itself. 

LOFAR - Written update:
“Using resources to process LOFAR all sky survey imaging data to contribute to the creation of a new deeper low frequency all sky image.”

LSST  - SJ points out that LSST uses FNAL which is currently down/unavailable/

LZ    - No Update

UKQCD - No Update

UCLAN - Work has paused due to working on Thesis
    - TW working on writing up case study.

PRaVDA - No Update
 
GridPP DIRAC - Monitoring looks good for the VM side of things
         - Two sites need chased on the LCG side.

Meetings and Updates
====================
See Bulletin: 21st September 2015

EGI Management OPS Board:
- JC ask is there is anything the community want raised in the context of EGI Ops Board then contact him.

FTS3 Memory Leaks at RAL:
- AL has finished applying the updates to FTS3 at RAL which should hopefully improve the current issues.

Anomalous CPU usage for ILC jobs:
- Environment variable wasn’t set to restrict the number of threads being used.

Possible support of DEAP3600:
- Users at RHUL, RAL and Sussex

WLCG Ops Co-Ordination
=======================
See Bulletin: 21st September 2015
- Issues with rogue config tool at Glasgow & VOMS settings discussed.

Of interest to sites:
https://wlcg-ops.web.cern.ch/sys-admins/tasks

Middleware Readiness
====================
See Bulletin: 21st September 2015

Tier-1
======
See Bulletin: 21st September 2015

Storage & Data Management
=========================
See Bulletin: 21st September 2015

Tier-2 Evolution
================
- Task force around ideas to simplify Tier-2 sites.
- Recommendations produced will be optional but hopefully will be attractive.
- GridPP JIRA set up hosted at CERN
- GridPP Technical Meeting will now fit in with the Tier-2 evolution work.
   - DC states we have a relatively short timeframe, any sites that are willing to try things out are appreciated.

Accounting
==========
See Bulletin: 21st September 2015
- No update

Documentation
=============
See Bulletin: 21st September 2015
- No update

Interoperation
==============
See Bulletin: 21st September 2015
- Meetings appear to be in hiatus

Monitoring
==========
See Bulletin: 21st September 2015
- Half day Grafana conference in New York in October

On-Duty
=======
See Bulletin: 21st September 2015
- DB says Dashboard is not showing alarms. 
- GridPP nagios might not be available.
- KM Nagios is fine but alarms not being routed to Dashboard.

Rollout
=======
See Bulletin: 21st September 2015
- No update

Security
========
See Bulletin: 21st September 2015
- No update

Services
========
See Bulletin: 21st September 2015
- RAL downtime may lead to possible service disruption

Tickets
=======
- No update this week.

Tools
=====
See Bulletin: 21st September 2015
- No update

Site Discussion:
================
- RALPP (Chris Brew):     SL5 to SL6 migration
            ELK for monitoring

- QMUL (Dan Traynor):    20Gbps WAN
            Optimising Storage

- IC (Daniela Bauer):    DIRAC deploying LSST

- GLA (Gareth Roy):    ViaB
            Openstack
            OpenSOC
            Centos 7 DPM testing
            Optimising memory limits

- RHUL (Govind Songara): Firewall configs (utilise 10Gb link)

- Liverpool (John Bland): Electrical work
              Moving to new network range
       (Steve Jones): Testing GridPP docs

- Oxford (Kashif Mohammad): IPv6
                Cleaning puppet modules

- Manchester (Andrew McNab): Increase ViaB deployment
         (Robert Frank): Replacing existing provisioning system with foreman

- Tier-1 (Brian): Preparing End to End talk for the E2E meeting

Chat:
=====
Jeremy Coles: (22/09/2015 10:59)
Yes.
Tom Whyntie: (11:01 AM)
This week
Andrew McNab: (11:07 AM)
https://twiki.cern.ch/twiki/bin/view/LCG/WMTEGEnvironmentVariables
https://twiki.cern.ch/twiki/bin/view/LCG/MachineJobFeatures
Federico Melaccio: (11:09 AM)
Could this help? https://twiki.cern.ch/twiki/bin/view/LCG/MachineJobFeatures
Andrew McNab: (11:09 AM)
that second link is the WLCG task force
Daniela Bauer: (11:13 AM)
https://cmsweb.cern.ch/phedex/prod/Activity::Routing?tofilter=T2_UK_SGrid_Bristol&fromfilter=.*&priority=any&blockfilter=.*
Jeremy Coles: (11:16 AM)
Raja here you are: lcg-admin@hep.lancs.ac.uk
If Matt is away then Robin should pick up on the issue.
Raja Nandakumar: (11:20 AM)
Thanks Jeremy.
Terry Froy: (11:24 AM)
I wish Vidyo had a volume control per participant - that way, I could hear those with quiet microphones without getting my head blown off by those with somewhat louder microphones :-P
Jeremy Coles: (11:28 AM)
Yes I know Terry.
Steve Jones: (11:29 AM)
ReL LSSPT- The FNAL voms server is back - it must have been a transient. 
Re: LSST!!!
Daniela Bauer: (11:30 AM)
My audio has just given up, will restart Vidyo
Federico Melaccio: (11:35 AM)
https://ggus.eu/index.php?mode=ticket_info&ticket_id=116238 
It is spelled DEAP apparently http://deap3600.ca/
Jeremy Coles: (11:43 AM)
oops. Thanks. I had an email from them with DEEP!
Federico Melaccio: (11:50 AM)
how can I join the t2evo mailing list?
I am on the ops
and I have a lightweight account
yes I can see it
thanks!
thank you Andrew
Andrew McNab: (11:54 AM)
no problem
David Crooks: (11:57 AM)
http://grafanacon.org
David Colling: (12:04 PM)
apologies, I need to leave now.
TTFN
Jeremy Coles: (12:04 PM)
Thanks.
Terry Froy: (12:05 PM)
Nothing to add.
Federico Melaccio: (12:05 PM)
the unmuted was me, sorry
Daniela Bauer: (12:07 PM)
Dashboard ticket
https://ggus.eu/index.php?mode=ticket_info&ticket_id=116356
I guess I should have mentioned Simon's and Duncan's adventures with NDN.
(as a site adventure)
Jeremy Coles: (12:10 PM)
Let's ask Duncan!

11:00 → 11:01 Ops meeting minutes

* This is a reminder that this is an important task. The minute taker gives access to the discussions for those not present and provides a reference for others to refer back to afterwards. * The team composition has been changing. If everybody contributes then the task comes around less often. * From the start of GridPP4+ those in fully funded GridPP positions will be expected to contribute. Others are welcome to volunteer! * The minutes should contain a list of who attended; apologies; note who took the minutes and highlight actions. * A count is maintained at https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items. * After uploading minutes to the agenda page the minute taker is expected to: ** Update the list of ops actions. ** Update their 'count' so the task can be shared fairly. Thank you for your support!

11:01 → 11:20 Experiment problems/issues

Review of weekly issues by experiment/VO - LHCb - CMS https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T2_UK_London_Brunel See comment last week: https://twiki.cern.ch/twiki/bin/view/CMSPublic/SpaceMonSiteAdmin - ATLAS - Other -- DiRAC: Jens -- LIGO: Catalin Jeremy's summary: initial success getting statically compiled applications onto the CVMFS and running on the RAL cluster. Started to worry about supporting the dependencies and software for ALL of the pipelines used by LIGO. After some messing about settled on building the dependencies up one at a time. Just installed one of the Python workflows and have started to run that at RAL. It terms of running, mostly supporting people who are running data analysis pipelines: this includes examining the data for gravitational wave signals, but also injecting artificial signals to compare any discoveries to. Question of how best to manage and update CVMFS. Noticed that /cvmfs/atlas.cern.ch/ is using PacMan to organise its software. Are there any recommendations on organising software and dependency installs. LIGO already has RPM and Debian packaging repositories and was hoping to make use of these if possible. -- LOFAR: George Using resources to process LOFAR all sky survey imaging data to contribute to the creation of a new deeper low frequency all sky image. -- LSST: Alessandra -- LZ: David -- UKQCD: Jeremy -- UCLan/GalDyn: Tom -- PRaVDA: Mark/Matt - DIRAC status -- http://www.gridpp.ac.uk/php/gridpp-dirac-sam.php?action=view

11:20 → 11:40 Meetings & updates

With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest - General updates - WLCG ops coordination - Tier-1 status - Storage and data management - Accounting - Documentation - Interoperation - Monitoring - On-duty - Rollout - Security - Services - Tickets - Tools - VOs - Site updates

11:40 → 12:00 Discussion

1. Current site priorities/activities.

12:00 → 12:05 Actions & AOB

* https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items