DOMA / QoS Meeting

Monday 5 Nov 2018, 10:00 → 11:00 Europe/Zurich

Announcements

Oliver will cochair the group.

Some upcoming presentations:

• This wed - WLCG cost modelling WG.  Will describe what we're doing and understand what the connection between the two groups will be.
• DOMA access - possibly next Tuesday.

We can use these to build up a standard QoS slide deck.

VO talk

A survey of VO workflows that could benefit from QoS.

We can use ATLAS' Technical Collaboration Board (TCB), which meets Monday mornings to kick this off.  Try to get a slot in 2018-11-19 or 2018-11-26.

Aim at mid December for some preliminary results.  This would be in time for an ATLAS workshop (which?).  Vincent already presented some QoS activity during the June ATLAS workshop at DESY.

Paul shared a slide skeleton.

Discussion on difference between static/dynamic QoS; i.e., between an experiment exploiting a diverse infrastructure and a single storage system offering different QoS.

Side 3

Transitions are managed by Rucio, not by individual storage systems.

Paul -- model now is DISK.  Sites that don't do anything automatically inherit this ... but DISK can cover a lot of different things: 2 replicas, RAID, enterprise -> consumer + JBOD + 1 replica.

Can we add "handling data loss" to experiment survey?

Would reduced durability requirements save us any money?

Slide 4

TAPE has two independent qualities: CUSTODIAL -> high durability COLD -> requires time to "warm up", but cheaper.

Can we separate these concepts?

Sites could offer COLD storage that isn't CUSTODIAL; for example, public cloud offering?

Possible output of group -- what QoS possibilities are out there?  We hope to get this from the site survey.

Slide 5

How do VOs use more expensive storage?

SSDs separate endpoint (i.e., requires copying data) or software defined (i.e., requires API call).

What benchmarks are available now for all these QoS alternatives?

There was a discussion on whether SSDs could be deployed as a cache.

Mario - if we know the content of a storage system (i.e., what is on the SSD) then it's not a cache.

Site survey

Paul shared an initial skeleton version.

Get an idea of what's out there at sites.

Use this to try to harvest all the QoS thinking that sites have done up to now.

We'll discuss in the group what we want to ask.

Try with a couple of volunteer sites: CERN & DESY

The idea is to discover QoS landscape.

Do we want to describe a reference system and ask if sites deviate from that?

Better to ask what sites have.  We can send the survey with ~3 sites already filled in as example answers: CERN, DESY and a T2 DPM site?

We'll iterate the questions in the next couple of weeks.

Martin Gasthuber -- T2s want to understand what our QoS options are; e.g., low endurance, can only be written a small number of time (~200).  How do we understand if this is interesting or not?

How do we reward sites (at WLCG level) for investigating different QoS options?  There is a risk involved for them.