Login

FTS3 Steering Meeting

→ Europe/Zurich
31/S-028 (CERN)

31/S-028

CERN

30
Show room on map
Alejandro Alvarez Ayllon (CERN)
MultiSourceDownloads.pdf
Hide

Roadmap

  • Multihop looks like not used by anybody. ATLAS has, a priori, no intention to use it. May be possible to just drop the functionality.

ATLAS requests/suggestions/comments

  • Can transfers be done with multiple sources used in parallel?
    • A student (Alin Andrei Grigorean), supervised by María Arsuaga, has been evaluating the performance of this possibility (I attach to this Indico event the slides)
    • Preliminary results look promising
    • Protocol support may be a problem
  • FTS3 could automatically choose the protocol combination for a transfer
    • Implies relying on an infosys
    • What if there is no protocol advertised?
    • Probably not worth on the short or medium term
  • Automatic session reuse
    • On a first stage, ATLAS will try to group on their side transfers that can be potentially reuse sessions into jobs
    • FTS3 has a prototype implementation capable of automatically deciding if it is worth or not, but disabled due to side effects (see FTS-891)
    • FTS3 aims to fix this scheduling issue for 3.7
  • Network information and SDN integration?
    • Gfal2 has a placeholder plugin to hook logic to notify an SDN controller about a transfer
    • ATLAS will evaluate if it is worth doing this per transfer or per campaign (depending on the time it takes the SDN controller to be set up)
  • File structure knowledge (i.e. transfer files within a tar)
    • Hard to do, since random IO would be needed, and FTS also needs to rebuild the file structure on the destination
    • ATLAS proposes to give the offsets to FTS
    • Not sure what's the protocol status about partial third party copies. FTS to fill this knowledge gap (DMC-960)
  • Activity shares based on throughput
    • A link for a given activity may be saturated with small files, so the throughput is lower even though the priority is higher
    • This problem may be solved with session reuse, since smaller files would be grouped
  • Transfers to object stores
    • Supported by FTS. ATLAS (or whichever VO is interested) need to configure the corresponding keys so FTS can sign the requests.
    • Checksum is not supported on the S3 side (but FTS 3.7 will support source, destination, end2end or no checksums)
    • Checksum validation, size checks and the like have to be disabled for these transfers (strict copies)

Staging

  • Issue: requests piling up due to timeouts (they do not really go away from the tape backend)
  • FTS should probably not try to be clever, or it will defeat the tape system scheduler
  • Need tape experts feedback
  • Note: For ATLAS, simulation data has 1 replica in tape, raw data 2

HTTP Third Party Copies

Pretty much everyone agrees that the +3rd would be better dropped.

It was proposed that, by default, FTS should try third party copy and fail if not supported by any.

However, others propose that FTS should, sometimes, allow to stream the transfer.

davs+3rd was a way of letting the user decide which behavior to use but it is perceived as unintuitive and non standard. Dropping the +3rd means that FTS can

  1. Always try 3rd party copies and fail if not supported
  2. Always try 3rd party copies and always fallback to stream
  3. Always try 3rd party copies and fallback to stream if the user chooses to
  4. By default try 3rd party copy, fail if not supported. The user can ask not to use third party copy

3 and 4 imply the protocol detail will have to leak through FTS all the way up to REST.

To be decided.

AOB

  • The optimizer is not aggressive enough. ATLAS and LHCb agree on this.
  • FTS should start stronger (either faster steps, or start halfway to the maximum). FTS-992.
There are minutes attached to this event. Show them.