WLCG DOMA BDT Meeting

Wednesday 7 Jun 2023, 16:30 → 17:30 Europe/Zurich

Brian Paul Bockelman (University of Wisconsin Madison (US)), Maria Arsuaga Rios (CERN), Petr Vokac (Czech Technical University in Prague (CZ))

Topic: WLCG DOMA BDT Meeting (twiki)

1 News

2 Tape REST access

Speaker: Mihai PATRASCOIU (CERN)

3 Transfers with tokens

Speaker: Francesco Giacomini (INFN CNAF)

Introduction for FTS transfers with tokens

• basic draft sent by FTS team to wlcg-doma-bdt email group
• discussed related question about storage support for renaming files
  • normal for disk storage, but not so common for TAPE
    • no rename on CTA, name or path is used to decide which tape to use
    • dCache use PNFSID for TAPE - no change during file rename
    • StoRM?
  • also not common for object storage
    • rename would require a copy of data
• decouple token support in two independent steps (Brian)
  • step 1: start with powerful tokens with delete (may be even delete whole storage)
  • step 2: reduce token privilege
    • make special upload area and atomic rename
    • failed transfers could be safely cleaned
  • CMS suggestion
    • one token per RSE with storage.modify
    • avoid temporary staging areas

4 Packet marking

Speakers: Marian Babik (CERN), Shawn Mc Kee (University of Michigan (US))

Meeting with EOS/FTS teams last week, we have agreed to go ahead and explore what needs to be done and when (in what sequence). Technical discussions will follow up.

Proposal to change Scitags HTTP/TPC extensions and make it more compact, introducing just one header:

• TransferFlow:  <expID> << 6 | <actID>
• TransferHeaderFlow:  <expID> << 6 | <actID>
• expID and actID are numeric codes as found in the scitags registry API (api.scitags.org)

Technical specification was updated to refer to numeric codes (expID/actID) in HTTP/TPC proposal, this was agreed some time ago but wasn't updated in the spec. 

Submitted SC23 NRE - https://docs.google.com/document/d/1tYqihu7Ts_ohe0Ylakq50BflzRmGl93r/edit?usp=sharing&ouid=112028852513778613204&rtpof=true&sd=true

Scitags project will be presented this week at the TNC2023 (https://tnc23.geant.org/sessions/#s269).

Reminder: Packet marking now a regular topic in the LHCONE R&D Call agenda (bi-weekly, next meeting will be Tuesday 20th of June at 17:00 UTC+2, this week skipped due to TNC conference).

 

Discussion notes - Scitags HTTP-TPC extension:

• From FTS developers point of view structure of headers not important / can be opaque, prefer one input that define flow details
• Marian will send email with more details to speedup final specification
• Links to final specification will be included in our HTTP-TPC extension updates

5 WebDAV Error Message Improvement Project & unified error message format

Discuss with experts improvements in the error messages produced by failed transfers.
https://twiki.cern.ch/twiki/bin/view/LCG/WebdavErrorImprovement

Speaker: Stephan Lammel (Fermi National Accelerator Lab. (US))

6 AOB

HTTP-TPC protocol update#4

• discussion issue#7058 was "stuck on XRootD"
• updated proposal
  • use (multiple) Connection lines
  • connection data structure protocol:source_addr:source_port:destination_addr:destination_port

Connection: tcp:[2001:718:401:6017:2::28]:24081:[2001:1458:301:105::100:5]:8443\n

• new perf marker could look like (in this case with 2 connections)

Perf Marker\n
Timestamp: 1537788010\n
Stripe Index: 0\n
Stripe Bytes Transferred: 238745\n
Total Stripe Count: 1\n
RemoteConnections: tcp:147.231.25.166:21234,tcp:[2001:718:401:6017:2::28]:24081\n
Connection: tcp:147.231.25.166:21234:128.142.49.200:8443\n
Connection: tcp:[2001:718:401:6017:2::28]:24082:[2001:1458:301:105::100:5]:8443\n
End\n

• deprecate "Stripe" lines and add useful alternatives: "Bytes Transferred"

Perf Marker\n
Timestamp: 1537788010\n
Stripe Index: 0\n
Stripe Bytes Transferred: 238745\n
Total Stripe Count: 1\n
Bytes Transferred: 238745\n
RemoteConnections: tcp:147.231.25.166:21234,tcp:[2001:718:401:6017:2::28]:24081\n
Connection: tcp:147.231.25.166:21234:128.142.49.200:8443\n
Connection: tcp:[2001:718:401:6017:2::28]:24082:[2001:1458:301:105::100:5]:8443\n
End\n

• after transition period we would like to remove deprecated entries

Perf Marker\n
Timestamp: 1537788010\n
Bytes Transferred: 238745\n
Connection: tcp:147.231.25.166:21234:128.142.49.200:8443\n
Connection: tcp:[2001:718:401:6017:2::28]:24082:[2001:1458:301:105::100:5]:8443\n
End\n

StoRM progress with HTTP-TPC push to GOOGLE GGUS:158487

• GGUS ticket updated with progress details
• ATLAS integrated GOOGLE RSE
  • currently Rucio multi-hop necessary for transfers from StoRM to GOOGLE

Next meeting on July 21 dedicated to Rucio & FTS with tokens (conflict with EGI Conference 2023)