HTTP Deployment Task Force

Wednesday 15 Jul 2015, 16:00 → 18:00 Europe/Zurich

31/S-023 (CERN)

AGENDA

• Monitoring - functional validation, availability
  • Probes/tests
    • What do we want to test?
    • Existing tests for data access (HTTP or other)
    • Probe implementation and maintenance
  • Frameworks
    • How should we organise test scheduling, result transport & storage, visualisation etc
    • SAM, HammerCloud, ...
  • Configuration dbs / infosys
    • How do we know which endpoints to test?
• Data Access Monitoring
  • Current status
    • The xrootd model (UDP, GLED etc), supported by xrootd and DPM
  • Complementary solutions
  • A proposal: http://wdtmon.web.cern.ch/wdtmon/dash/http.html
• Update on outstanding questions on HTTP functionality from meeting on 3rd June

ADDITIONAL MATERIAL

• https://twiki.cern.ch/twiki/bin/view/LCG/HTTPDeployment

Minutes for HTTP TF meeting 15 July 2015:

https://indico.cern.ch/event/401396/

 

Functional validation

The TF decided to adopt the approach of a Nagios plugin, common to the experiments and maintained by CERN IT-SDC, which would be integrated into each experiment SAM instance. The feasibility of this approach is dependent upon technicalities related to the use of configuration databases other than GOCDB, ie the requirement to take the endpoints to be tested from either Agis or the LHCb CS. To be followed up with CERN IT-SDC-MI who are responsible for SAM.

The tests will represent the functionality declared in the previous meeting as “required”, with the addition of a check that the actions of the probe are properly reported through monitoring systems (where this check is possible).

Action : Atlas and LHCb to inform the TF how to acquire the list of endpoints to be tested.

Action : Oliver to take proposal to SAM team and report back to TF.

Access monitoring

Luca Magnoni presented a summary of the current situation and described two different ways to support HTTP access monitoring:

• A properly formatted UDP stream, as implemented in the xrootd f-stream.

  • http://xrootd.org/doc/dev4/xrd_monitoring.htm#_Toc346032371

• Access summaries conforming to a documented json schema delivered over the messaging system

  • http://wdtmon.web.cern.ch/wdtmon/dash/http.html

DPM (and xrootd of course) support the xrootd f-stream solution. dCache and StoRM will consider their options.

 

Discussions on json schema:

Q: Should failures be reported?
A: Add a “final resolution” field which can indicate success or error code

Q: Are metadata operations expected to be logged?
A: No, only I/O

Q: Should 3rd party copies be explicitly identified?
A: Could use Application Specific Metadata to indicate who initiated the transfer. Note that this is unstructured data and any general consumption of this info would benefit from a format convention. This metadata is transmitted via a "ClientInfo" header (details below).

Q: Can VO be made mandatory?
A: Yes, but can be NULL.

Q: What is the relation between [server|client]_domain and [server|client]_host?
A: Difficult to define usefully in a distributed system – drop *_domain from the schema.

Q: VO, FQAN, role can all be multiple, how should this be handled?
A: No answer for now – just implement “first” in the list.

Q: What about attributes which may be subject to constraints on transmission outside a particular jurisdiction (e.g. sending the DN over the Atlantic)?
A: These are all optional, that's enough as far as the schema goes. Storage providers can make their publication configurable if they wish.

 

Other decisions:

Transmission of application meta-data should be via the interface recently agreed between the FTS team and dCache, namely the “ClientInfo” HTTP header with an arbitrary string as the value. (https://its.cern.ch/jira/browse/FTS-245 for some more information).

 

Update on outstanding questions on HTTP functionality from meeting on 3rd June

LHCb needs ADLER32 checksum support.

 

AOB

Next meeting : 16^th September, 16hr.