CTA operations

Wednesday 12 Aug 2026, 10:30 → 12:00 Europe/Zurich

513/R-068 (CERN)

Julien Leduc (CERN)

This meetings covers EOSCTA service operations at CERN: minutes, community

 

10:30 → 10:40 Rota

• Review the past week rota
• Operations forecast for next week
  • Ongoing significant user activities
  • Planned interventions that affect CTA

Speaker: Julien Leduc (CERN)

• Rota ticket
  • 2026-08-05/2026-08-12
• Upcoming interventions affecting CTA
  • 2026-08-17 10:00/10:30: OTG0162677 - CTA Service unavailable during database intervention
    • opportunistic intervention as tape activity is stopped: [2016-08-17 10:00/10:30] Upgrade remaining eoscta instances to 5.3.38
• Upcoming meeting feedback
  • eos devops (Pedro): When is PDS planning to switch to eos 5.5 in production?

10:40 → 10:45 Tape infrasrtructure

Ongoing tape hardware orders and tape infrastructure activities

Speaker: Vladimir Bahyl (CERN)

• Diamond back test library is broken (IBMLIBD)
  • accessor changed on 2026-08-07
  • supposed to be user serviceable but DOWN longer than other libraries
    • OTG0162616 - IBMLIBD tape library unavailable: created as CMS is recalling some files from there
• Next big thing: Delivery of SPECTRALIB4
• rename of tape libraries, tape pools,...
  • finalize implementation of CTA tapepool usages and conventions
• conventions may evolve to contain library name, lower tapepool count,...

10:45 → 11:05 Use cases

Review operations tickets related to current context

• Review incidents and next identified steps in created linked issues
• software deployment planning
  • report ongoing tests and integration
  • announce upcoming deployments
• Tickets that require a short discussion
• List item

Speaker: Julien Leduc (CERN)

• [2026-07-26] EOS CRL incident
  • Conversion was turned off in EOS 5.3.16, affected eosctapublicdisk 1 year after it already affected eosctaalice...
    • important to keep software stack consistent accross the service to avoid double work
  • [2016-08-17 10:00/10:30] Upgrade remaining eoscta instances to 5.3.38
• Drive process of cta-taped is core dumping
  • Some fixes needed
• Deploy CTA v5.11.18.1-1
  • Need v5.11.18.2-1 after incident
• Upgrading CTA rpms changes local service configuration
  • Best practices needed
• cta-verify-file jobs hang and gradually consume all memory on CTA Frontend
  • Stuck SSI client requires some workarounds until xrootd is gone
  • Fixes for gRPC versions of standalone_cli_tools in a ctaops module ? (ticket needed)
• Review Mattermost alarm channel purposes
  • Document purpose of alarm mattermost channels by audience

11:05 → 11:10 AOBs

Anything else that we may have forgotten relevant to the operations audience in the middle of the week.

This could also be a reminder for other matters (upcoming meetings,...).

Speaker: Julien Leduc (CERN)

11:10 → 11:45 Topic of the day

Feel free to leave: AT THIS POINT THE OPERATIONS MEETING IS FINISHED

Topics are announced at the beginning of this session and interested people stay to discuss them.

Discussed topics:

• 1 or 2 pre-selected topics requiring discussions announced before the meeting
• space for overflow discussion in case one Use case ticket requires it

Topic of the day

After operations meeting on 2026-08-12

Participants

@vlado, @mdavis, @pferreir, @okilicki and @jleduc

Decisions:

• move ch subcommand to tf command
• options:

cta-admin tf ch --storageclass/-s <storage_class_name> [--fxidfile/-F <filename>] [--fxid/-f <eos_fxid>]
   [--diskfileid/--dfid <disk_file_id>] [--instance/-i <disk_instance>]
   [--id/-I <archive_file_id>]

Note:

• --fxidfile and --fxid are EOS specific and cannot be used by dCache
• no batching on integer representation or dCache files (another concern)

Will go to 5.11.18.3-1 in 5.11.18 branch for deployment on ctaproductionfrontend13 for @vlado to test sc changes in production.

Statistics to be cherry-picked in the same release.