Versioning Room: 31/S-023 CTA v1.0-1 released! Next target is v1.1 (31 January 2020) * Cédric: DB Schema versioning status * Review CASTOR's DB Schema versioning and update procedures * `cta-admin version` command

CTA Website Room: 31/S-023 We need several elements to update the website: * Management monitoring (see below) * Logo: Melissa Loyse Perrey (IR-ECO-DVS) will come back to us with some ideas * Michael: introductory text about CTA

Monitoring Room: 31/S-023 ### Management reporting ("Frédéric plot") We need a common way to store tape usage information for CASTOR and CTA, so we can have a dashboard for each as well as a combined dashboard. Easiest solution is to use a DB, but this is not part of core CTA functionality so should not be in the CTA schema. So, a separate DB for tracking this (preferable to manage this ourselves rather than having relying on monitoring service)? Change from PL/SQL DB job (Oracle dependency) to external polling script (Rundeck?) #### Comments from Germán For monitoring, there are actually two “Frédéric plots” (also used by Ian, Bernd, Alberto and others): * [Volume sent to tape](https://monit-grafana.cern.ch/d/dfpOzRnmz/data-volume?orgId=29) with data in Grafana / InfluxDB both for CASTOR and CTA. What needs to be done on that page is to create an additional plot showing the sum of CASTOR+CTA traffic. This shouldn’t take much effort. * [Data in CASTOR](https://filer-carbon.cern.ch/grafana/d/000000067/castor-dashboard?orgId=1). This shows the amount of ‘live’ (undeleted) data in CASTOR, extracted from the Castor NS. An equivalent plot needs to be created for CTA (extracting information from the CTA catalogue), and as above, we also need an overview plot with the sum of CASTOR + CTA data. #### Comments from Giuseppe For the total namespace-extracted data volume plot, I'm agnostic as to what technology you prefer, but I suggest that the output of the GROUP BY "reduction" query - which one way or another has to be executed against the CTA catalogue with the logic currently in catalogue/oracle_catalogue_usage_stats.sql - be stored in a SQL database (the CTA catalogue itself being the most natural choice, a separate one is also an option). In other words, I won't rely on the storage offered by the monitoring systems - yesterday it was a custom text-based format, today is Grafana's internal format + InfluxDB, tomorrow it will be something else, and we have historical data from 2001... BTW, the data pushed to https://filer-carbon.cern.ch/grafana/d/000000067/castor-dashboard?orgId=1 is also pushed to InfluxDB (work done by D. Lanza). *German:* good and important point. Equally applicable to the tape read/write/mount statistics - we still have an Oracle logging DB that keeps all mount events since a dozen years, having survived 3 different IT monitoring systems... ### Review of previous Action Points Access restrictions on Grafana were put in place as historically the graphs were expensive to generate. Julien will check with monitoring team whether this is still the case and whether we can remove the restrictions. EOS logging: some improvements were made in EOS 4.6.6, some additional logging can go into the next EOS release (Julien will report) Separating/consolidating current dashboards into the "tape operator" and "developer" use cases is targetted for v1.1. * Michael: detail what monitoring is missing (by use case) * Michael: detail what scripts are running on which days (check the Rundeck tape instance) and what data sources they are using #### Dashboards (for reference) We currently have the following dashboards: * [meter-cta.web.cern.ch](https://meter-cta.web.cern.ch/): empty dashboard with no data sources * [monit-grafana.cern.ch/d/000000116/host-metrics](https://monit-grafana.cern.ch/d/000000116/host-metrics?orgId=1&var-availability_zone=All&var-bin=$__auto_interval_bin&var-environment=All&var-hostgroup=tapeserver&var-hostgroup=tapeserver%2Fibm1l8&var-hostgroup=tapeserver%2Fibm355&var-hostgroup=tapeserver%2Fibm360&var-hostgroup=tapeserver%2Fibm3jd&var-hostgroup=tapeserver%2Fibm455&var-hostgroup=tapeserver%2Ft10kd6&var-hostname=All&var-rp=one_week): host metrics for tape servers * [monit-grafana.cern.ch/d/19Z7vUMmk/timings-in-house](https://monit-grafana.cern.ch/d/19Z7vUMmk/timings-in-house?orgId=29): CTA timings and metrics * [monit-grafana.cern.ch/d/5tAh216mk/read-write-throughput-cta](https://monit-grafana.cern.ch/d/5tAh216mk/read-write-throughput-cta?orgId=29&var-vo=atlas&var-vo=cms&var-clientHost=All): Realtime tape I/O performance * [filer-carbon.cern.ch/grafana/d/000000036/eos-control-tower](https://filer-carbon.cern.ch/grafana/d/000000036/eos-control-tower?orgId=1): EOS Control Tower (filter by EOSCTA instance) * [monit-grafana.cern.ch/d/dfpOzRnmz/data-volume](https://monit-grafana.cern.ch/d/dfpOzRnmz/data-volume?orgId=29): Data Volume, (including "Frédéric’s plot”), both CASTOR + CTA (scroll down for CTA) * [monit-grafana.cern.ch/d/rgcoo0vmk/tape-stats-by-density-history-castor-cta](https://monit-grafana.cern.ch/d/rgcoo0vmk/tape-stats-by-density-history-castor-cta?orgId=29): Tape stats by density, both CASTOR + CTA * [monit-grafana.cern.ch/d/000000786/infrastructure-castor-and-cta](https://monit-grafana.cern.ch/d/000000786/infrastructure-castor-and-cta?orgId=29): Infrastructure, both CASTOR + CTA * [monit-grafana.cern.ch/d/obWK_dAik/tape-pools-statistics](https://monit-grafana.cern.ch/d/obWK_dAik/tape-pools-statistics?orgId=29): CTA tape pool statistics

Testing Room: 31/S-023 There are about 15 tickets tagged with "Testing" which need to be done. Can everyone pick one and implement the test? If we all do 3 tests each they will be covered.

Repack Room: 31/S-023 ATLAS has to be repacked and it's probably not possible to do it before migration to CTA (1500 tapes, Vlado estimates 3 months) We will take a 2-pronged approach: 1. Vlado to repack as much of ATLAS as possible in CASTOR within the constraints of the reprocessing campaign. ATLAS recall exercise must achieve 10 GB/s for a sustained period (min. 24 hours/max 5 days) so no repack can be done during this stress test. 2. Fix CTA repack: * EOS CLOSEW fixes (see below) * Are there any CTA repack issues which are blocking? * Repeat CTA repack tests #### Comments from Germán I recommend to not entangle IBM 7TB tape repacking with ATLAS testing/migration, as it’s a potential source for inconsistencies and headaches for CASTOR-CTA metadata and tape consistency. There is no operational hurry to start CASTOR repacking now as we have sufficient free IBM enterprise slots (corresponding to 180PB of additional data - not counting free LTO slots on LIB1 nor on the new Spectralogic library). I suggest that 7TB repacking is postponed after the switchover and used as a field test for mass CTA repacking. There will be sufficient time to complete 7TB repacking before Run-3 (and could even stretch into it if really necessary).

Other known issues for v1.1 milestone Room: 31/S-023 Outstanding issues on EOS: * Improvements to eosreport * Do not execute CLOSEW in case of error flushing file to disk * Delete file if CTA CLOSEW event returns an error * "has a tape backend" configuration option * Refactoring of FTS close write code Outstanding CTA issues: * Issues flagged up by testing * Issues tagged "Tape Server" * Issues tagged "Operator Tools/Frontend"

AOB Room: 31/S-023 * Date of section lunch: Tue 17 Dec