Choose timezone
Your profile timezone:
EOS DevOps Meeting
Description
Weekly meeting to discuss progress on EOS rollout.
- please keep content relevant to (most of) the audience, explain context
- Last week: major issues, preferably with ticket
- This week/planning: who, until when, needs what?
Add your input to the "contribution minutes" before the meeting. Else will be "AOB".
● EOS production instances (LHC, PUBLIC, USER)
LHC instances
- Updated ATLAS and CMS headnodes to 4.2.23
- There's a "protocol" change that requires the FSTs to run the same version (rewrite of the atomic upload feature) which can lead to having the FST deleting the file under some rare circumstances, had an impact on EOSATLAS as they use the feature to upload files.
- AP: "file delete" also happened with earlier versions. Now MGM sends new error code, only 4.2.23 knows how to handle
- There's a "protocol" change that requires the FSTs to run the same version (rewrite of the atomic upload feature) which can lead to having the FST deleting the file under some rare circumstances, had an impact on EOSATLAS as they use the feature to upload files.
- Updated FSTs of these 2 instances to 4.2.23 as a consequence
- Please clarify the situation of "orphan files" in PUBLIC (posted by Massimo)
- Namely the risk to lose them via "automatic repair"
- Elvin: boot FSTs filesystem-by-filesystem, number of orphans will shrink and settle at "real" number. AP: there is an additional check, "fsck repair" has safeguard against false orphans.
- Hervé: not running 4.2.23 - update FSTs first.
- Namely the risk to lose them via "automatic repair"
- New EOS Control Tower contains Atlas Tier0 activity and ALICE(DAQ) P2 CDR
- Hervé cannot find this anymore?
Andreas:"catastrophe"?
● EOS clients, FUSE(X)
(Andreas) eosxd
- fixed file recovery mechanism recovering files, which have been deleted in the meanwhile
- fixed inlined file functionality (works now for full compilations)
- fixed bug with too small buffer used when recovering a file from remote
- fixed auto-tuning mechanism in read-ahead mechanism ... now reaching 600 MB/s streaming read if FST can provide the data
- deep-dived into Steve Traylon /eos/scratch report about building kernel on eosxd
- fixed missing logic to update symbolic link meta data server side
- found generic kernel FUSE deadlock of Linus Torvals code (recordmcount) doing fancy mmap + write
- 100% reproducable on EL7 also with ceph-fuse
- did some performance comparisons with ceph-fuse
- added ACL support for authorization by key which is transported via sss endorsement
- allows clients to access to certain directories setting an environment variable key as authorization
- requires XRootD modification (did merge request, but no response yet)
- implemented squashfs console tool allowing to pack,unpack squashfs images
(Massimo)
- EOS-2625: Lost possibility to mount uat (bad for scale test):
- LXPLUS allows user-mount, and has /eos/scratch. But the FUSEX cache needs (user-)xattr (otherwise abort()s on write()), not enabled anywhere as mount option. EOS-2172
- BATCH did not yet have /eos/scratch, except for the "batchtest" sub-cluster (and need at least one FUSEX mount to install the "eosxd" RPM).
- EOS-2621 : Metadata problem (invalid mtime) from concurrent opening of a file (on FUSE). Not clear the effect on FUSEX (seems less likely but I can block the mount though :(
Anderas: need a decision whether to cherrypick all production+FUSEX fixes (currently are on "dev" branch), or release as "dev" version, or merge all into "master". Numbered releases happen from "master".
Luca: afraid of changes e.g on locking - last time such a change caused months of instability. How to evaluate/validate? (EOSPPS is on "dev" and has this already). How to provoke such load-related locking issues - without installing on EOSATLAS (or similar)? Ops probably cannot meaningfully test this via scripts (but at least need to make an effort..). Can put this on smallest prod instance but might have not enough load.
Massimo: ATLAS wants postmortem on 12h stop, should include our efforts on testing, and proposal to update during tech stop (MD on 12.06, tech stop 18.06. - 21.06)
EOSPPSLEGACY is also closer to ATLAS - should install new release there. Have to add a few FSTs.
Also move EOSUAT to citrine? cannot validate "aquamarine" = EOSUSER anymore
Andreas: explains background: old locking favoured reads; Massimo then started heavy write, so new code preferred writes, so starved readers - no "good" solution for all use cases. Georgios has some read/write tests (should concentrate on metadata, dir creation), Luca suggest to add ops stuff such as drain/balance/boot). Do we need a "mock" infrastructure (could replay from log?)? somebody would need to implement this.. and cannot within one week.
Decision:
- merge locking commits (for the ATLAS incident), and all of FUSEX changes -> new release.
- deploy on EOSPPS and EOSPPSLEGACY, test there
- Massimo to run metadata-heavy tests (mkdir())
- Georgios to run read/write tests
- Ops should exercise drain/balance/boot
- deploy to production EOSATLAS during tech stop
Kuba - have example of contributed test by ops - "git checkout" timing test
Dan: have cleanup script for eosxd, needs one "real" case to try this out. Please tell him in case you find one stuck "eosxd" mount..
Elvin: no 4.2.24 release yet.
● Development issues
(Massimo)
- Which is the procedure to contribute tests? How can I contribute tests requiring 2 different nodes to do things "at the same time"?
- (ask Kuba, just doing this- can document on "EOS ops gitbook" or "EOS community")
Luca: updated 1.5 EOSHOME instances (to GIT snapshot,), everything broken (but at xroot level - SSS auth? FST crash?)
● AOB
(Jan, after meeting) - Alberto sent PoW slides that contain planning info - these have been agreed with Luca.
EOS detailed plan (with dates)
- May 2018
- New deployment preparation
- Batch tests (new FUSE)
- New MGM stress tests
- EOSHOME00 up
- June 2018
- EOSHOME01 up
- Migration tool to move users from EOSUSER to EOSHOME)
- Move of IT-ST accounts (EOSUSER aka CERNBox)
- July - September 2018
- Move of IT accounts
- New accounts are created on EOSHOME
- Before end of 2018
- Transparent move of larger groups
- Finalise the move
- e.g. Critical account that might have an impact on LHC data taking
FuseX detailed deployment plan
- May 2018
- FuseX deployed under /eos/scratch on lxplus + lxbatch
- Validations and tests form ST, CM, CF, CDA and several other IT power users
- June 2018
- Scale test on /eos/scratch: minimum support for 2000 simultaneous clients
- July -August 2018
- Enable FuseX on EOSHOMExx and on EOSLHCB.
- FuseX will become the default access software for migrated eos users and for everything under /eos/lhcb
- September - End of 2018
- Following the LHCB validation, transparent move of all other instances
There are minutes attached to this event.
Show them.