• Hanging Judge evaluator
  • Recently there was an issue of a hanging judge evaluator which lead to some unwanted replica deletion
  • As the application of replication rules (replica locks) to files is done asynchronously for performance reasons, a specific workflow lead to the deletion of replicas which should not have been deleted
    • 1: File₁ with Replica₁^A gets added to Dataset_X (which has a rule for RSE^A)
    • 2: Dataset_X gets deleted --> Rule gets removed --> Replica₁^A gets a tombstone
    • 3: File₁ gets added to Dataset_Y (which has a rule)
      • As the Judge evaluator was hanging, Replica₁^A never got the tombstone removed
    • 4: As RSE^A is full, Replica₁^A gets deleted within 3-4 hours
  • This was the first time the evaluator was hanging since 2014
  • The workflow was adapted to not remove Dataset_X immediately, but give it an expiration of 6h
  • Discuss if #1578 should be implemented, which would prevent the reaper from deleting data if it detects a backlog in the evaluator
    • --> Go ahead with the check