Choose timezone
Your profile timezone:
RAL Tier1 Experiments Liaison Meeting
→
Europe/London
Access Grid (RAL R89)
Access Grid
RAL R89
RAL Tier1 Experiments Liaison Meeting
-
-
Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)
Operational issues:
- Gateway problems after DNS change last Thursday (GGUS:1001119)
- Some gateways redirected to IP addresses rather than names
- That causes authentication failures since addresses are not in cert CN, nor in SAN list
- Could be because of the peculiar PTR records in the reverse DNS zone for this machines
- Some redirected to
(s3)?.echo.stfc.ac.ukalias
- Redirecting to
(s3)?.echo.stfc.ac.ukis problematic for two reasons:- it breaks load balancing
- there were 3 hosts without SANs behind this alias (svc0[789]), that caused authentication failures
- Redirecting to
- Summary in the diagram below:
- Some gateways redirected to IP addresses rather than names
-
- On Friday the gateways that were "addressed" by their IPv6 addresses by the redirector (3rd from the left branch in the diagram above) were removed from prod
- On Saturday many gateways were overloaded (redirection to
(s3)?.echo.stfc.ac.ukmight have played a role in it) and killed by OOM-killer. Annoyingly enough, EL9 ones did not restart after that, because of the missing systemd unit file fix. Fixed on Saturday afternoon. - On Monday gateways without SAN were removed from the
(s3)?.echoaliases, and two more gateways (correctly configured, without DNS problems) were added to prod, that fixed (mitigated) the problem
- Bookkeeping DB outage on Monday.
- Bookkeeping DB at CERN is down since ~11am Monday
- Because of it many DIRAC services are down as well, and the grid is drained.
- Expected to be back this afternoon.
- Gateway problems after DNS change last Thursday (GGUS:1001119)
-