Usage of analytix cluster by IT Security

Friday 30 Nov 2018, 11:15 → 12:15 Europe/Zurich

31/S-028 (CERN)

· Extrapolating monthly size of bro_* folders and anticipated 5x increase in data volumes of csl_*, execlog & netlog, the estimated capacity requirements is ~3.6 PB (details here)

· Quota to be put in place after implementing functionality to notify users on reaching 85% (WARNING) & 95% (CRITICAL) usage (Hadoop Service)

· Data is kept only for 1 year, after which it is deleted manually

· Discussed the possibility of enabling compression on flume hdfs sink , will not be pursued now as it requires testing to understand data guarantees and most likely compression ratio will not be optimal

· Vincent & Liviu to discuss how IT Monit implements compression and repurpose the functionality if it is suitable (IT Security)

· Evaluate the possibility of using Hadoop Streaming to compress hdfs file (IT Security)

· A new service account will be created by IT Security to run compression workloads, Hadoop service to create a dedicated queue with resource limits (IT Security, Hadoop Service)

· IT security asked us to expire the data older than 1 year from the backups to be complaint with OC11, GDPR (Hadoop Service)

11:15 → 12:15 Optimizing the current usage and future demand