Speakers
Description
The dCache project provides an open-source, highly scalable distributed storage system deployed at numerous laboratories worldwide. Its modular architecture supports high-rate data ingestion, WAN data distribution, efficient HPC access, and long-term archival storage. Although initially developed for high-energy physics, dCache now serves a broad range of scientific communities with diverse performance and consistency requirements.
This contribution introduces an automated anomaly detection system for dCache billing data using Long Short-Term Memory (LSTM) neural networks, designed to simplify monitoring and operation of large-scale dCache installations. By analyzing time-series patterns in file access logs, the model learns typical usage behavior and identifies deviations that may indicate operational irregularities, security risks, or performance bottlenecks. The workflow integrates Apache Spark for scalable data retrieval and PyTorch for model training, ensuring efficiency and reproducibility. The LSTM’s memory mechanism enables adaptability to both short-term fluctuations (e.g., access bursts) and long-term trends, while prediction-based anomaly detection—comparing predicted vs. observed values—ensures sensitivity to meaningful deviations. A key challenge involves hyperparameter tuning (e.g., lookback windows, training epochs) to balance accuracy and computational cost.