Speaker
Description
Large Language Models (LLMs) are transforming software development and data analysis workflows in many fields, including nuclear and particle physics experiments.
However, deploying LLMs in production research environments requires careful attention to scalability, security, and resource efficiency.
This work presents a versatile production-grade LLM inference and document intelligence server designed for GSI/FAIR computing infrastructure.
The system employs a modern containerized architecture built on vLLM for GPU-accelerated inference, OpenWebUI for user interaction, and Milvus vector database for Retrieval-Augmented Generation (RAG) capabilities.
The deployment utilizes several specialized vLLM instances running different model sizes optimized for specific tasks: a flagship thinking multi-modal model, a faster thinking model for text-based general-purpose tasks, a smaller multi-modal model for fast inference, and a dedicated embedding model for document vectorization and retrieval.
This multi-model approach, deployed on an 8xH200 GPU server, balances computational cost with task-specific performance requirements.
The RAG implementation enables users to upload and query documentation, code repositories, and technical papers using natural language.
API access allows users to explore and deploy their own solutions while leveraging high performance hardware.
The architecture supports production deployment on a server with enterprise authentication and development configurations for testing on resource-constrained hardware.
Performance characteristics, deployment patterns, and lessons learned from production usage are presented, along with token consumption metrics across different workflows.
This work demonstrates that modern LLM technologies can be successfully integrated into GSI/FAIR computing infrastructure while maintaining security, auditability and performance.