Speaker
Dr
Takashi SUGIMOTO
(Japan Synchrotron Radiation Research Institute)
Description
An X-ray free electron laser (XFEL) facility, SACLA, is generating ultra-short, high peak brightness, and full-spatial-coherent X-ray pulses [1]. The unique characteristics of the X-ray pulses, which have never been obtained with conventional synchrotron orbital radiation, are now opening new opportunities in a wide range of scientific fields such as atom, molecular and optical physics, ultrafast science, material science, and life science. More than 100 experiments have been performed since the first X-ray delivery to experimental users in March 2012. In this paper, we present an overview of SACLA data acquisition (DAQ) and analysis system with a special emphasis on the analysis scheme and its infrastructure. In the case of serial femotosecond protein crystallography experiments [2], a typical experiment collects diffraction image patterns of order of $10^6$, which demands heavy load to the data analysis system. Each pattern is recorded by a $2000 \times 2000$ pixel detector that consists of eight Multiport Charge-Coupled Device (MPCCD) sensors [3]. The resolution of the single MPCCD sensor is $1024 \times 512$ pixel and data depth of each pixel is 16 bits. The DAQ system consists of detector front-ends [4], data-handling servers, hardware-based event-tag distribution system [5], event-synchronized database [6], two cache storages, tape archive system, and physically-segregated two network system [7]. The DAQ has data bandwidth of maximum 6 Gbps to support other experiment setups with various detector configuration of up to twelve MPCCD sensors. In addition to the currently operational beamline BL3, BL2 will operate concurrently through a fast-switching operation mode in 2015. To support this operation mode, the cache storage with capacities of 200 TB (250 TB) is assigned to the beamline BL2 (BL3) respectively [8]. These capacities correspond to the accumulated data size for one week operation. Experimental data are periodically moved into the tape archive system. The tape archive system has a capacity of 7 PB, and extendable up to 26 PB by installing additional tape cartridges.
The analysis section has two functions: one is run-by-run analysis to monitor the experimental conditions, and the other is off-line analysis. To implement these functions, the analysis system consists of a PC cluster and a supercomputer. The PC cluster is based on x86_64 processors and has a computing power of 14 TFLOPS. A 160 TB storage is connected to the PC cluster via Infiniband QDR network. To pick up raw image data, the PC cluster is connected to the cache storages and the tape archive system via 10 Gigabit Ethernet. The run-by-run analysis is performed using the PC cluster. The results are saved on the storage in HDF5 format [9]. The PC cluster is also used for off-line analysis, using analysis code developed by the scientific community, such as CrystFEL [10] and SITENNO [11]. The supercomputer with 90 TFLOPS SPARC-based processors (Fujitsu FX10) was installed for the data analysis that requires higher computing power. Storage of the supercomputer is 100+500 TB Lustre-based file system (Fujitsu FEFS). Another 1 PB Lustre file system is also connected to both the supercomputer and the PC cluster to interaccess the experimental data from the two systems. Data analysis that requires much higher computing power is foreseen. For these cases, we are developing a joint analysis mode using both the supercomputer and the 10-PFLOPS K computer [12]. The results of the feasibility study on data transfer and quasi-realtime job submission to the K computer will also be discussed.
[1] T. Ishikawa et al., "A compact X-ray free-electron laser emitting in the sub-angstrom region", Nature Photonics 6, 540-544 (2012).
[2] M. Sugahara et al., Nature Methods submitted.
[3] T. Kameshima et al., "Development of an X-ray pixel detector with multi-port charge-coupled device for X-ray free-electron laser experiments", Rev. Sci. Instrum. 85, 033110 (2014).
[4] A. Kiyomichi, A. Amselem, et al., "Development of Image Data Acquisition System for 2D Detector at SACLA", Proceedings of ICALEPCS2011, WEPMN028 (2011).
[5] T. Abe et al., "DEVELOPMENT OF NEW TAG SUPPLY SYSTEM FOR DAQ FOR SACLA USER EXPERIMENTS", Proceedings of IPAC2014, TUPRI108 (2013).
[6] M. Yamaga et al., "Event-Synchronized Data Acquisition System of 5 Giga-bps Data Rate for User Experiment at the XFEL Facility, SACLA", Proceedings of ICALEPCS2011, TUCAUST06 (2011).
[7] T. Sugimoto et al., "Large-bandwidth Data-acquisition Network for XFEL Facility, SACLA", Proceedings of ICALEPCS2011, WEBHAUST03 (2011).
[8] K. Okada, et al., "UPGRADE OF SACLA DAQ SYSTEM ADAPTS TO MULTI-BEAMLINE OPERATION", Proceedings of PCaPAC2014, WCO205 (2014).
[9] The HDF Group, http://www.hdfgroup.org/HDF5/.
[10] T. A. White, et al.. "CrystFEL: a software suite for snapshot serial crystallography". J. Appl. Cryst. 45, pp.335–341 (2012).
[11] Y. Sekiguchi et al., "Data processing software suite SITENNO for coherent X-ray diffraction imaging using X-ray free electron laser SACLA", Journal of Synchrotron Radiation 21/ 4, 600-612 (2014).
[12] M. Yokokawa, et al., "The K computer: Japanese next-generation supercomputer development project", ISLPED '11 Proceedings of the 17th IEEE/ACM international symposium on Low-power electronics and design, pp.371-372 (2011).
Primary author
Dr
Takashi SUGIMOTO
(Japan Synchrotron Radiation Research Institute)
Co-authors
Dr
Kensuke OKADA
(Japan Synchrotron Radiation Research Institute)
Dr
Makina YABASHI
(RIKEN SPring-8 Center)
Dr
Mitsuhiro YAMAGA
(Japan Synchrotron Radiation Research Institute)
Dr
Ryotaro TANAKA
(Japan Synchrotron Radiation Research Institute)
Dr
Takaki Hatsui
(RIKEN SPring-8 Center)
Dr
Takashi KAMESHIMA
(Japan Synchrotron Radiation Research Institute)
Dr
Toshinori Abe
(Japan Synchrotron Radiation Research Institute)
Dr
Yasumasa JOTI
(Japan Synchrotron Radiation Research Institute)
