Error summary CSCS-LCG2 ANALY_CSCS-HPC ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4654387059 25/2/2020 12:50 nid01762 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: nid01762 pilot:::1310 Exception caught during payload execution 4634209210 3/2/2020 19:12 nid01885 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: nid01885 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-03 19:16:08 sup:::2 worker already finished at 2020-02-03 19:29:24 with None 4631738052 1/2/2020 20:42 nid01715 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: nid01715 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 20:46:09 sup:::2 worker already finished at 2020-02-01 21:03:12 with None 4656819359 28/2/2020 0:20 nid01839 data18_13TeV:data18_13TeV.0034926 modificationHost: nid01839 ddm:::200 expected output 7470c493-e717-4119-b172-d598e326731c_98178.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : LRMS error: (1) Job failed 4655694986 26/2/2020 23:09 nid01784 data18_13TeV:data18_13TeV.0034926 modificationHost: nid01784 ddm:::200 expected output 13598583-259c-47d1-a925-95b4d40295d3_1795.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasscratchdisk/rucio/user/gangarbt/c3/2d/user.gangarbt.13598583-259c-47d1-a925-95b4d40295d3_1795.DAO 4655865363 27/2/2020 2:44 nid01806 data18_13TeV:data18_13TeV.0034926 modificationHost: nid01806 ddm:::200 expected output acb2945f-5231-477b-9227-b841b1bb6c1f_23105.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasscratchdisk/rucio/user/gangarbt/43/f9/output.1.acb2945f-5231-477b-9227-b841b1bb6c1f_23105.pool.ro 4655992876 27/2/2020 6:16 nid01876 data18_13TeV:data18_13TeV.0034926 modificationHost: nid01876 ddm:::200 expected output 2722be02-5d82-49dc-a3d1-b285220a18a4_37897.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasscratchdisk/rucio/user/gangarbt/84/b6/output.1.2722be02-5d82-49dc-a3d1-b285220a18a4_37897.pool.ro 4631556586 1/2/2020 18:41 nid01699 data18_13TeV:data18_13TeV.0034926 modificationHost: nid01699 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 18:53:10 sup:::2 worker already finished at 2020-02-01 19:09:28 with None 4655950967 27/2/2020 5:10 nid01808 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: nid01808 ddm:::200 expected output user.gangarbt.6ea02af0-94fd-4666-998f-c7cfa9058062_14114.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasscratchdisk/rucio/user/gangarbt/b4/dc/user.gangarbt.6ea02af0-94fd-4666-998f-c7cfa9058062_14114.jo CSCS-LCG2-HPC_MCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4655267043 27/2/2020 6:51 nid01846 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01846 ddm:::200 expected output a82bb569-c81e-4ffb-a704-8071652363a6_35651.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/10/9f/a82bb569-c81e-4ffb-a704-8071652363a6_35651.job.log.tgz:checksumtype= 4633696508 3/2/2020 11:02 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: NULL ddm:::200 expected output 8352d10e-1ed4-4048-9005-439caad73095_95734.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : LRMS error: (15) Job was cancelled by SLURM 4650590928 20/2/2020 9:21 nid01811 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01811 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-20 09:32:21 4632956019 2/2/2020 23:01 nid01848 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01848 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 23:14:51 sup:::2 worker already finished at 2020-02-02 23:27:02 with None 4655900505 27/2/2020 8:07 nid01819 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01819 ddm:::200 expected output e48e9d3c-fb5a-4c5b-9894-b5f1d67b7bc5_80249.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/bf/7e/Hits.hc_20163526.CSCS-LCG2-HPC_MCORE.e48e9d3c-fb5a-4c5b-9894-b5f1d67 4653581322 24/2/2020 9:06 nid01806 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01806 pilot:::1187 Payload metadata does not exist 4633886364 3/2/2020 11:35 nid01853|nid0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01853|nid01885 ddm:::200 expected output fed96a75-12d9-4852-97a5-e0e0f85cc959_52575.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Error reading user generated output file list 4633045610 2/2/2020 23:01 nid01903 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01903 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 23:22:07 sup:::2 worker already finished at 2020-02-02 23:34:27 with None 4658761829 1/3/2020 10:39 nid01847|nid0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01847|nid01912 ddm:::200 expected output 6010a538-f92a-4267-b63e-1a67edb47b36_38324.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : LRMS error: (1) Job failed 4655152775 26/2/2020 23:57 nid01735 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01735 ddm:::200 expected output 737546f2-da5b-4c83-8abb-eda61ea7f09f_2826.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/3a/14/Hits.hc_20163385.CSCS-LCG2-HPC_MCORE.737546f2-da5b-4c83-8abb-eda61ea 4655153061 26/2/2020 23:52 nid01706 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01706 ddm:::200 expected output 72b2222a-e66d-462b-a94e-1663b18762a8_48889.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/b7/8b/output.1.72b2222a-e66d-462b-a94e-1663b18762a8_48889.pool.root:checks 4640898245 9/2/2020 12:50 nid01718 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01718 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4640313398 8/2/2020 23:18 nid01699 hc_test:mc15_13TeV.361106.PowhegP modificationHost: NULL ddm:::200 expected output 772bf805-32a7-4cd2-9f5b-1d4d93d1e5fe_39000.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Error reading user generated output file list 4634213589 3/2/2020 23:06 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: NULL ddm:::200 expected output 8411333b-bd3e-432c-ae9a-53ad5aaf64c7_21296.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Error reading user generated output file list 4631343237 1/2/2020 19:58 nid01840 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01840 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 20:18:50 sup:::2 worker already finished at 2020-02-01 20:28:32 with None 4655148499 27/2/2020 1:53 nid01854 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01854 ddm:::200 expected output 9b5a33d2-b689-44ee-91f6-faeb754d4104_75723.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/01/76/Hits.hc_20163381.CSCS-LCG2-HPC_MCORE.9b5a33d2-b689-44ee-91f6-faeb754 4655957759 27/2/2020 8:46 nid01760 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01760 ddm:::200 expected output 241d943c-bd6c-46e5-9151-d0f3c942f3c1_33757.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/0e/41/output.1.241d943c-bd6c-46e5-9151-d0f3c942f3c1_33757.pool.root:checks 4652033473 21/2/2020 19:20 nid01832 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01832 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4646620361 15/2/2020 3:41 nid01788 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01788 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4631693700 1/2/2020 20:33 nid01762 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01762 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 20:39:48 sup:::2 worker already finished at 2020-02-01 21:00:33 with None CSCS-LCG2-HPC_UCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4655152860 26/2/2020 23:26 nid01762 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01762 ddm:::200 expected output 9979d87f-8e3e-41c7-b038-704ccc737bd3_19049.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/7c/8b/Hits.hc_20163385.CSCS-LCG2-HPC_UCORE.9979d87f-8e3e-41c7-b038-704ccc7 4646663729 15/2/2020 5:29 nid01911 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01911 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4631323939 1/2/2020 19:40 nid01848 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01848 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 19:59:30 sup:::2 worker already finished at 2020-02-01 20:16:41 with None 4631915949 2/2/2020 2:27 nid01840 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01840 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 02:46:28 sup:::2 worker already finished at 2020-02-02 03:03:44 with None 4653329583 24/2/2020 1:10 nid01848 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01848 ddm:::200 expected output 1d8c4b47-20cb-42c3-8faa-c62fcf3d33a6_59633.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/85/68/1d8c4b47-20cb-42c3-8faa-c62fcf3d33a6_59633.1.job.log.tgz:checksumtyp 4640893054 9/2/2020 12:44 nid01718 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01718 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4635451763 4/2/2020 18:30 nid01845 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01845 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4633252678 2/2/2020 23:09 nid01706 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01706 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 23:28:35 sup:::2 worker already finished at 2020-02-02 23:40:36 with None 4655900479 27/2/2020 9:27 nid01846 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01846 ddm:::200 expected output c86c8eb9-6147-4bef-8388-2cccbccc7e96_13848.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/ea/f5/Hits.hc_20163526.CSCS-LCG2-HPC_UCORE.c86c8eb9-6147-4bef-8388-2cccbcc 4655156766 26/2/2020 23:34 nid01706 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01706 ddm:::200 expected output 4cc81607-506a-4d67-88cd-826030242e92_8283.1.job.log.tgz is missing in pilot XML sup:::9000 Diag from worker : Failed in data staging: Failed to prepare destination srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/atlas/atlasdatadisk/rucio/hc_test/37/b2/output.1.4cc81607-506a-4d67-88cd-826030242e92_8283.pool.root:checksu 4631702508 1/2/2020 23:03 nid01840 hc_test:mc15_13TeV.361106.PowhegP modificationHost: nid01840 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 23:31:59 sup:::2 worker already finished at 2020-02-01 23:52:34 with None ====== CYFRONET-LCG2 ANALY_CYF ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- CYFRONET-LCG2 ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4637782907 6/2/2020 14:40 n1021-amd.zeu hc_test:mc15_13TeV.361106.PowhegP modificationHost: n1021-amd.zeus pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from CYFRONET-LCG2_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4637009788 5/2/2020 17:41 n1018-amd.zeu hc_test:mc15_13TeV.361106.PowhegP modificationHost: n1018-amd.zeus taskBuffer:::300 The worker was finished while the job was running : None 4638176494 7/2/2020 3:32 n1019-amd.zeu hc_test:mc15_13TeV.361106.PowhegP modificationHost: n1019-amd.zeus pilot:::1137 Failed to stage-out file: hc_test:output.1.8ae1afa2-0286-4d95-9b6f-50ab4293aa51_41818.pool.root to CYFRONET-LCG2_DATADISK, Cannot connect to the Rucio server.',)]:failed to transfer files using copytools=[u'rucio'] ====== DESY-HH ANALY_DESY-HH ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4657621394 28/2/2020 19:43 slot1_13@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_13@batch0220.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.52a99329-7422-4b39-a539-41628bd078c6_33094.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644945645 13/2/2020 6:46 slot1_5@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_5@batch0319.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:hc_20160924.ANALY_DESY-HH.9c9d8775-f77f-4bb4-bc7c-3507d55ca6f3_35469.hist-out.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644961804 13/2/2020 7:23 slot1_6@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_6@batch0612.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:18e34652-ce3f-4da5-a2eb-97c967a11c04_79249.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644975864 13/2/2020 7:57 slot1_13@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_13@batch0675.desy.de 4644990786 13/2/2020 8:17 slot1_3@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_3@batch0611.desy.de 4645002706 13/2/2020 8:37 slot1_3@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_3@batch0619.desy.de 4645016004 13/2/2020 9:02 slot1_13@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_13@batch0526.desy.de 4645025160 13/2/2020 9:21 slot1_16@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_16@batch1019.desy.de 4645272297 13/2/2020 16:50 slot1_15@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_15@batch0916.desy.de 4645304621 13/2/2020 17:43 slot1_12@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_12@batch0659.desy.de taskBuffer:::300 The worker was finished while the job was running : None 4645590733 14/2/2020 1:24 slot1_6@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_6@batch0105.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:075ec049-cd87-4c11-8201-9306bfb870c6_5153.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 075ec049-cd87-4c11-8201-9306bfb870c6_5153.1.job.log.tgz is missing in pilot XML 4645628944 14/2/2020 2:35 slot1_18@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_18@batch0421.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:39ee0916-3a11-4bc0-aa9e-22cec0c99e41_93584.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 39ee0916-3a11-4bc0-aa9e-22cec0c99e41_93584.1.job.log.tgz is missing in pilot XML 4645647393 14/2/2020 3:26 slot1_14@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_14@batch0507.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:dec25288-6f65-4718-af76-9af00cde9c53_95244.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output dec25288-6f65-4718-af76-9af00cde9c53_95244.1.job.log.tgz is missing in pilot XML 4645661351 14/2/2020 4:06 slot1_12@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_12@batch0233.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.5eaa332c-6343-43b2-93ea-e481f4c427d7_49456.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644012658 12/2/2020 11:47 slot1_24@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_24@batch0908.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.88f16c02-bda5-4ecb-a07b-474cbb196740_43569.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644145497 12/2/2020 14:28 slot1_17@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_17@batch0602.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:57452e7a-dd3a-48d5-9015-cab76f8cd33d_29253.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 57452e7a-dd3a-48d5-9015-cab76f8cd33d_29253.1.job.log.tgz is missing in pilot XML 4644195520 12/2/2020 15:21 slot1_25@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_25@batch0948.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.e58358b5-9702-428b-913e-d023ff77a298_61059.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644255338 12/2/2020 15:58 slot1_4@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_4@batch0650.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:bb877ad3-1ba0-41b7-8153-9838cc4fe533_88568.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output bb877ad3-1ba0-41b7-8153-9838cc4fe533_88568.1.job.log.tgz is missing in pilot XML 4644305359 12/2/2020 16:39 slot1_25@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_25@batch0919.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.860768a9-8f10-49fd-83a5-61d74bd34f25_21631.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644354171 12/2/2020 17:24 slot1_18@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_18@batch0655.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.3aa61e08-df04-43be-9764-947eb46ddc57_99354.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644405353 12/2/2020 18:24 slot1_8@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_8@batch0658.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.b03d088b-4499-4677-b257-ce110a7642d7_21738.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644464988 12/2/2020 19:13 slot1_12@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_12@batch0926.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.0ad06642-1304-4c07-a9a3-9796092fdf65_93903.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644531312 12/2/2020 20:14 slot1_16@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_16@batch0543.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:821b72eb-b4cd-4167-bdac-74382e330b71_46323.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 821b72eb-b4cd-4167-bdac-74382e330b71_46323.1.job.log.tgz is missing in pilot XML 4644634559 12/2/2020 21:46 slot1_19@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_19@batch0664.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.a1168a0b-d56c-4631-af13-bbf07dccb816_16776.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644673362 12/2/2020 22:30 slot1_9@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_9@batch0527.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.348647b4-6193-40a3-accf-10018b02d0d8_2064.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644745259 13/2/2020 0:09 slot1_15@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_15@batch0664.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.0f4f26ad-29c4-42db-8d6d-dde06da25746_42248.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644777820 13/2/2020 0:57 slot1_4@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_4@batch0939.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.94784891-06c1-4727-8177-ef520eb47b84_3273.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644816596 13/2/2020 1:58 slot1_20@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_20@batch0917.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:f9b5cedc-7c6c-4686-9078-14907dda2e6d_61895.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output f9b5cedc-7c6c-4686-9078-14907dda2e6d_61895.1.job.log.tgz is missing in pilot XML 4644838843 13/2/2020 2:43 slot1_4@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_4@batch0808.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.95d159ea-7c6c-41ee-bb71-030cd5e8bcd6_52787.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644862834 13/2/2020 3:51 slot1_13@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_13@batch0602.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:d6b656af-babc-49e8-8b9d-ed644d26419a_85796.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output d6b656af-babc-49e8-8b9d-ed644d26419a_85796.1.job.log.tgz is missing in pilot XML 4644885121 13/2/2020 4:31 slot1_20@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_20@batch0418.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:e8592593-b7a4-4452-bab8-768eb4735e14_5053.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output e8592593-b7a4-4452-bab8-768eb4735e14_5053.1.job.log.tgz is missing in pilot XML 4644914803 13/2/2020 5:33 slot1_10@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_10@batch0427.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.06afb878-01fe-44ff-90fc-0f19dddd4fb0_47978.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4641619274 10/2/2020 10:23 slot1_15@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_15@batch0121.desy.de 4656112195 27/2/2020 7:06 slot1_6@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_6@batch0931.desy.de pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-HH_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656128302 27/2/2020 7:46 slot1_1@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_1@batch0938.desy.de pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-HH_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656136980 27/2/2020 8:04 slot1_2@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_2@batch0652.desy.de pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-HH_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656148633 27/2/2020 8:27 slot1_14@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_14@batch0226.desy.de pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-HH_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656160522 27/2/2020 8:54 slot1_3@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_3@batch0334.desy.de pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-HH_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656171270 27/2/2020 9:18 slot1_19@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_19@batch0533.desy.de pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-HH_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4645327705 13/2/2020 18:19 slot1_18@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_18@batch0601.desy.de taskBuffer:::300 The worker was finished while the job was running : None 4645571522 14/2/2020 0:43 slot1_8@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_8@batch0609.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:658c09e2-aca2-44c5-9abf-6f27eff8e7de_97780.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 658c09e2-aca2-44c5-9abf-6f27eff8e7de_97780.1.job.log.tgz is missing in pilot XML 4645630060 14/2/2020 2:36 slot1_9@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_9@batch0618.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:fd8f6ac6-339a-4821-9948-09b713dcc468_25155.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output fd8f6ac6-339a-4821-9948-09b713dcc468_25155.1.job.log.tgz is missing in pilot XML 4645647658 14/2/2020 3:31 slot1_12@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_12@batch0530.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.71f8c9e1-95c4-49f0-83ef-a81a140ea55e_39752.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645666366 14/2/2020 4:22 slot1_3@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_3@batch0536.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:e7d6f9fe-b737-42bd-ad7d-ec3b2d840073_96868.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output e7d6f9fe-b737-42bd-ad7d-ec3b2d840073_96868.1.job.log.tgz is missing in pilot XML 4644158923 12/2/2020 14:41 slot1_26@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_26@batch0146.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:565f4438-7be3-46ac-8a81-5ddcc8a14156_76985.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 565f4438-7be3-46ac-8a81-5ddcc8a14156_76985.job.log.tgz is missing in pilot XML 4644277982 12/2/2020 16:15 slot1_5@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_5@batch0632.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.54fbf8df-7e72-45b9-9371-83ff6eaa8b3f_67911.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644319498 12/2/2020 17:17 slot1_11@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_11@batch0926.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.f08f2fad-d308-46f4-becf-4720ef0d3785_90656.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644418782 12/2/2020 18:24 slot1_20@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_20@batch0160.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:a2291c17-a20c-4456-8ca6-89adfe4908db_87658.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output a2291c17-a20c-4456-8ca6-89adfe4908db_87658.1.job.log.tgz is missing in pilot XML 4644457589 12/2/2020 19:11 slot1_28@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_28@batch0207.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.73f71aba-23c6-4e0e-8aad-ee13bafc49d7_34929.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644525999 12/2/2020 20:12 slot1_14@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_14@batch0627.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.2d73c315-7bd9-4830-a136-51c85bc685e0_62665.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644569090 12/2/2020 20:52 slot1_2@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_2@batch0664.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c0d41cf0-cdd3-444b-b720-64916b22e692_38225.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644622220 12/2/2020 21:40 slot1_18@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_18@batch0603.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.f5a37dcb-9dae-4385-9f21-cf3e1dc68d48_80610.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644671365 12/2/2020 22:28 slot1_6@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_6@batch0940.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.e9f2d93d-d26a-4b8d-8ef0-929470049d02_6059.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644703775 12/2/2020 23:18 slot1_6@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_6@batch0641.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.3af8efa4-7736-4934-a9db-c74a648513c1_66876.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644738690 12/2/2020 23:59 slot1_8@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_8@batch0931.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d5b043f4-6135-4b50-a86f-cd8113fe1ff4_72118.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644772004 13/2/2020 0:47 slot1_3@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_3@batch0609.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.8cf4c4c5-ed17-462b-a44e-409ce531d850_17360.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644801615 13/2/2020 1:35 slot1_20@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_20@batch0917.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.db63a6e8-e862-40e4-9131-8bb7f836fde7_78561.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644825038 13/2/2020 2:18 slot1_7@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_7@batch0668.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.74f22ac9-3871-4756-8e38-785e75180aa1_65476.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644845961 13/2/2020 3:07 slot1_6@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_6@batch0629.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1ca1851d-b765-4df6-b724-e95b10ab2792_71508.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644865060 13/2/2020 3:53 slot1_8@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_8@batch0607.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.b35b919d-bf3d-47b6-ac59-9d907adcb64d_42957.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644894274 13/2/2020 4:53 slot1_7@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_7@batch0609.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.4443f6e7-2b21-4615-bff8-30a989f3252e_53655.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644919432 13/2/2020 5:57 slot1_25@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_25@batch0912.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.59ad217c-a00f-4e79-b798-ae47708dee4f_80988.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644940146 13/2/2020 6:36 slot1_12@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_12@batch0657.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.045f6ac7-ac1f-48f2-81a8-bdedea37ab14_1397.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635125433 4/2/2020 14:16 slot1_5@batch data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_5@batch0426.desy.de ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4630723434 1/2/2020 7:07 slot1_20@batc data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_20@batch0216.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.79cec418-1d20-4be2-b9ea-61a5d26e02f3_21461.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656148869 27/2/2020 8:30 slot1_11@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_11@batch0622.desy.de pilot:::1324 Service not available at the moment: user.gangarbt:user.gangarbt.1e982c55-09d9-4922-b895-2ea33fa7c382_50995.job.log.tgz to DESY-HH_SCRATCHDISK, Connection timed out',)]:failed to transfer files using copytools=[u'rucio'] 4656159330 27/2/2020 8:54 slot1_48@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_48@batch0302.desy.de pilot:::1324 Service not available at the moment: user.gangarbt:79467920-a275-4128-9b45-be028dadd513_32681.1.job.log.tgz to DESY-HH_SCRATCHDISK, Connection timed out',)]:failed to transfer files using copytools=[u'rucio'] 4656171047 27/2/2020 9:21 slot1_22@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_22@batch0314.desy.de pilot:::1324 Service not available at the moment: user.gangarbt:797e9fb4-6b91-4fb4-b2fd-8b85859cd0cf_82307.1.job.log.tgz to DESY-HH_SCRATCHDISK, Connection timed out',)]:failed to transfer files using copytools=[u'rucio'] 4655951020 27/2/2020 2:14 slot1_10@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_10@batch0922.desy.de 4655958471 27/2/2020 2:28 slot1_3@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_3@batch0940.desy.de 4655970032 27/2/2020 2:46 slot1_10@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_10@batch0239.desy.de 4655988107 27/2/2020 3:18 slot1_9@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_9@batch0625.desy.de 4655997714 27/2/2020 3:34 slot1_28@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_28@batch0668.desy.de 4656009485 27/2/2020 3:53 slot1_23@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_23@batch0203.desy.de 4656022885 27/2/2020 4:12 slot1_5@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_5@batch0931.desy.de 4656032056 27/2/2020 4:26 slot1_11@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_11@batch0556.desy.de 4656040590 27/2/2020 4:39 slot1_9@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_9@batch0927.desy.de 4656048590 27/2/2020 4:54 slot1_40@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_40@batch0206.desy.de 4656056868 27/2/2020 5:10 slot1_6@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_6@batch0942.desy.de 4656062129 27/2/2020 5:22 slot1_4@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_4@batch0678.desy.de 4656069055 27/2/2020 5:37 slot1_21@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_21@batch0945.desy.de 4656075422 27/2/2020 5:51 slot1_31@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_31@batch0224.desy.de 4656081273 27/2/2020 6:02 slot1_6@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_6@batch0902.desy.de 4656091283 27/2/2020 6:25 slot1_16@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_16@batch0937.desy.de 4651892193 21/2/2020 16:45 slot1_18@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_18@batch0911.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:user.gangarbt.user.gangarbt.hc20162679.tid952.ANALY_DESY-HH.70.9925e26b-8fd2-459b-a9c5-0c996b82de9e_12547.hist-output.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645555588 14/2/2020 0:18 slot1_13@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_13@batch0233.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.6c03408e-905a-4896-ac41-e73e9d06b8f6_31830.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645576896 14/2/2020 0:57 slot1_8@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_8@batch0638.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:8befcb25-9022-4a1f-8e8f-3a4ce0734249_87789.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 8befcb25-9022-4a1f-8e8f-3a4ce0734249_87789.1.job.log.tgz is missing in pilot XML 4645591190 14/2/2020 1:24 slot1_8@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_8@batch0105.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d59745ea-31dc-4f33-be5b-a31ab334efbc_36803.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645621950 14/2/2020 2:13 slot1_11@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_11@batch0406.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1f2ed64b-83ff-4458-bd7b-ce57340741e8_20432.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645635634 14/2/2020 2:47 slot1_14@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_14@batch0119.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:605d407b-c4b2-49fc-80d6-7a914b62c5c8_98630.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 605d407b-c4b2-49fc-80d6-7a914b62c5c8_98630.1.job.log.tgz is missing in pilot XML 4645648166 14/2/2020 3:31 slot1_18@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_18@batch0530.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:43cf700d-d8c2-413a-a238-90b206e21f7b_8532.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 43cf700d-d8c2-413a-a238-90b206e21f7b_8532.1.job.log.tgz is missing in pilot XML 4645662867 14/2/2020 4:07 slot1_13@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_13@batch0656.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.ad47da9d-e02c-4ed7-b549-89ad1a477395_54094.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644549573 12/2/2020 20:27 slot1_12@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_12@batch0623.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:user.gangarbt.user.gangarbt.hc20160833.tid952.ANALY_DESY-HH.61.0523c806-8935-4256-aebc-0d482e2b5dad_34337.hist-output.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644574309 12/2/2020 20:54 slot1_4@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_4@batch0919.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.81339558-5d33-470c-bf63-1de8c277efe9_83563.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644598438 12/2/2020 21:19 slot1_18@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_18@batch0677.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d7afa888-619c-499f-a439-b6615c5800a7_5707.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644633566 12/2/2020 21:48 slot1_19@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_19@batch0210.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.cf5d7ab7-cd02-407a-8135-2405f741eb50_37046.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644664688 12/2/2020 22:27 slot1_23@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_23@batch0668.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:9dda1956-ae43-4e58-9b4a-44fd7e07e80a_57026.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 9dda1956-ae43-4e58-9b4a-44fd7e07e80a_57026.1.job.log.tgz is missing in pilot XML 4644691671 12/2/2020 22:54 slot1_11@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_11@batch0208.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.11d6a351-6b7a-4bb8-b9ce-5f29cca43f1a_44637.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644734403 12/2/2020 23:56 slot1_15@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_15@batch0140.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.71cb7af1-8ad1-408f-8f05-8c1d826397bf_18688.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644793541 13/2/2020 1:23 slot1_24@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_24@batch0934.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.7ba16a90-3a7c-4e71-9c42-9bc82ba4b206_50198.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644813251 13/2/2020 1:54 slot1_21@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_21@batch0672.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.690cfe3d-39d4-4d4a-9ab6-b03b5ba2c1e1_17716.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644844584 13/2/2020 3:03 slot1_26@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_26@batch0233.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.554239e1-cab9-4d3a-bec1-8029a578b830_27969.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644860739 13/2/2020 3:41 slot1_8@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_8@batch0429.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:025ba0e8-95b9-4c26-a14c-78d3ebcd3215_94587.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 025ba0e8-95b9-4c26-a14c-78d3ebcd3215_94587.1.job.log.tgz is missing in pilot XML 4644902115 13/2/2020 5:09 slot1_18@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_18@batch0667.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.332d0d24-7722-42fa-8f4f-a676b6fa906b_56237.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644917449 13/2/2020 5:42 slot1_5@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_5@batch0536.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.e75d3e98-130d-4156-8172-cb38b18471c6_26797.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644932760 13/2/2020 6:17 slot1_24@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_24@batch0922.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.fef1d5dc-eda5-4490-855c-c2f807590f13_36311.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644967126 13/2/2020 7:29 slot1_3@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_3@batch0641.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:aed6589e-99da-442c-aaea-5a2ce57bc07f_28902.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644983838 13/2/2020 8:09 slot1_3@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_3@batch0112.desy.de 4645008094 13/2/2020 8:49 slot1_5@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_5@batch0948.desy.de 4645021340 13/2/2020 9:09 slot1_23@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_23@batch0946.desy.de 4644030923 12/2/2020 12:26 slot1_23@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_23@batch1014.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.139dc924-7948-4e2f-a20c-82730dc7e854_74115.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644055485 12/2/2020 13:55 slot1_10@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_10@batch0814.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.31037cd0-24ef-4950-aaca-f4b0b7e6a8a7_97953.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644183344 12/2/2020 15:01 slot1_6@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_6@batch0213.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:094af81b-3e81-4f83-943a-7eaed3814cdd_56626.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 094af81b-3e81-4f83-943a-7eaed3814cdd_56626.1.job.log.tgz is missing in pilot XML 4644248845 12/2/2020 15:57 slot1_23@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_23@batch0939.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.27ccb5d3-594c-4f6b-88ab-eeb69c867d1e_9271.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644307179 12/2/2020 16:41 slot1_7@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_7@batch0946.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:f4f7daff-9a43-4a5e-b0a6-858e246cabd3_45210.1.job.log.tgz to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output f4f7daff-9a43-4a5e-b0a6-858e246cabd3_45210.1.job.log.tgz is missing in pilot XML 4644345776 12/2/2020 17:18 slot1_4@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_4@batch0947.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.f6b4e245-246d-4165-b8f2-295d5c661d8f_42166.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644382346 12/2/2020 17:53 slot1_8@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_8@batch0678.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.fccf9292-6976-4a8a-9e5b-e3c316d1ce79_27881.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644443455 12/2/2020 18:49 slot1_7@batch mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_7@batch0621.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.4a8805dd-e9b9-459a-8ad1-20ccb5db60d8_59988.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644478705 12/2/2020 19:28 slot1_23@batc mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_23@batch0646.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.503fbb86-051e-4de3-a954-37530ce168f8_3828.pool.root to DESY-HH_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] DESY-HH_UCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4658503014 29/2/2020 21:33 slot1_6@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_6@batch0642.desy.de pilot:::1152 File transfer timed out during stage-out: hc_test:c18c5de7-b27b-4112-b7f6-2f3a5eb1bf4c_15390.1.job.log.tgz to DESY-HH_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output c18c5de7-b27b-4112-b7f6-2f3a5eb1bf4c_15390.1.job.log.tgz is missing in pilot XML 4645275345 13/2/2020 17:00 slot1_22@batc hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_22@batch0139.desy.de taskBuffer:::300 The worker was finished while the job was running : None 4640306312 8/2/2020 19:26 slot1_4@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@batch0149.desy.de pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 6 overflow 0 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4637782956 6/2/2020 13:53 slot1_6@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_6@batch0656.desy.de taskBuffer:::300 The worker was failed while the job was running : LRMS error: (271) job killed: vmem sup:::9000 Diag from worker : LRMS error: (271) job killed: vmem 4633893025 3/2/2020 11:07 slot1_5@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@batch0930.desy.de pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20158927.DESY-HH_UCORE.288bc670-b5f3-427d-b5c2-4e2b9c3d4262_47435.HITS.pool.root to DESY-HH_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4650272789 19/2/2020 18:27 slot1_19@batc hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_19@batch0211.desy.de taskBuffer:::300 The worker was failed while the job was running : LRMS error: (271) job killed: vmem sup:::9000 Diag from worker : LRMS error: (271) job killed: vmem 4646517103 14/2/2020 23:54 slot1_2@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@batch0673.desy.de taskBuffer:::300 The worker was failed while the job was running : LRMS error: (271) job killed: vmem sup:::9000 Diag from worker : LRMS error: (271) job killed: vmem 4642752905 11/2/2020 2:14 slot1_26@batc hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_26@batch0140.desy.de taskBuffer:::300 The worker was failed while the job was running : LRMS error: (271) job killed: vmem sup:::9000 Diag from worker : LRMS error: (271) job killed: vmem 4630669692 1/2/2020 6:25 slot1_11@batc hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_11@batch0506.desy.de pilot:::1137 Failed to stage-out file: hc_test:Hits.hc_20158461.DESY-HH_UCORE.75cf35f9-b839-48fd-b70f-d77e0cfbb215_20397.HITS.pool.root to DESY-HH_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4630739822 1/2/2020 7:26 slot1_13@batc hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_13@batch0906.desy.de pilot:::1137 Failed to stage-out file: hc_test:output.1.9c924028-c2b1-45a0-be86-02c467822307_42071.pool.root to DESY-HH_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645222392 13/2/2020 16:36 slot1_9@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_9@batch0442.desy.de taskBuffer:::300 The worker was finished while the job was running : None 4644558378 12/2/2020 20:36 slot1_3@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_3@batch0914.desy.de pilot:::1152 File transfer timed out during stage-out: hc_test:75694607-ed5c-4009-b9d5-9333db332bad_74564.1.job.log.tgz to DESY-HH_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 75694607-ed5c-4009-b9d5-9333db332bad_74564.1.job.log.tgz is missing in pilot XML 4636932764 5/2/2020 16:00 slot1_7@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_7@batch0628.desy.de pilot:::1152 File transfer timed out during stage-out: hc_test:da1f2285-8b55-4663-9ce0-92ecea7842c1_1079.1.job.log.tgz to DESY-HH_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output da1f2285-8b55-4663-9ce0-92ecea7842c1_1079.1.job.log.tgz is missing in pilot XML 4635404451 4/2/2020 14:53 slot1_9@batch hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_9@batch0629.desy.de ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' ====== DESY-ZN ANALY_DESY-ZN ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4645304629 13/2/2020 17:53 jux8b.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux8b.zeuthen.desy.de taskBuffer:::300 The worker was finished while the job was running : None 4637696093 6/2/2020 11:04 juxdc.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxdc.zeuthen.desy.de 4637708481 6/2/2020 11:46 juxac.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxac.zeuthen.desy.de 4637735902 6/2/2020 12:16 juxgf.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxgf.zeuthen.desy.de 4637751936 6/2/2020 12:37 juxf0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf0.zeuthen.desy.de 4637761085 6/2/2020 13:08 juxh6.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh6.zeuthen.desy.de 4637774985 6/2/2020 13:29 juxe0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxe0.zeuthen.desy.de 4637785271 6/2/2020 13:59 juxgc.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxgc.zeuthen.desy.de 4637795605 6/2/2020 14:32 juxf3.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf3.zeuthen.desy.de 4637810378 6/2/2020 14:56 juxg9.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxg9.zeuthen.desy.de 4637828047 6/2/2020 15:26 lux08.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux08.zeuthen.desy.de 4637843016 6/2/2020 15:56 juxf1.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf1.zeuthen.desy.de 4637863093 6/2/2020 16:23 lux01.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux01.zeuthen.desy.de 4637876901 6/2/2020 16:47 juxhe.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxhe.zeuthen.desy.de 4637889979 6/2/2020 17:21 juxh0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh0.zeuthen.desy.de 4637905570 6/2/2020 17:41 juxh2.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh2.zeuthen.desy.de 4637924116 6/2/2020 18:14 juxf7.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf7.zeuthen.desy.de 4637946577 6/2/2020 18:45 juxh8.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh8.zeuthen.desy.de 4637970517 6/2/2020 19:39 jux8f.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux8f.zeuthen.desy.de 4637995013 6/2/2020 20:12 juxfa.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxfa.zeuthen.desy.de 4638010638 6/2/2020 20:51 juxcf.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxcf.zeuthen.desy.de 4638020235 6/2/2020 21:24 juxac.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxac.zeuthen.desy.de 4638032837 6/2/2020 22:12 juxgb.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxgb.zeuthen.desy.de 4638051883 6/2/2020 22:46 jux8b.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux8b.zeuthen.desy.de 4638069679 6/2/2020 23:33 lux03.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux03.zeuthen.desy.de 4638085510 7/2/2020 0:09 juxc3.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxc3.zeuthen.desy.de 4638123026 7/2/2020 1:02 jux92.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux92.zeuthen.desy.de 4638142189 7/2/2020 1:55 juxb8.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxb8.zeuthen.desy.de 4638165234 7/2/2020 2:49 juxa6.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxa6.zeuthen.desy.de 4638199143 7/2/2020 3:49 juxh4.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh4.zeuthen.desy.de 4638228782 7/2/2020 4:31 juxb6.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxb6.zeuthen.desy.de 4638259024 7/2/2020 5:25 juxfd.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxfd.zeuthen.desy.de 4638277344 7/2/2020 6:01 lux06.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux06.zeuthen.desy.de 4638295610 7/2/2020 6:37 juxb0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxb0.zeuthen.desy.de 4637609107 6/2/2020 8:43 juxf1.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf1.zeuthen.desy.de pilot:::1137 Failed to stage-out file: user.gangarbt:5a61db5c-73a3-47c4-aa40-27f3ec8e51ff_2654.1.job.log.tgz to DESY-ZN_SCRATCHDISK, DESY-ZN_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4637623092 6/2/2020 9:10 juxf7.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf7.zeuthen.desy.de pilot:::1137 Failed to stage-out file: user.gangarbt:473ec53f-2d84-407b-91a9-bc2c3e88354c_97912.1.job.log.tgz to DESY-ZN_SCRATCHDISK, DESY-ZN_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4637639411 6/2/2020 9:55 juxgc.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxgc.zeuthen.desy.de ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4637669089 6/2/2020 10:31 juxed.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxed.zeuthen.desy.de 4635217299 4/2/2020 12:08 juxd4.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxd4.zeuthen.desy.de ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635366837 4/2/2020 14:01 juxg7.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxg7.zeuthen.desy.de ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4636545202 5/2/2020 9:54 jux90.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux90.zeuthen.desy.de taskBuffer:::300 The worker was finished while the job was running : None 4647361805 16/2/2020 8:38 jux9b.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux9b.zeuthen.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:6a696e6a-a9a5-443f-89db-a0ed89e96344_29881.1.job.log.tgz to DESY-ZN_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 6a696e6a-a9a5-443f-89db-a0ed89e96344_29881.1.job.log.tgz is missing in pilot XML 4637713024 6/2/2020 11:46 juxb4.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxb4.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637739075 6/2/2020 12:19 juxg8.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxg8.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637755969 6/2/2020 12:48 lux02.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: lux02.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637770316 6/2/2020 13:28 juxe0.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxe0.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637785654 6/2/2020 13:58 juxgc.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxgc.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637795327 6/2/2020 14:24 juxf3.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxf3.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637807646 6/2/2020 14:51 juxe5.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxe5.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637827201 6/2/2020 15:26 lux08.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: lux08.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637844883 6/2/2020 15:57 juxf1.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxf1.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637865667 6/2/2020 16:35 juxf3.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxf3.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637886706 6/2/2020 17:20 juxh0.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxh0.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637911459 6/2/2020 17:50 jux9c.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux9c.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637936732 6/2/2020 18:30 juxef.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxef.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637957322 6/2/2020 19:03 jux9b.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux9b.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637978690 6/2/2020 19:38 jux8f.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux8f.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637996248 6/2/2020 20:20 juxgd.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxgd.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638013514 6/2/2020 21:00 jux7a.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux7a.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638024734 6/2/2020 21:39 juxed.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxed.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638037351 6/2/2020 22:24 juxg8.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxg8.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638059654 6/2/2020 23:12 juxhd.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxhd.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638079245 7/2/2020 0:03 jux70.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux70.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638118072 7/2/2020 0:51 jux90.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux90.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638138662 7/2/2020 1:54 juxac.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxac.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638156964 7/2/2020 2:34 juxgf.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxgf.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638190078 7/2/2020 3:43 juxa7.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxa7.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638225252 7/2/2020 4:19 juxc4.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxc4.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638248943 7/2/2020 5:13 jux7f.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux7f.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638276973 7/2/2020 6:01 lux06.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: lux06.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4638296613 7/2/2020 6:46 juxh7.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxh7.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637017494 5/2/2020 18:04 juxf5.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxf5.zeuthen.desy.de taskBuffer:::300 The worker was finished while the job was running : None 4637614478 6/2/2020 8:49 juxe7.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxe7.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637634023 6/2/2020 9:46 juxa6.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxa6.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4637667818 6/2/2020 10:31 juxed.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: juxed.zeuthen.desy.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0161._0001.1 from DESY-ZN_DATADISK, None of the requested files have been downloaded.',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/scratc 4635148011 4/2/2020 11:19 jux76.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux76.zeuthen.desy.de pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.ca629e52-fc92-4d79-bc5f-8becc16935dc_95116.pool.root to DESY-ZN_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635204337 4/2/2020 11:59 jux89.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux89.zeuthen.desy.de ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4636536635 5/2/2020 9:54 jux90.zeuthen data18_13TeV:data18_13TeV.0034926 modificationHost: jux90.zeuthen.desy.de taskBuffer:::300 The worker was finished while the job was starting : None 4637803930 6/2/2020 14:45 juxf7.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf7.zeuthen.desy.de 4637813400 6/2/2020 15:08 lux08.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux08.zeuthen.desy.de 4637831090 6/2/2020 15:35 lux05.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux05.zeuthen.desy.de 4637845112 6/2/2020 15:57 juxf1.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf1.zeuthen.desy.de 4637861709 6/2/2020 16:20 juxc6.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxc6.zeuthen.desy.de 4637876991 6/2/2020 16:50 juxf2.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxf2.zeuthen.desy.de 4637891394 6/2/2020 17:20 juxh0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh0.zeuthen.desy.de 4637910367 6/2/2020 17:50 jux9c.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux9c.zeuthen.desy.de 4637928458 6/2/2020 18:17 juxb2.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxb2.zeuthen.desy.de 4637945288 6/2/2020 18:44 juxh8.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh8.zeuthen.desy.de 4637959370 6/2/2020 19:18 juxc1.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxc1.zeuthen.desy.de 4637982042 6/2/2020 19:48 lux01.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux01.zeuthen.desy.de 4637996845 6/2/2020 20:21 juxgd.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxgd.zeuthen.desy.de 4638010135 6/2/2020 20:51 juxcf.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxcf.zeuthen.desy.de 4638019180 6/2/2020 21:24 juxac.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxac.zeuthen.desy.de 4638029784 6/2/2020 21:57 juxg3.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxg3.zeuthen.desy.de 4638039512 6/2/2020 22:24 juxg8.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxg8.zeuthen.desy.de 4638053783 6/2/2020 22:57 juxbd.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxbd.zeuthen.desy.de 4638066311 6/2/2020 23:33 lux03.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux03.zeuthen.desy.de 4638081387 7/2/2020 0:06 juxbd.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxbd.zeuthen.desy.de 4638111534 7/2/2020 0:36 jux8c.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux8c.zeuthen.desy.de 4638126125 7/2/2020 1:08 juxh1.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh1.zeuthen.desy.de 4638139402 7/2/2020 1:55 juxb8.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxb8.zeuthen.desy.de 4638155711 7/2/2020 2:34 juxgf.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxgf.zeuthen.desy.de 4638177564 7/2/2020 3:02 jux91.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux91.zeuthen.desy.de 4638203538 7/2/2020 3:49 juxh4.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh4.zeuthen.desy.de 4638225781 7/2/2020 4:25 juxc8.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxc8.zeuthen.desy.de 4638247482 7/2/2020 4:57 juxh1.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh1.zeuthen.desy.de 4638261191 7/2/2020 5:25 juxfd.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxfd.zeuthen.desy.de 4638276066 7/2/2020 6:01 lux06.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux06.zeuthen.desy.de 4638294898 7/2/2020 6:34 juxd0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxd0.zeuthen.desy.de 4637608582 6/2/2020 8:37 juxfb.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxfb.zeuthen.desy.de pilot:::1137 Failed to stage-out file: user.gangarbt:a85186c9-b704-4af9-a35c-42a1dec8e7db_99422.1.job.log.tgz to DESY-ZN_SCRATCHDISK, DESY-ZN_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4637618470 6/2/2020 8:59 juxa0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxa0.zeuthen.desy.de pilot:::1137 Failed to stage-out file: user.gangarbt:8024c6cd-6837-4e6b-8d20-b9294d98f179_20601.1.job.log.tgz to DESY-ZN_SCRATCHDISK, DESY-ZN_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4637635207 6/2/2020 9:46 juxa3.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxa3.zeuthen.desy.de pilot:::1137 Failed to stage-out file: user.gangarbt:8f085eec-356e-4d51-b64f-ff42ff744c05_25998.1.job.log.tgz to DESY-ZN_SCRATCHDISK, DESY-ZN_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4637663001 6/2/2020 10:19 lux02.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lux02.zeuthen.desy.de 4637692916 6/2/2020 11:04 juxdc.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxdc.zeuthen.desy.de 4637717112 6/2/2020 11:49 juxge.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxge.zeuthen.desy.de 4637742909 6/2/2020 12:25 juxh2.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh2.zeuthen.desy.de 4637760676 6/2/2020 13:08 juxh6.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxh6.zeuthen.desy.de 4637781853 6/2/2020 13:48 juxe0.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: juxe0.zeuthen.desy.de 4636542030 5/2/2020 9:54 jux90.zeuthen mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: jux90.zeuthen.desy.de jobDispatcher:::100 lost heartbeat : 2020-02-05 09:54:16 DESY-ZN_UCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4644774731 13/2/2020 0:48 jux87.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux87.zeuthen.desy.de pilot:::1322 Failed to get replicas from Rucio: HTTPSConnectionPool(host='voatlasrucio-server-prod.cern.ch', port=443): Read timed out. (read timeout=600)",)]:failed to transfer files using copytools=[u'rucio'] 4642860408 11/2/2020 4:03 juxhc.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxhc.zeuthen.desy.de pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.aef4dbde-72c6-49e7-9c65-61d4e84ff990_25961.pool.root to DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637783021 6/2/2020 13:55 jux90.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux90.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637791638 6/2/2020 14:19 juxg4.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg4.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637800789 6/2/2020 14:43 juxe6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637811724 6/2/2020 14:58 juxh6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637819032 6/2/2020 15:16 lux08.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux08.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637832883 6/2/2020 15:34 juxee.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxee.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637842165 6/2/2020 15:52 lux05.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux05.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637855293 6/2/2020 16:13 juxf2.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxf2.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637870998 6/2/2020 16:35 jux99.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux99.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637879450 6/2/2020 16:52 juxh9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637886797 6/2/2020 17:07 juxf2.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxf2.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637894920 6/2/2020 17:24 juxfb.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxfb.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637905903 6/2/2020 17:44 juxee.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxee.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637919524 6/2/2020 18:02 jux77.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux77.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637930603 6/2/2020 18:23 juxh2.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh2.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637946144 6/2/2020 18:44 juxh8.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh8.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637957451 6/2/2020 19:05 juxh7.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh7.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637970146 6/2/2020 19:26 juxc6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637984333 6/2/2020 19:53 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637999157 6/2/2020 20:21 juxgd.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxgd.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638009775 6/2/2020 20:44 juxg2.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg2.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638014614 6/2/2020 21:02 juxgc.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxgc.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638021424 6/2/2020 21:32 juxc9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638030497 6/2/2020 21:57 jux77.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux77.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638041000 6/2/2020 22:26 juxa4.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa4.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638051906 6/2/2020 22:48 juxcd.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxcd.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638062089 6/2/2020 23:21 jux85.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux85.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638075419 6/2/2020 23:54 juxb3.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxb3.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638102204 7/2/2020 0:24 jux80.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux80.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638119074 7/2/2020 0:51 jux90.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux90.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638127903 7/2/2020 1:13 jux92.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux92.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638138862 7/2/2020 1:39 jux74.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux74.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638145579 7/2/2020 2:04 juxa7.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa7.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638157492 7/2/2020 2:33 juxgf.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxgf.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638174510 7/2/2020 3:03 juxa7.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa7.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638203005 7/2/2020 3:42 juxa7.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa7.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638222141 7/2/2020 4:08 lux06.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux06.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638237211 7/2/2020 4:36 juxe6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638252076 7/2/2020 5:08 juxh4.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh4.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638268554 7/2/2020 5:38 juxg0.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg0.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638279479 7/2/2020 6:08 juxe0.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe0.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638295937 7/2/2020 6:40 jux98.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux98.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638307434 7/2/2020 7:11 juxe8.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe8.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637613167 6/2/2020 8:54 juxg1.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg1.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637625366 6/2/2020 9:15 juxb4.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxb4.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637644361 6/2/2020 9:42 juxdb.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxdb.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637655178 6/2/2020 10:01 juxe3.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe3.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637672929 6/2/2020 10:33 juxed.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxed.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637692159 6/2/2020 10:57 juxh3.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh3.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637709798 6/2/2020 11:25 juxe0.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe0.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637722773 6/2/2020 11:52 juxg4.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg4.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637741028 6/2/2020 12:19 lux02.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux02.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637754005 6/2/2020 12:46 lux02.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux02.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637763633 6/2/2020 13:07 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647146414 15/2/2020 22:48 jux8d.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux8d.zeuthen.desy.de pilot:::1137 Failed to stage-out file: hc_test:output.1.318cf14d-2d5f-4f4d-b4d4-4d3cd1cbe503_87082.pool.root to DESY-ZN_DATADISK, no error information passed (http status code: 503 ('service_unavailable', 'unavailable'))",)]:failed to transfer files using copytools=[u'rucio'] 4637608195 6/2/2020 8:42 juxc6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637618264 6/2/2020 9:03 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637694856 6/2/2020 11:03 juxdc.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxdc.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637762492 6/2/2020 13:04 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637813859 6/2/2020 15:13 lux08.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux08.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637885702 6/2/2020 17:04 juxa7.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa7.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637958706 6/2/2020 19:08 juxh5.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh5.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638017797 6/2/2020 21:15 lux05.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux05.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638061632 6/2/2020 23:15 juxda.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxda.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638129589 7/2/2020 1:27 jux70.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux70.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638188902 7/2/2020 3:24 juxh4.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh4.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638260878 7/2/2020 5:24 juxg6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000037.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658625864 1/3/2020 1:14 jux83.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux83.zeuthen.desy.de pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4651118974 20/2/2020 22:02 lux05.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux05.zeuthen.desy.de pilot:::1152 File transfer timed out during stage-out: hc_test:bb6f7d81-6726-4155-b5c8-c1758ce8ec0b_71958.1.job.log.tgz to DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output bb6f7d81-6726-4155-b5c8-c1758ce8ec0b_71958.1.job.log.tgz is missing in pilot XML 4648009351 17/2/2020 11:13 jux7c.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux7c.zeuthen.desy.de pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.eb765b96-e242-4be3-a7e7-0d5652131ee1_50049.pool.root to DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638298546 7/2/2020 6:50 jux8b.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux8b.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000021.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638303753 7/2/2020 7:07 juxa9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000021.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637606300 6/2/2020 8:42 juxc6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637618256 6/2/2020 9:03 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637630084 6/2/2020 9:19 juxgd.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxgd.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637638373 6/2/2020 9:34 jux7e.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux7e.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637650002 6/2/2020 9:51 juxf1.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxf1.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637658244 6/2/2020 10:06 juxf5.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxf5.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637668548 6/2/2020 10:25 jux8e.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux8e.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637681643 6/2/2020 10:43 juxf0.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxf0.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637693254 6/2/2020 11:03 juxdc.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxdc.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637704082 6/2/2020 11:22 juxe0.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe0.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637719049 6/2/2020 11:39 juxge.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxge.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637730286 6/2/2020 12:04 juxd1.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxd1.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637741462 6/2/2020 12:22 juxg0.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg0.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637754244 6/2/2020 12:46 lux02.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux02.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637763826 6/2/2020 13:07 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637773581 6/2/2020 13:28 juxe0.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe0.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637782044 6/2/2020 13:49 lux05.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux05.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637789040 6/2/2020 14:08 lux03.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux03.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637797618 6/2/2020 14:31 juxf3.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxf3.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637807499 6/2/2020 14:52 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637818985 6/2/2020 15:13 lux08.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux08.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637831978 6/2/2020 15:31 juxd6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxd6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637843960 6/2/2020 15:55 juxh2.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh2.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637860370 6/2/2020 16:22 jux70.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux70.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637875318 6/2/2020 16:43 juxg9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637883518 6/2/2020 17:04 juxa7.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa7.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637897374 6/2/2020 17:31 juxg5.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxg5.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637914831 6/2/2020 17:56 juxe3.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe3.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637927972 6/2/2020 18:17 jux87.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux87.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637942295 6/2/2020 18:39 juxd5.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxd5.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637956252 6/2/2020 19:00 juxaa.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxaa.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637977244 6/2/2020 19:35 jux90.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux90.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637989874 6/2/2020 19:58 juxc5.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc5.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638001266 6/2/2020 20:24 juxd4.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxd4.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638011079 6/2/2020 20:48 jux70.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux70.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638022257 6/2/2020 21:32 juxc9.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc9.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638030983 6/2/2020 21:59 lux08.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: lux08.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638041434 6/2/2020 22:29 jux92.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux92.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638056212 6/2/2020 22:56 juxba.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxba.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638068266 6/2/2020 23:25 juxbf.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxbf.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638079490 7/2/2020 0:00 juxde.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxde.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638108822 7/2/2020 0:30 jux8d.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux8d.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638125451 7/2/2020 1:06 juxh1.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxh1.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638138957 7/2/2020 1:40 juxc3.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc3.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638147452 7/2/2020 2:18 juxac.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxac.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638167738 7/2/2020 2:45 juxa6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa6.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638202530 7/2/2020 3:39 juxa7.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxa7.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638224016 7/2/2020 4:21 jux84.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux84.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638255483 7/2/2020 5:10 juxc5.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxc5.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638272573 7/2/2020 5:50 jux73.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: jux73.zeuthen.desy.de pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000035.pool.root.1 from DESY-ZN_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4636318368 5/2/2020 3:46 juxe6.zeuthen hc_test:mc15_13TeV.361106.PowhegP modificationHost: juxe6.zeuthen.desy.de pilot:::1322 Failed to get replicas from Rucio: HTTPSConnectionPool(host='voatlasrucio-server-prod.cern.ch', port=443): Read timed out. (read timeout=600)",)]:failed to transfer files using copytools=[u'rucio'] ====== FMPhI-UNIBA ANALY_FMPhI-UNIBA ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4644896977 13/2/2020 4:51 lcg501.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg501.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4641463881 10/2/2020 6:33 lcg212.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg212.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4642375002 10/2/2020 21:04 lcg411.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg411.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4635444044 4/2/2020 15:10 lcg504.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg504.dnp.fmph.uniba.sk ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4634116407 3/2/2020 16:22 lcg708.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg708.dnp.fmph.uniba.sk jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-03 16:24:53 sup:::2 worker already finished at 2020-02-03 16:32:36 with None 4634424143 4/2/2020 9:20 lcg722.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg722.dnp.fmph.uniba.sk taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4657896470 29/2/2020 2:29 lcg413.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg413.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4656576567 27/2/2020 18:35 lcg504.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg504.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4648530971 18/2/2020 2:41 lcg501.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg501.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4647633416 16/2/2020 18:14 lcg708.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg708.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4647767768 17/2/2020 0:24 lcg708.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg708.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4645290953 13/2/2020 17:12 lcg412.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg412.dnp.fmph.uniba.sk 4644703909 12/2/2020 23:28 lcg420.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg420.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4643544387 11/2/2020 21:21 lcg216.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg216.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4641284206 10/2/2020 0:19 lcg707.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg707.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4641604496 10/2/2020 10:32 lcg711.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg711.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4640684028 9/2/2020 8:11 lcg721.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg721.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4639213303 8/2/2020 0:27 lcg451.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg451.dnp.fmph.uniba.sk pilot:::1098 Not enough local space: [NoLocalSpace(('not enough local space for staging input files and run the job (need 4068236953 B, but only have 2414870528 B)',), {})]:failed to transfer files using copytools=[u'rucio'] 4638049974 6/2/2020 23:57 lcg451.dnp.fm data18_13TeV:data18_13TeV.0034926 modificationHost: lcg451.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4655951216 27/2/2020 2:26 lcg501.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg501.dnp.fmph.uniba.sk 4655966394 27/2/2020 2:39 lcg503.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg503.dnp.fmph.uniba.sk 4647922631 17/2/2020 7:32 lcg414.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg414.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4644905997 13/2/2020 5:09 lcg501.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg501.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4645249038 13/2/2020 16:24 lcg722.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg722.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4645293444 13/2/2020 17:16 lcg458.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg458.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4638035343 6/2/2020 23:24 lcg451.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg451.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4635423340 4/2/2020 14:49 lcg412.dnp.fm mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: lcg412.dnp.fmph.uniba.sk ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' FMPhI-UNIBA ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4644312443 12/2/2020 16:57 lcg412.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg412.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4644959979 13/2/2020 8:00 lcg424.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg424.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4642190943 10/2/2020 18:39 lcg711.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg711.dnp.fmph.uniba.sk pilot:::1137 Failed to stage-out file: hc_test:be2a7fb7-1186-4607-8474-a919a8859a04_81895.1.job.log.tgz to FMPHI-UNIBA_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output be2a7fb7-1186-4607-8474-a919a8859a04_81895.1.job.log.tgz is missing in pilot XML 4640525551 9/2/2020 2:05 lcg209.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg209.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4636919721 5/2/2020 15:44 lcg412.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg412.dnp.fmph.uniba.sk pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output fc44b8ae-53b6-43db-b934-b1b043ebf80f_40962.job.log.tgz is missing in pilot XML 4631935985 2/2/2020 0:40 lcg501.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg501.dnp.fmph.uniba.sk jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 00:51:26 sup:::2 worker already finished at 2020-02-02 01:02:54 with None 4644950948 13/2/2020 6:55 lcg501.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg501.dnp.fmph.uniba.sk exe:::65 Non-zero return code from EVNTtoHITS (33); Logfile error in log.EVNTtoHITS: "PyG4AtlasAlg FATAL Standard std::exception is caught" 4642098712 10/2/2020 17:07 lcg711.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg711.dnp.fmph.uniba.sk exe:::65 Non-zero return code from EVNTtoHITS (33); Logfile error in log.EVNTtoHITS: "PyG4AtlasAlg FATAL Standard std::exception is caught" 4643496022 11/2/2020 18:48 lcg456.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg456.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4643608484 11/2/2020 21:52 lcg216.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg216.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4641871052 10/2/2020 13:30 lcg711.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg711.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault 4639796499 8/2/2020 7:36 lcg451.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg451.dnp.fmph.uniba.sk pilot:::1098 Not enough local space 4637794564 6/2/2020 14:24 lcg451.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg451.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4638030987 6/2/2020 21:59 lcg451.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg451.dnp.fmph.uniba.sk taskBuffer:::300 The worker was finished while the job was running : None 4636925550 5/2/2020 16:56 lcg418.dnp.fm hc_test:mc15_13TeV.361106.PowhegP modificationHost: lcg418.dnp.fmph.uniba.sk pilot:::1328 SIGSEGV: Invalid memory reference or a segmentation fault ====== FZK-LCG2 ANALY_FZK ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4656203879 27/2/2020 10:01 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656243253 27/2/2020 11:20 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4645268616 13/2/2020 16:39 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4635217474 4/2/2020 12:11 slot1_2@c01-1 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_2@c01-115-127 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:574df39f-a6df-40dc-984b-3cdb38bc167d_91335.job.log.tgz to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4631412038 1/2/2020 16:01 slot1_7@c01-0 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_7@c01-005-104 pilot:::1150 Looping job killed by pilot 4631754022 1/2/2020 20:33 slot1_29@c01- mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_29@c01-116-112 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:54f5792c-3f31-4134-961e-e41b63b76c0e_92535.1.job.log.tgz to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655694964 26/2/2020 22:15 aipanda403.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (271) job killed: vmem sup:::9000 Diag from worker : LRMS error: (271) job killed: vmem 4655863320 26/2/2020 23:46 slot1_13@c01- data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_13@c01-004-124 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1 from FZK-LCG2_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4655903434 27/2/2020 1:09 aipanda404.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655963130 27/2/2020 2:33 aipanda403.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656026750 None aipanda404.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656078228 27/2/2020 5:58 aipanda404.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4648242704 17/2/2020 18:08 slot1_13@c01- data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_13@c01-121-112 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.2e6ba548-80f8-404e-9fec-6872c2368cd7_52942.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4648275649 17/2/2020 18:59 slot1_11@c01- data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_11@c01-115-101 taskBuffer:::300 The worker was finished while the job was running : None 4648336626 17/2/2020 20:39 slot1_9@c01-0 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_9@c01-005-155 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.a37fc2da-4096-4473-98e9-eb827ac1c638_22003.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644158918 12/2/2020 14:52 slot1_7@c01-1 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_7@c01-122-116 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:user.gangarbt.74dcbb21-6c80-4ff6-b5f9-38a26f700921_55498.DAOD_EXOT12.test.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644671029 12/2/2020 22:27 slot1_16@c01- data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_16@c01-114-117 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.7373fa93-e535-40b0-80e6-f01ebbcba760_17030.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4643414343 11/2/2020 17:24 slot1_6@c01-1 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_6@c01-121-162 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (271) job killed: wall sup:::9000 Diag from worker : LRMS error: (271) job killed: wall 4643715674 12/2/2020 0:49 slot1_4@c01-0 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_4@c01-009-174 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.6d269bf7-c9f2-4cf5-9aac-ed0308ab018a_2020.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4641347171 10/2/2020 1:57 slot1_5@c01-1 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_5@c01-114-116 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.98cb4b35-04c9-4e5c-b74f-f33045ed6f91_7556.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637890179 6/2/2020 17:21 slot1_16@c01- data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_16@c01-127-176 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ca9409a9-b2a6-4926-a832-abea626d76f5_82452.pool.root to FZK-LCG2_SCRATCHDISK, Cannot connect to the Rucio server.',)]:failed to transfer files using copytools=[u'rucio'] 4638034500 6/2/2020 22:11 slot1_5@c01-0 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_5@c01-007-105 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.9d33c2b1-1aa8-4982-92e9-f2bbd8eb9e48_46334.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637154693 5/2/2020 21:40 slot1_4@c01-0 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_4@c01-009-153 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.7d26d8e7-bb4f-406b-9221-902500211f88_85882.pool.root to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635125656 4/2/2020 12:03 slot1_7@c01-0 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_7@c01-006-174 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635245534 4/2/2020 12:31 slot1_17@c01- data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_17@c01-028-185 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4633877000 3/2/2020 10:43 aipanda404.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 4633896717 3/2/2020 11:11 aipanda404.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 4656155917 27/2/2020 8:44 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656220183 27/2/2020 10:38 slot1_8@c01-0 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_8@c01-008-103 4656236743 27/2/2020 11:10 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4635260683 4/2/2020 12:41 slot1_32@c01- mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_32@c01-115-105 pilot:::1152 File transfer timed out during stage-out: user.gangarbt:0133f01b-16b1-4323-a44e-a9c1ce7a0c09_84953.1.job.log.tgz to FZK-LCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633861179 3/2/2020 10:25 slot1_4@c01-1 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_4@c01-127-163 taskBuffer:::300 The worker was finished while the job was running : None 4633895371 3/2/2020 11:09 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 DE-TARDIS ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- FZK-LCG2 ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4657264382 28/2/2020 11:33 slot1_9@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_9@c01-117-115 taskBuffer:::300 The worker was finished while the job was running : None 4656132549 27/2/2020 7:55 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (271) job killed: vmem sup:::9000 Diag from worker : LRMS error: (271) job killed: vmem 4648776779 18/2/2020 8:40 slot1_9@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_9@c01-009-103 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.cb38d926-eb4d-4bed-afc7-a7f16d62cf1d_97996.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645275316 13/2/2020 16:49 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4645314183 13/2/2020 17:45 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4642913689 11/2/2020 5:19 slot1_15@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_15@c01-127-113 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.089f3083-afad-47d7-a581-969f07bd357a_48303.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633892925 3/2/2020 11:04 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 4633906896 3/2/2020 11:24 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 4632958897 2/2/2020 19:04 slot1_23@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_23@c01-116-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.851f8f78-d4b2-4af8-827f-a48487c11e43_1671.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4631497029 1/2/2020 17:23 slot1_9@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_9@c01-008-116 pilot:::1152 File transfer timed out during stage-out: hc_test:16fc7038-97be-45c2-ac14-79a63c8a5015_98068.1.job.log.tgz to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 16fc7038-97be-45c2-ac14-79a63c8a5015_98068.1.job.log.tgz is missing in pilot XML 4631556559 1/2/2020 18:14 slot1_30@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_30@c01-116-112 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.733b84c4-27ff-4b81-8d28-f17d0969b9ca_60121.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=311 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4631736291 1/2/2020 20:13 slot1_28@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_28@c01-118-126 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 20:23:50 sup:::2 worker already finished at 2020-02-01 20:43:22 with None 4656064792 27/2/2020 5:32 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656093989 27/2/2020 6:49 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656140201 27/2/2020 8:11 slot1_5@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@c01-008-177 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656205012 27/2/2020 10:00 slot1_19@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_19@c01-001-155 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656213676 27/2/2020 10:26 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4645257808 13/2/2020 16:28 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4634999028 4/2/2020 10:51 slot1_17@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_17@c01-115-103 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.a382b79d-bacd-430f-baf8-4f8cd1c4dc72_91365.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=320 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635157878 4/2/2020 11:39 slot1_16@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_16@c01-115-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.b0100612-8f1e-4737-95d8-8e59d7cac68c_46367.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=320 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635594713 4/2/2020 17:31 slot1_6@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_6@c01-025-157 pilot:::1152 File transfer timed out during stage-out: hc_test:dce0b40a-d43c-4587-b7a0-10733c5a5059_31634.1.job.log.tgz to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output dce0b40a-d43c-4587-b7a0-10733c5a5059_31634.1.job.log.tgz is missing in pilot XML 4633714383 3/2/2020 6:49 slot1_50@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_50@c01-119-105 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (271) job killed: vmem sup:::9000 Diag from worker : LRMS error: (271) job killed: vmem 4633854726 3/2/2020 10:04 slot1_14@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_14@c01-127-135 taskBuffer:::300 The worker was finished while the job was running : None 4655900510 27/2/2020 0:45 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655945310 27/2/2020 2:02 slot1_4@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@c01-003-117 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655972946 27/2/2020 2:49 slot1_8@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_8@c01-026-162 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655988875 27/2/2020 3:17 slot1_11@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_11@c01-000-186 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656002548 27/2/2020 3:38 slot1_2@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@c01-025-129 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656022002 27/2/2020 4:07 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656069794 27/2/2020 5:36 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656098197 27/2/2020 6:49 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656139215 None aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656198152 27/2/2020 9:44 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656239181 27/2/2020 11:14 slot1_57@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_57@c01-116-124 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000023.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656267762 27/2/2020 12:01 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4649854525 19/2/2020 7:11 slot1_10@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_10@c01-118-124 pilot:::1152 File transfer timed out during stage-out: hc_test:f3294aa6-56e5-4e80-8575-66af42d774b6_23028.1.job.log.tgz to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output f3294aa6-56e5-4e80-8575-66af42d774b6_23028.1.job.log.tgz is missing in pilot XML 4634799643 4/2/2020 10:50 slot1_16@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_16@c01-115-128 pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20159095.FZK-LCG2.75bb7850-69de-4ac9-a792-97df233b2706_13086.HITS.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635248723 4/2/2020 12:40 slot1_57@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_57@c01-115-113 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.c75721c9-b9d9-4c43-b86d-edcda0ea3c55_14010.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635430611 4/2/2020 15:06 slot1_112@c01 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_112@c01-116-124 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.b994b10f-5e00-4819-8d3f-489605ded9d6_89441.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635639860 4/2/2020 18:27 slot1_36@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_36@c01-116-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.e4d88b29-793e-466d-b42c-8267f05593be_51827.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635765267 4/2/2020 20:25 slot1_53@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_53@c01-116-104 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.f3e1cbbb-4557-45d5-a5a7-7b6e950b507b_3888.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633849722 3/2/2020 9:54 slot1_28@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_28@c01-116-112 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.0c73a0e4-a490-4c2d-bd3e-1da42e431205_29119.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4632502980 2/2/2020 10:56 slot1_4@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@c01-115-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.82de218c-9c6f-441f-8f9f-79782a2ef9fd_99296.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4632822454 2/2/2020 16:32 slot1_24@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_24@c01-115-114 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.4037c4b8-b226-4b73-95fb-1905d26c450d_67917.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4632859117 2/2/2020 17:16 slot1_14@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_14@c01-116-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.370f38c8-2ca8-47ff-9f08-9f0e029dbed3_57218.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4631423991 1/2/2020 16:17 slot1_26@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_26@c01-115-114 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.6b7790a7-f687-45fc-acda-e3148ea1859c_30082.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] FZK-LCG2_MCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4655332252 26/2/2020 17:20 slot1_10@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_10@c01-004-136 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000026.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655564791 26/2/2020 18:30 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655703828 26/2/2020 20:04 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655795000 26/2/2020 22:51 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4638302820 7/2/2020 7:09 slot1_5@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@c01-008-116 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.57fbbb71-5a81-4ac9-86c0-a19db4d17161_93364.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655376656 26/2/2020 17:22 slot1_20@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_20@c01-006-153 pilot:::1152 File transfer timed out during stage-out: hc_test:13457fdb-20e2-4b54-b79f-e2ae15d1a7f4_34237.job.log.tgz to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 13457fdb-20e2-4b54-b79f-e2ae15d1a7f4_34237.job.log.tgz is missing in pilot XML 4655575287 26/2/2020 18:30 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655725487 26/2/2020 22:51 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655866106 27/2/2020 0:01 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655940119 27/2/2020 2:31 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656011266 27/2/2020 4:45 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656092113 27/2/2020 7:12 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4653622351 24/2/2020 12:08 slot1_59@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_59@c01-117-128 taskBuffer:::300 The worker was finished while the job was running : None 4645201954 13/2/2020 17:36 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4637962441 6/2/2020 19:37 slot1_6@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_6@c01-009-157 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.ba5704dc-d1d3-4d5e-af13-8a7884fafb4d_36134.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635137039 4/2/2020 11:39 slot1_82@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_82@c01-115-113 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.a74c857d-3733-4776-9fdf-f5aa7ed66cc1_7739.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656215001 27/2/2020 10:29 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656247598 27/2/2020 11:26 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4635396115 4/2/2020 14:33 slot1_5@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@c01-116-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.1a6e7f60-b32c-4619-954e-bcdf26ed4385_62755.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=318 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658503269 29/2/2020 21:37 slot1_1@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@c01-008-156 taskBuffer:::300 The worker was finished while the job was running : None 4655900464 27/2/2020 0:44 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (271) job killed: wall sup:::9000 Diag from worker : LRMS error: (271) job killed: wall 4655956948 27/2/2020 2:24 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655991444 27/2/2020 3:21 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656036632 None aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656066550 27/2/2020 5:29 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656109935 27/2/2020 7:03 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656150083 27/2/2020 8:30 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656210187 27/2/2020 10:17 slot1_7@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_7@c01-025-154 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000030.pool.root.1 from FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656220533 27/2/2020 10:38 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656251253 27/2/2020 11:33 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4648047085 17/2/2020 12:29 slot1_1@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@c01-122-173 pilot:::1152 File transfer timed out during stage-out: hc_test:6b8850e0-8185-4555-a9f5-2bb0927a6c63_77219.1.job.log.tgz to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 6b8850e0-8185-4555-a9f5-2bb0927a6c63_77219.1.job.log.tgz is missing in pilot XML 4646115506 14/2/2020 13:19 slot1_9@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_9@c01-004-134 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.b5d95e11-058b-43ef-a877-980ce7a2a2c9_17879.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645273519 13/2/2020 16:47 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4645326307 13/2/2020 18:13 slot1_96@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_96@c01-117-113 taskBuffer:::300 The worker was finished while the job was running : None 4637779605 6/2/2020 13:43 slot1_7@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_7@c01-008-116 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.e98ed518-4aee-4f17-a96e-727da1dc5382_55040.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4634799669 4/2/2020 10:56 slot1_81@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_81@c01-116-114 pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20159095.FZK-LCG2_MCORE.c705eec3-ec48-4726-931c-2d0914b0ff26_95326.HITS.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635199716 4/2/2020 11:58 slot1_15@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_15@c01-000-173 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635240257 4/2/2020 12:32 slot1_33@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_33@c01-115-124 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.dd786e54-0ba5-4902-a864-5653747ea4f8_6635.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635330969 4/2/2020 13:38 slot1_12@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_12@c01-120-176 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635627743 4/2/2020 18:14 slot1_56@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_56@c01-116-114 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.56769eb8-a85b-4f10-b531-33c2c584844e_69371.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635731977 4/2/2020 19:55 slot1_31@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_31@c01-116-101 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.18efc8ef-b984-404f-8197-acb6eb805b1c_92425.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635823261 4/2/2020 21:21 slot1_27@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_27@c01-116-127 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.b6a8601c-2798-4cba-ad5f-a3e5c28f9f1f_80471.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633699889 3/2/2020 6:37 slot1_25@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_25@c01-115-102 pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20158885.FZK-LCG2_MCORE.876c5a43-9fb1-43cb-a794-c3da9f11d4b0_67081.HITS.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633877453 3/2/2020 10:42 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 4656132516 27/2/2020 7:55 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656182080 27/2/2020 9:23 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656222754 27/2/2020 10:42 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656251477 27/2/2020 11:34 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4653823677 24/2/2020 15:23 slot1_7@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_7@c01-117-124 pilot:::1137 Failed to stage-out file: hc_test:b5b9f88e-1ad7-45a0-95e9-14aa7c8778c6_87173.1.job.log.tgz to FZK-LCG2_DATADISK, globus_ftp_client: the server responded with an error 451 Failed to deliver PoolMgrSelectWritePoolMsg message <1582558467304:188644049> to [SpaceManager@local:>PoolManager@poolmanager-f01-081-1 ddm:::200 expected output b5b9f88e-1ad7-45a0-95e9-14aa7c8778c6_87173.1.job.log.tgz is missing in pilot XML 4645275501 13/2/2020 16:50 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4645305824 13/2/2020 17:35 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4641036188 9/2/2020 16:44 slot1_16@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_16@c01-025-104 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.8b5c1669-4f6d-4152-9142-ad03f58e5042_97991.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638142176 7/2/2020 1:54 slot1_9@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_9@c01-008-107 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.b19a4196-05bf-4192-822a-f1956c89a5fb_50979.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4637203805 5/2/2020 22:39 slot1_4@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@c01-009-117 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.b9def787-7da2-4046-b65c-a8f75f0343f1_49155.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635317063 4/2/2020 13:28 slot1_2@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@c01-006-167 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635630505 4/2/2020 18:19 slot1_30@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_30@c01-116-102 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.83ff9626-59ec-4bf6-939c-566a1d602509_3225.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635776274 4/2/2020 20:35 slot1_41@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_41@c01-116-124 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.a23386a4-d818-4064-bf23-030f15205952_21317.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4632675559 2/2/2020 14:09 slot1_19@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_19@c01-116-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.0e7020dc-dfa8-4a0a-a263-08ef94342117_35978.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656084322 27/2/2020 6:07 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656136685 27/2/2020 8:01 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656180011 27/2/2020 9:22 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4656225330 27/2/2020 10:49 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4655139006 26/2/2020 8:34 slot1_48@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_48@c01-116-115 pilot:::1152 File transfer timed out during stage-out: hc_test:e36f8536-17bf-4190-a1e4-346d5a604dbb_6636.1.job.log.tgz to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output e36f8536-17bf-4190-a1e4-346d5a604dbb_6636.1.job.log.tgz is missing in pilot XML 4647922223 17/2/2020 7:30 slot1_113@c01 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_113@c01-118-115 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.dd5a61ce-bd33-4197-af1f-3e8aed85241e_96742.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=326 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645230323 13/2/2020 16:13 slot1_32@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_32@c01-118-114 taskBuffer:::300 The worker was finished while the job was running : None 4645296096 13/2/2020 17:18 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was finished while the job was starting : None 4635309380 4/2/2020 13:26 slot1_19@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_19@c01-115-103 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.d197f134-a2a0-44e6-bf1f-e3a90dfdb341_72611.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=319 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633749038 3/2/2020 7:39 slot1_14@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_14@c01-115-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.a51631eb-17fe-42bd-bbf6-9486bd459c30_9780.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=324 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657897244 29/2/2020 2:37 slot1_1@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@c01-007-105 taskBuffer:::300 The worker was finished while the job was running : None 4641253747 9/2/2020 22:56 slot1_3@c01-0 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_3@c01-002-112 pilot:::1099 Failed to stage-in file: [CannotAuthenticate("x509 authentication failed for account=pilot with identity={'client_proxy': '/tmp/condor_execute/dir_12142/job.9y5MDmugALwn97UVzn6VJfgoABFKDmABFKDm2I3KDmABFKDmdHY7Qn.proxy'}",), CannotAuthenticate("x509 authentication failed for account=pilot with identity={ 4635554147 4/2/2020 16:51 slot1_7@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_7@c01-116-116 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.088796bd-f4e6-4a2f-a6fb-d4d5d1baad6b_53117.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635393618 4/2/2020 14:28 slot1_4@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@c01-116-128 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.4193195c-d88c-41e0-8fc9-cda497e1c396_36374.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4632595671 2/2/2020 12:58 slot1_3@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_3@c01-116-112 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.e05f6fab-3c7b-4b96-aa14-d8ea2ee6fade_90544.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633641859 3/2/2020 5:15 slot1_5@c01-1 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@c01-115-104 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.5372b3b6-3a0e-40fa-88b8-d338dc6b467f_66807.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4633899983 3/2/2020 11:16 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 4630507835 1/2/2020 4:20 slot1_12@c01- hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_12@c01-116-104 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.236b493d-e1de-49ba-9958-772abe426423_16888.pool.root to FZK-LCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] ====== GoeGrid ANALY_GOEGRID ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4659082081 1/3/2020 19:41 compute-19-10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-10.local pilot:::1324 Service not available at the moment: user.gangarbt:7efd93c2-da2d-4062-9289-41f4cc3d26fb_4365.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 7efd93c2-da2d-4062-9289-41f4cc3d26fb_4365.job.log.tgz is missing in pilot XML 4659087656 1/3/2020 19:57 compute-17-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.cb931d9f-6f3e-497a-a195-4214f5fbe3c2_27686.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659108544 1/3/2020 21:14 compute-19-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-9.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.9ab6b6a0-b7ca-488a-a4dd-7ae2c190a96d_13330.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659177946 1/3/2020 22:09 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1324 Service not available at the moment: user.gangarbt:9b01aa56-9451-4c52-afff-822d0b15ee3b_79152.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 9b01aa56-9451-4c52-afff-822d0b15ee3b_79152.1.job.log.tgz is missing in pilot XML 4659194467 1/3/2020 22:36 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:7ef648fa-9063-4934-8aef-28990f07ad8e_70420.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 7ef648fa-9063-4934-8aef-28990f07ad8e_70420.1.job.log.tgz is missing in pilot XML 4659206463 1/3/2020 23:12 compute-19-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-9.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.a4136acc-c4b7-46b0-80bb-49ccb01a631d_11243.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4659225480 1/3/2020 23:31 compute-19-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-9.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:fdea4b7a-6abe-41cf-8a05-fe38da646d23_1506.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output fdea4b7a-6abe-41cf-8a05-fe38da646d23_1506.1.job.log.tgz is missing in pilot XML 4658388377 29/2/2020 18:03 compute-19-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-9.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:d1ec4af6-2a1e-4567-8584-ebba786422c7_51520.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output d1ec4af6-2a1e-4567-8584-ebba786422c7_51520.job.log.tgz is missing in pilot XML 4658402548 29/2/2020 18:36 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.5324f69d-c879-4714-a2bd-ea1d3e20f149_16301.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658425859 29/2/2020 19:13 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d88ca0da-766b-4c44-9b20-c3e411443f43_67274.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658439743 29/2/2020 19:34 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:41319d89-ac9d-4ae9-932b-55c9f1151f6e_26622.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 41319d89-ac9d-4ae9-932b-55c9f1151f6e_26622.1.job.log.tgz is missing in pilot XML 4658451057 29/2/2020 19:55 compute-17-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.92a82cfa-8efd-4f9a-ae29-713bba7d6a8f_29995.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658464679 29/2/2020 20:21 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:e096bcff-429b-4c31-9505-ef1b6ff12cca_13132.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output e096bcff-429b-4c31-9505-ef1b6ff12cca_13132.1.job.log.tgz is missing in pilot XML 4658474935 29/2/2020 20:42 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:af4d1a34-147f-4371-8458-0b05290dc554_11781.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output af4d1a34-147f-4371-8458-0b05290dc554_11781.1.job.log.tgz is missing in pilot XML 4658496606 29/2/2020 21:23 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658551740 29/2/2020 23:10 compute-17-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:cf29e4cb-ea46-478f-8943-93736f2daceb_22952.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output cf29e4cb-ea46-478f-8943-93736f2daceb_22952.1.job.log.tgz is missing in pilot XML 4658574129 29/2/2020 23:39 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.761c72ec-d807-4f7f-b2d5-d8e3ea7fb5b9_68651.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658593728 1/3/2020 0:06 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.7f6c52d4-04ee-4989-aaa3-51a0ab1a7463_79613.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658605649 1/3/2020 0:37 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:83329fa8-e8a2-484a-87bb-095f5edf60a9_60458.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 83329fa8-e8a2-484a-87bb-095f5edf60a9_60458.1.job.log.tgz is missing in pilot XML 4658618725 1/3/2020 0:58 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.34926acf-0f3e-4ee3-a613-e3eb43fc27af_21602.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658630849 1/3/2020 1:26 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.7451cedf-87f2-4ac8-87e7-4054ca8641ec_81778.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658641809 1/3/2020 1:53 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.e05fefd3-1dd2-4c74-bcef-afc3aa72e3bf_64675.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658654384 1/3/2020 2:17 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:8eb55fbe-8950-4666-858e-3ed8a4d37861_56137.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 8eb55fbe-8950-4666-858e-3ed8a4d37861_56137.1.job.log.tgz is missing in pilot XML 4658667085 1/3/2020 2:44 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.bcff4ade-59dc-4143-8c62-4b7d1c7a36f3_57849.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658684545 1/3/2020 3:15 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:ea3b4561-5c08-4276-ab65-fc3c7348df7b_27592.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output ea3b4561-5c08-4276-ab65-fc3c7348df7b_27592.1.job.log.tgz is missing in pilot XML 4658694413 1/3/2020 3:39 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:b45e1d99-6700-47bc-b8d2-60bac071cff3_59478.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output b45e1d99-6700-47bc-b8d2-60bac071cff3_59478.1.job.log.tgz is missing in pilot XML 4658706892 1/3/2020 4:15 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.54a3e9aa-a398-48d3-b494-077d9527b9a8_86729.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658716651 1/3/2020 4:42 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.60377150-8201-49c6-b864-f88fea471186_44980.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658724504 1/3/2020 5:14 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:2ce8f26e-c23d-474e-aa88-63a464c60725_7843.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 2ce8f26e-c23d-474e-aa88-63a464c60725_7843.1.job.log.tgz is missing in pilot XML 4658733014 1/3/2020 5:39 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.4f0fdf2e-a3df-40c6-83b8-fadca5303e71_60886.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658742973 1/3/2020 6:09 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:8e70372d-75a2-4f29-a4e0-3aa4602a865f_87019.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 8e70372d-75a2-4f29-a4e0-3aa4602a865f_87019.1.job.log.tgz is missing in pilot XML 4658752720 1/3/2020 6:40 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658815911 1/3/2020 9:55 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.afeae05c-45b0-4f4c-b917-6764a9f6c95c_92058.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658827959 1/3/2020 10:32 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d7be081b-c8a0-498d-8dc8-c51e4e88a2ca_81666.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658868069 1/3/2020 12:06 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:924b960d-7376-44e2-942c-80cc99c83afa_90862.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 924b960d-7376-44e2-942c-80cc99c83afa_90862.1.job.log.tgz is missing in pilot XML 4658888828 1/3/2020 13:12 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1324 Service not available at the moment: user.gangarbt:9809d3e6-8e13-470f-b5e4-ccc85a9b36e4_56972.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 9809d3e6-8e13-470f-b5e4-ccc85a9b36e4_56972.1.job.log.tgz is missing in pilot XML 4658914305 1/3/2020 13:55 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:01acf68a-3732-487e-82e9-a8d80acfdb3e_17328.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 01acf68a-3732-487e-82e9-a8d80acfdb3e_17328.1.job.log.tgz is missing in pilot XML 4658965365 1/3/2020 16:13 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.e4de6303-abf9-4a44-984c-725ca526d911_56874.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658982695 1/3/2020 16:49 compute-19-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-9.local pilot:::1324 Service not available at the moment: user.gangarbt:f7c3714d-f721-4bed-8152-80ef5d7f4155_67382.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output f7c3714d-f721-4bed-8152-80ef5d7f4155_67382.1.job.log.tgz is missing in pilot XML 4658991859 1/3/2020 17:22 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.4436c387-8fe7-4e54-8e7a-7458a8b52d8a_28189.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659013140 1/3/2020 18:07 compute-17-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.6cbfe06a-f180-4bb3-87ea-67521fe08053_75604.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657600316 28/2/2020 19:19 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:hc_20163936.ANALY_GOEGRID.ebb299b7-506a-4c70-b501-7ebe8e40abc6_27430.hist-out.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657606729 28/2/2020 19:29 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.42539f99-0b9e-48b1-8c9a-e0d8c8ae7a9e_73417.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657624271 28/2/2020 19:48 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ea2a551a-2795-4a07-a1f0-e09a46d7aaec_585.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657643462 28/2/2020 20:04 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.0161846f-51b5-4744-b3a4-32830e9014e5_36443.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657659703 28/2/2020 20:21 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.5959f1a5-581d-4ec5-95cb-1c494c72c1fb_58922.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657670924 28/2/2020 20:41 compute-19-12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.df88ce24-b151-4ba2-a336-fcf394c6acbd_81234.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657687437 28/2/2020 20:57 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.84537280-0dca-484e-a7f0-f41f5e09ecb8_45179.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657696240 28/2/2020 21:10 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.6747405a-2c4a-4fc7-abfd-9dbfd1577f65_9341.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657702638 28/2/2020 21:24 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.12e088e6-f1c2-499c-a9fa-07679ac9c2f7_44411.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657711923 28/2/2020 21:41 compute-19-7. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-7.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ceb38db5-91fb-46a7-bf1c-46e7f72b77de_63445.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657723029 28/2/2020 21:55 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.75f79c25-37c8-42a2-808d-5ec9dcc97ad5_50587.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657735064 28/2/2020 22:14 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.6fad5f02-961a-420d-8db0-eea6def5efd8_2130.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657747397 28/2/2020 22:31 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.a07b1eb1-4c68-48fe-8c8e-300ddbb8629a_22281.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657754615 28/2/2020 22:47 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.0ad18f40-56f3-4d36-b9ce-8992115f01c5_58940.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657762949 28/2/2020 23:02 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.a41ed3ab-71f8-450c-ad20-d8b8a4e125d3_18095.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657771342 28/2/2020 23:24 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ef797c2d-dca3-43c5-9c93-30781fd8d4de_98752.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657794384 28/2/2020 23:40 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4657861811 29/2/2020 1:26 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.2fe799fb-bace-4827-8642-f5a4a3898f26_53560.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657870337 29/2/2020 1:39 compute-17-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.1e90263f-c6aa-4d98-a00f-b6ea518889a4_80385.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657881383 29/2/2020 2:05 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.8102a3b5-7cf4-4db4-9579-21a74f5866d5_63998.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657898130 29/2/2020 2:40 compute-17-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.78c1b1e5-222e-4621-bd10-831dd43bc033_4259.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657913181 29/2/2020 2:55 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.10b00dbb-0c64-458e-854a-0f56512d4074_84910.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657929559 29/2/2020 3:22 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.beaf6064-c9fb-4188-962b-9acec10a5dac_16016.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657940367 29/2/2020 3:40 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658004014 29/2/2020 5:28 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.d4be75ad-d2ac-46c0-b650-833269dc21b5_84259.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658014522 29/2/2020 6:02 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.6e46b016-bcd7-4162-b6bc-60577a7c52e5_54193.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658024800 29/2/2020 6:38 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.856f4f56-bf88-487f-92d7-a13ce8e86928_74037.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658040845 29/2/2020 7:02 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.4687893a-8be7-4883-a86b-e918ae55f4ee_88283.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658052953 29/2/2020 7:24 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.a300848e-3cd4-445a-957e-7fac60cb6308_38093.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658079370 29/2/2020 7:50 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.d1f439c0-8c1f-4ded-80ef-2a445bcb54c5_16099.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658095623 29/2/2020 8:13 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.709d7f8a-e23a-4129-a208-5e2c53a4883c_54416.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658104360 29/2/2020 8:30 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.8964e5f2-b257-47c4-8476-9f138bf03087_20309.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658112411 29/2/2020 8:48 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.b4c56a30-4730-4a9c-b192-4c4d97f2639c_3470.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658120108 29/2/2020 9:09 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.fcd70c2e-9b16-44ab-bca0-cf3d2cb32028_61687.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658129434 29/2/2020 9:35 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.64933cf9-5ead-4aea-9910-2fbb9472b5a2_20065.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658138794 29/2/2020 9:57 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.3b250c77-1581-432d-ba87-55f2fa9eef12_88712.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658145884 29/2/2020 10:17 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.03d59092-700b-4642-8de7-3b2aec422e72_95311.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658154010 29/2/2020 10:38 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.4ac037cb-32b4-4c4e-8690-476077e5d5dc_1453.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658169403 29/2/2020 11:06 compute-19-12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.cf9ccbb1-f543-4873-b573-9cb5a5845724_51004.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658182292 29/2/2020 11:32 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.0bfc8b00-9e35-4bbd-a7de-c27a2096abec_87686.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658198157 29/2/2020 12:00 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.fd08e128-8080-48aa-b72b-ea8b0c6742bf_39759.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658214920 29/2/2020 12:36 compute-19-14 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-14.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.04a70ce0-b2a1-4406-8624-02bbd40038e1_24072.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658226956 29/2/2020 12:59 compute-19-14 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-14.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c78c092f-456e-4576-a951-a00cbeaf7ffe_93369.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658245311 29/2/2020 13:34 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.113a789e-7b7e-4dc7-adab-e94e1f123dc7_90843.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658268626 29/2/2020 14:12 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.999cf595-fd32-4574-a008-451159874cda_26973.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658284590 29/2/2020 14:44 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4658302536 29/2/2020 15:09 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4658324017 29/2/2020 15:47 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.21963269-37d0-495a-bb3c-1f7b8a31de3f_80434.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658336390 29/2/2020 16:20 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.4d3959de-fb4e-4713-ac59-c3f0d78a5970_91321.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658349174 29/2/2020 16:44 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ced3fdc6-3c4b-4b76-9f19-213fe089820d_2763.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658361216 29/2/2020 17:21 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.f6e9db73-18f2-41e2-9596-99324a37aa10_77117.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4656830099 28/2/2020 0:33 compute-19-14 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-14.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656843726 28/2/2020 0:54 compute-19-7. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-7.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656855600 28/2/2020 1:15 compute-19-7. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-7.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656866701 28/2/2020 1:31 compute-19-7. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-7.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656875540 28/2/2020 1:51 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656887742 28/2/2020 2:10 compute-19-7. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-7.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4648216322 17/2/2020 17:23 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646815966 15/2/2020 11:56 compute-17-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647166970 15/2/2020 23:29 compute-18-12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-18-12.local pilot:::1201 Job killed by signal: SIGTERM 4646544521 15/2/2020 0:37 compute-17-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4645314286 13/2/2020 17:49 compute-17-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-3.local 4645642174 14/2/2020 3:04 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4644389024 12/2/2020 17:57 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local taskBuffer:::300 The worker was finished while the job was running : None 4644464971 12/2/2020 19:14 compute-15-10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-15-10.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c3cb61f0-ae36-4ff2-8ab2-3ba3c2915c60_16689.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4640738942 9/2/2020 8:01 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:770da8e6-8c9f-4cf3-90c5-ad528db8ccd4_13375.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 770da8e6-8c9f-4cf3-90c5-ad528db8ccd4_13375.1.job.log.tgz is missing in pilot XML 4638069829 6/2/2020 23:36 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635217307 4/2/2020 12:08 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4659106463 1/3/2020 21:12 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.cb1e7488-7c6c-4892-a2dd-baa8dbd0b086_84014.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659166676 1/3/2020 21:47 compute-19-6. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:df3eda2e-28ff-46f2-bd12-4082924089cd_2574.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output df3eda2e-28ff-46f2-bd12-4082924089cd_2574.1.job.log.tgz is missing in pilot XML 4659186457 1/3/2020 22:21 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.3c632bf8-e0a5-4cff-bd53-175a6ba4ee65_5476.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659214002 1/3/2020 23:13 compute-19-7. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-7.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.89483c7e-e334-444c-9c87-5b449878f3f5_56854.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658472573 29/2/2020 20:40 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:user.gangarbt.7352e01c-20e5-4c02-98ff-628d6321b194_40298.DAOD_EXOT12.test.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658495472 29/2/2020 21:19 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.bfb711f4-dc30-452b-b2e2-afd754226219_93396.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658511858 29/2/2020 21:49 compute-19-10 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-10.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:af44146a-ab63-4aaf-9fd2-8b2816a811f0_33161.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output af44146a-ab63-4aaf-9fd2-8b2816a811f0_33161.1.job.log.tgz is missing in pilot XML 4658533525 29/2/2020 22:39 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.2dc18cc8-579f-47f5-9f71-1a1fa555fe43_77098.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658550475 29/2/2020 23:11 compute-19-1. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-1.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:648fc9dc-a7ce-4cda-9d26-4706f3bb430e_73057.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 648fc9dc-a7ce-4cda-9d26-4706f3bb430e_73057.1.job.log.tgz is missing in pilot XML 4658573636 29/2/2020 23:36 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.45c5ab12-5f8a-4f00-bc75-d61a5f596a16_2957.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658592591 1/3/2020 0:07 compute-19-1. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-1.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.8fe7d709-36b9-49d8-bf69-3b82ffd96438_27997.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658605285 1/3/2020 0:37 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.7e9cbea7-708c-4776-a1e9-8e236e347d62_50134.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658623563 1/3/2020 1:11 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.8e44658c-44b4-47a4-8d38-e6c51609cd4e_95554.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658633849 1/3/2020 1:36 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.b4aeeb35-d4f7-48de-9c90-c5636bf6a5f8_17672.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658647322 1/3/2020 2:14 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:6f5708ea-1747-4dab-a3b6-df4763dcbbd7_22907.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 6f5708ea-1747-4dab-a3b6-df4763dcbbd7_22907.1.job.log.tgz is missing in pilot XML 4658668684 1/3/2020 2:44 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.66330c56-cdd2-4771-8118-9ed925b04cc4_49157.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658712450 1/3/2020 4:25 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:a60bde4c-e288-4680-8945-fe5c5389254f_49059.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output a60bde4c-e288-4680-8945-fe5c5389254f_49059.1.job.log.tgz is missing in pilot XML 4658719768 1/3/2020 4:52 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.556fe27d-fd00-4f67-b2ba-1e0bc2c8eb01_2684.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658747754 1/3/2020 6:20 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:aa11d248-19ce-4021-82d6-689796163b62_54397.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output aa11d248-19ce-4021-82d6-689796163b62_54397.1.job.log.tgz is missing in pilot XML 4658754322 1/3/2020 6:46 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.175c3e4d-3c2b-459c-afd1-462e900ad1d6_48739.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658767652 1/3/2020 7:36 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.d7b0024b-9243-4100-8acf-651427a0dfb5_26979.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658773025 1/3/2020 7:56 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:37a2c52c-44e8-4782-a082-e56f8b031823_54454.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 37a2c52c-44e8-4782-a082-e56f8b031823_54454.1.job.log.tgz is missing in pilot XML 4658783938 1/3/2020 8:30 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.e917ed4d-ad19-4dc0-b7aa-c10b70f43d5c_66650.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658799659 1/3/2020 9:04 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:30341928-9dd5-4d83-b60b-16ed0f954456_97271.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 30341928-9dd5-4d83-b60b-16ed0f954456_97271.1.job.log.tgz is missing in pilot XML 4658808557 1/3/2020 9:52 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:c63c4dce-65f7-42fe-91b5-2db75eb9b4cb_22122.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output c63c4dce-65f7-42fe-91b5-2db75eb9b4cb_22122.1.job.log.tgz is missing in pilot XML 4658849612 1/3/2020 11:20 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1bd9044d-3bb4-436e-81a3-4993c0ae7fdd_44799.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658902132 1/3/2020 13:38 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: user.gangarbt:ca9294a0-5f5d-4c38-b4c1-e4da532ae60c_88178.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output ca9294a0-5f5d-4c38-b4c1-e4da532ae60c_88178.1.job.log.tgz is missing in pilot XML 4658928804 1/3/2020 14:49 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d09681d1-6dc7-4feb-918a-5dc0aa1f4c74_10245.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658946779 1/3/2020 15:24 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.fded9daf-e323-43e7-b62f-2b99929a66a8_29471.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658957872 1/3/2020 15:57 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: user.gangarbt:ceaa5419-91d0-4900-b8d9-b1b56b3c7208_41712.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output ceaa5419-91d0-4900-b8d9-b1b56b3c7208_41712.1.job.log.tgz is missing in pilot XML 4658983006 1/3/2020 17:08 compute-19-14 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-14.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1843dda8-0caa-425c-921b-62fdf245d1d7_19391.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659005241 1/3/2020 17:53 compute-15-6. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-15-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.9f2b785f-ab0b-4caa-8e82-7cc385bd1ece_57012.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657598788 28/2/2020 19:21 compute-17-6. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:user.gangarbt.4e39e77f-bb4f-4ae9-95b8-8efe0360fadb_82009.DAOD_EXOT12.test.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657620104 28/2/2020 19:38 compute-17-7. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-7.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.67b9ab84-7251-4630-b332-34bf9a34b055_75860.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657637317 28/2/2020 19:56 compute-17-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.77fa26f1-30bb-4132-8903-7396627b3d8f_79027.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657653412 28/2/2020 20:13 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.de564394-21db-4e1c-b2f0-4787e94170a2_81902.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657669348 28/2/2020 20:41 compute-19-12 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.c95e04a7-99e9-45dd-b547-a2ee95c5947c_79746.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657688860 28/2/2020 20:57 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.7f073550-80cd-4960-b2e4-6e17c03182b0_21978.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657697715 28/2/2020 21:13 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.57fecfe2-a2a8-467e-a3ba-f72a8fc7c214_38078.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657703388 28/2/2020 21:24 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.77275d2d-e4d2-4d03-9ea0-6f26cbeaba1d_42282.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657716019 28/2/2020 21:45 compute-19-6. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.875e349f-45df-4bd6-a020-1b852cfda794_16753.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657732010 28/2/2020 22:11 compute-19-6. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.6828a2fd-7ed4-4e1e-9eaf-bd3a1d63fa9c_7238.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657747331 28/2/2020 22:30 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.5253156e-c7ce-4e44-80d0-e449b6c0127b_14762.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657757014 28/2/2020 22:50 compute-19-6. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.1d567f64-1a9a-4d73-be62-0e5c4990c1a7_84638.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657766983 28/2/2020 23:08 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.afbe3cee-eb5b-4067-8aa2-15d84d18e514_4877.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657785119 28/2/2020 23:28 compute-17-6. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.564c7c38-b7f1-4eaf-8d4f-586f12fea5e0_95081.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657796391 28/2/2020 23:45 compute-17-7. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-7.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.3e9dee99-4747-4d95-9029-6e0ff3e16f27_95794.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657807206 29/2/2020 0:03 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.c75f2a4b-92d3-455b-85ac-c03b47e2968e_4618.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657826983 29/2/2020 0:27 compute-19-13 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.b43ec105-6061-440e-a87a-12a1b56f350d_47369.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657838913 29/2/2020 0:44 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.18453112-aa37-42bc-8596-6bfb8b328b61_63121.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657850947 29/2/2020 1:07 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.f2bb7dee-8f98-4db6-a0b0-208cef6575f7_69882.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657860876 29/2/2020 1:26 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.84bddcff-f7af-4b15-9cae-bdff16d2d796_52554.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657875581 29/2/2020 1:51 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.f334db39-f5f1-448b-b881-95f2eef2e083_24791.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657887691 29/2/2020 2:11 compute-17-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.4ce92bf0-6b51-4685-b9e6-07c9a2289625_6929.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657899404 29/2/2020 2:40 compute-17-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.1b5cdbc3-3193-42ad-bc0f-60a9f64f1688_23269.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657917456 29/2/2020 3:04 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.d5497ae4-f60c-4ffa-95fc-d2b46f27dbe1_77932.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657931219 29/2/2020 3:28 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.c74fc0eb-57d2-4ffc-9d4c-04ca2ab39e53_74282.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657948985 29/2/2020 3:48 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.e59d48e1-a455-47e2-9b78-54ced459559a_22460.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657961777 29/2/2020 4:14 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.86f18d0b-41d2-4f96-b3dc-3f8a5369a935_40723.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657974296 29/2/2020 4:43 compute-19-14 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-14.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ba716baa-beaf-4a8c-8387-3de28e0b8e89_42110.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657985988 29/2/2020 5:03 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.1d52fb0b-5cbb-4f03-aeda-18d63901e523_51674.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658003421 29/2/2020 5:28 compute-19-12 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.76bd35c9-3355-4e0c-8265-04042e6e7201_43552.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658012218 29/2/2020 5:48 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.75ff2492-0a57-4d06-9bbd-6052ac829c36_59960.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658020347 29/2/2020 6:11 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.c92804b5-0304-4ad6-ae60-20fc9bd4233c_98907.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658026260 29/2/2020 6:40 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.cfd7a212-e985-4fb1-8c7b-52d2570c6d0a_99836.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658040073 29/2/2020 7:02 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ab9c93ff-8ac1-4acd-9c97-dd6449d1b054_81082.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658057507 29/2/2020 7:25 compute-19-8. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.30b60bc0-9c3d-4be6-9e69-b791d380909b_38079.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658080080 29/2/2020 7:50 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.6fc30a15-bcd4-423b-a725-57a1a0a26b37_43095.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658097430 29/2/2020 8:15 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.4531cebe-cd4c-40d1-be31-821c20a4a4f3_19918.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658107014 29/2/2020 8:39 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.f4a41209-0405-48cb-87d7-1ca98649d3f9_97171.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658119555 29/2/2020 9:06 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.d7b72180-da0e-4c46-acca-9e7505db70f6_77547.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658126403 29/2/2020 9:29 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ede74562-ae55-49c5-99dd-120fb9e99f57_91766.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658137339 29/2/2020 9:53 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ebd47b68-2cb6-4bc2-b4c9-9845fb500a73_87285.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658150921 29/2/2020 10:33 compute-19-13 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.24042a4a-8b97-49e2-a403-7d9214ad44ff_12108.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658165397 29/2/2020 11:00 compute-19-12 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.43eb4cd8-bafc-4dc0-8c47-be483a4a5957_58124.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658181336 29/2/2020 11:24 compute-19-14 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-14.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.0a4c2ef9-a5a3-49b0-93d9-b675b9d3e4f9_15110.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658192958 29/2/2020 11:51 compute-19-12 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-12.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.8b0a3bd5-0690-40c8-a8e1-ac1ef3eaeaf6_53263.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658209425 29/2/2020 12:23 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.1cf61b80-2d08-4916-ac88-09443fe31475_19216.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658222836 29/2/2020 12:51 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658282641 29/2/2020 14:40 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4658302341 29/2/2020 15:09 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4658318845 29/2/2020 15:41 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.4a9bc0ae-7f84-4323-8991-5ae28e5812d0_5988.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658334674 29/2/2020 16:09 compute-19-13 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ece94636-d61c-4871-bd40-ebce97966c51_97626.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658344305 29/2/2020 16:44 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.8186fd42-7caf-4f89-bfa8-b73e9a22f7ef_57119.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658362937 29/2/2020 17:55 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.e33b5174-571e-45d8-83df-eff4bcbc08d9_86454.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658409704 29/2/2020 18:43 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.e6943bea-4f8b-42ef-b76f-b4c75c61913f_61233.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658432465 29/2/2020 19:21 compute-19-8. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-8.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.9773a6c0-d48a-40b0-aec2-e7064ea252f8_62573.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658450678 29/2/2020 19:54 compute-17-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.4ac626ef-636a-4930-b572-e76a151b89fd_41163.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656555116 27/2/2020 17:44 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656571385 27/2/2020 18:04 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656583689 27/2/2020 18:24 compute-15-7. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-15-7.local pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656593501 27/2/2020 18:42 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4648189704 17/2/2020 16:41 compute-19-10 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-10.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648638049 18/2/2020 4:07 compute-15-13 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-15-13.local pilot:::1186 Software directory does not exist 4648711870 18/2/2020 6:44 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648813586 18/2/2020 9:41 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647882429 17/2/2020 5:24 compute-19-4. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647064875 15/2/2020 18:31 compute-9-9.l data18_13TeV:data18_13TeV.0034926 modificationHost: compute-9-9.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646244667 14/2/2020 16:39 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646489018 14/2/2020 22:58 compute-19-11 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-11.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.0838013f-5e0e-43e5-b4ce-20a47c648d4a_91282.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4646569544 15/2/2020 1:29 compute-17-12 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-12.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4645160206 13/2/2020 14:06 compute-16-9. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-16-9.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:6409b509-30d4-4fa2-a2c4-976c5d3c8bcb_97147.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 6409b509-30d4-4fa2-a2c4-976c5d3c8bcb_97147.1.job.log.tgz is missing in pilot XML 4645211080 13/2/2020 15:34 compute-19-12 data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-12.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.3ce3591d-adac-4e9c-8ce9-5a0b85fa6b1c_44691.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4645301132 13/2/2020 17:26 compute-17-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-17-3.local taskBuffer:::300 The worker was finished while the job was running : None 4645521854 13/2/2020 23:00 compute-19-1. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-1.local pilot:::1322 Failed to get replicas from Rucio: An unknown exception occurred. Details: no error information passed (http status code: 503 ('service_unavailable', 'unavailable'))",)]:failed to transfer files using copytools=[u'rucio'] 4645666360 14/2/2020 4:14 compute-19-5. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4643776479 12/2/2020 3:19 compute-19-9. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-9.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.291e7a34-2818-4c91-a064-3d50ed474bac_82284.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4640199822 8/2/2020 16:37 compute-19-3. data18_13TeV:data18_13TeV.0034926 modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4659084969 1/3/2020 19:48 compute-17-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-9.local pilot:::1324 Service not available at the moment: user.gangarbt:d615c717-3570-4145-b380-9d27ab3c2a8d_68633.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output d615c717-3570-4145-b380-9d27ab3c2a8d_68633.1.job.log.tgz is missing in pilot XML 4659089782 1/3/2020 20:03 compute-17-10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-10.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.6ac4b997-509a-4447-bae2-e37df5a8b52c_190.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4659100376 1/3/2020 20:37 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.e0af0d8f-ed97-4363-9c56-fa0d5c50abef_36683.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659109473 1/3/2020 21:14 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:1c3cf7df-3bf3-4b49-aa13-0285d6ea169d_46037.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 1c3cf7df-3bf3-4b49-aa13-0285d6ea169d_46037.1.job.log.tgz is missing in pilot XML 4659163067 1/3/2020 21:35 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c1f4cbdd-eb37-46e3-8567-9e16097b3b08_68504.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659174004 1/3/2020 22:02 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.6795a2bd-3a3a-47e3-8bc1-f09c12534826_36997.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659203168 1/3/2020 22:54 compute-19-7. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-7.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:3b9ae398-88aa-4f46-ad87-5bf76941bec8_44389.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 3b9ae398-88aa-4f46-ad87-5bf76941bec8_44389.1.job.log.tgz is missing in pilot XML 4658415982 29/2/2020 18:47 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:user.gangarbt.user.gangarbt.hc20164158.tid952.ANALY_GOEGRID.136.f5201386-71c1-47c5-a7d4-df3b93482bff_87845.hist-output.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658429393 29/2/2020 19:15 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.eabc852a-dc92-4aa9-b672-54eecbcf97a7_53956.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658440433 29/2/2020 19:36 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658478182 29/2/2020 20:49 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c2ceab1f-e9ea-4f6a-b10a-11ff136d55af_54843.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658490673 29/2/2020 21:12 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.f1c94fec-cea3-4d85-b921-83a586bffb59_32223.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658506549 29/2/2020 21:41 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:64353827-3d45-4938-a223-23ccb034225a_56843.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 64353827-3d45-4938-a223-23ccb034225a_56843.1.job.log.tgz is missing in pilot XML 4658520610 29/2/2020 22:08 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:9be6b52b-30b0-4352-9f79-eb4a59bc840e_50573.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 9be6b52b-30b0-4352-9f79-eb4a59bc840e_50573.1.job.log.tgz is missing in pilot XML 4658532839 29/2/2020 22:38 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.32396b4d-2926-44ef-b7c3-03393f299c1f_58083.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658549466 29/2/2020 23:02 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.270fe1e1-03be-4adb-b604-b62993f8d4fb_31725.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658582197 29/2/2020 23:48 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.564a5262-a4b8-448a-b21e-45f97de3942d_32733.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658596044 1/3/2020 0:14 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d8f6ca05-bb5d-430b-ac1e-4528f1d80c64_29486.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658607736 1/3/2020 0:41 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:644a95cd-01a7-4a9a-a581-d5b684edb3cd_8999.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 644a95cd-01a7-4a9a-a581-d5b684edb3cd_8999.1.job.log.tgz is missing in pilot XML 4658621026 1/3/2020 1:08 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.feed491d-643a-4fb0-8b86-baf9cb37b5f2_58121.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658633845 1/3/2020 1:35 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:081afacf-ceff-4dcf-9bb2-a42bd40d1b5c_63116.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 081afacf-ceff-4dcf-9bb2-a42bd40d1b5c_63116.1.job.log.tgz is missing in pilot XML 4658643925 1/3/2020 1:57 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.80f47e0b-397b-4c44-8da0-31a5a18c4354_99915.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658657228 1/3/2020 2:24 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.f4ad4e8c-2f07-44fa-922d-7bf3c23f0caf_48584.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658671932 1/3/2020 2:46 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.ddca7073-3837-4c0a-b9ab-c4664e158a85_47929.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658688576 1/3/2020 3:27 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.5dfa6589-a760-46ef-962f-6b85a57dd928_19942.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658698677 1/3/2020 4:00 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:977c78f6-8e3e-4de2-acfb-213933719123_92121.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 977c78f6-8e3e-4de2-acfb-213933719123_92121.1.job.log.tgz is missing in pilot XML 4658712312 1/3/2020 4:22 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.a550f46c-8f58-4154-87c8-68b4bee78eca_29561.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658719144 1/3/2020 4:48 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.f16de14d-041b-43ef-b0d6-99f0632e4ee8_39157.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658725295 1/3/2020 5:16 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.e5eaaf88-5901-4903-bfc0-5b85c77031ea_13252.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658735583 1/3/2020 5:47 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.d98234a1-c5eb-4c35-9eea-2ff1e3be4798_49125.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658752834 1/3/2020 6:40 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1d841c27-e3e5-45df-8790-f800c208e227_16226.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658781016 1/3/2020 8:13 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.87e95a88-13c5-43d9-bc72-f0e43c5e3bf7_90962.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658791560 1/3/2020 8:44 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.5191c244-0c68-4db3-a925-c56ae4b4c0c7_22213.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658802942 1/3/2020 9:27 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.cd08be36-9821-459b-888e-deced803cd59_35439.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658814159 1/3/2020 9:51 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.413279b0-02bf-47b4-97ee-8cf99ca7a2fd_39039.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658826730 1/3/2020 10:32 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.f2ac6a30-ecf7-4442-bd52-a8e8f804ceb5_19119.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658871984 1/3/2020 12:17 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658909874 1/3/2020 13:51 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: user.gangarbt:72ac30a6-3301-4cad-9338-a8e8425abec1_24888.1.job.log.tgz to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 72ac30a6-3301-4cad-9338-a8e8425abec1_24888.1.job.log.tgz is missing in pilot XML 4658927129 1/3/2020 14:28 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:dc4556a7-58d9-4b83-b5f4-aecf213e9209_17358.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output dc4556a7-58d9-4b83-b5f4-aecf213e9209_17358.1.job.log.tgz is missing in pilot XML 4658946909 1/3/2020 15:24 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.4777b46b-238c-4185-88cb-3b99ff0d8631_54900.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658965041 1/3/2020 16:13 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:f4c4b8c8-d46c-4996-a068-22241160800b_48685.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output f4c4b8c8-d46c-4996-a068-22241160800b_48685.1.job.log.tgz is missing in pilot XML 4659050396 1/3/2020 18:16 compute-14-12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-14-12.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1a0ddcc6-2eb3-483d-9ef4-06bd2e387cf0_10350.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657601180 28/2/2020 19:19 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:user.gangarbt.user.gangarbt.hc20163935.tid952.ANALY_GOEGRID.19.0a43575b-5095-4e1a-ba1f-1d5c27759b54_39614.hist-output.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools 4657609579 28/2/2020 19:33 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.78b13023-2b49-4557-a22b-fca5db51a4fa_83105.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657630717 28/2/2020 19:48 compute-19-7. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-7.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.14e80ee3-2419-43eb-9f72-517359f751f8_27889.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657645252 28/2/2020 20:04 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.c65e5b8e-2b63-4064-ab5f-62851dae1669_348.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657659862 28/2/2020 20:21 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.105b4782-41e7-489e-8421-24f378c5100f_91739.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657672840 28/2/2020 20:41 compute-19-12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.7a0ee713-37d4-43c6-9ada-5e9b1ded5299_84249.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657688955 28/2/2020 20:57 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.ec44b6ed-2335-4e3a-9ceb-2aec5c8ac52b_94100.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657695879 28/2/2020 21:10 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.5b096c91-2b37-432b-83eb-3c1b083dbad1_19702.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657700707 28/2/2020 21:19 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.a84a35d9-03b4-4203-9d6f-45c1abe88e10_57113.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657707453 28/2/2020 21:33 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.b8959e8d-0de9-48cd-a451-8f480c4b0afe_72257.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657717974 28/2/2020 21:47 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.cac4bba9-7540-48b1-932f-1ee508de2979_48762.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657727050 28/2/2020 22:05 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.151db3c9-5046-452b-97b3-b83b660676d9_28776.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657741123 28/2/2020 22:22 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.b046e6c5-03a4-40ac-a2d0-16d0c8993789_89929.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657751841 28/2/2020 22:39 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.568115ad-5a89-475f-8bfe-0abc896d69c2_45123.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657758559 28/2/2020 22:53 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.cb458be9-932d-4efa-94df-42d777bbbdf8_26314.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657769657 28/2/2020 23:24 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.fd5a0892-2764-4fa7-ba7c-9afcc26647b3_29554.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657793529 28/2/2020 23:39 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.c299b4ff-6e95-4e0f-b2f9-e3ff59058133_75162.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657802767 28/2/2020 23:54 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.de5ef9fb-965a-4036-b1c0-b602d20ea81e_78862.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657819464 29/2/2020 0:15 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.4ab58f7d-8866-4f88-b526-94709b05f75f_18832.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657830091 29/2/2020 0:32 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.7c13d1f3-c40c-4963-84c8-5b069c3e4bcd_32547.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657841996 29/2/2020 0:48 compute-17-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.c8e1affd-1877-46dd-a7ef-fa8bd469aaae_7678.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657851107 29/2/2020 1:07 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.5e4af271-9771-41f1-a4db-ba57bb6dbf88_12928.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657860684 29/2/2020 1:24 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.5b3f4456-4249-4541-81b2-e886d7e1dced_50510.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657873681 29/2/2020 1:44 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.b54b586f-6401-4ed6-b5f0-5a3575502548_38849.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657885246 29/2/2020 2:05 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.f42cafd9-634f-417b-b656-d43986a19e3d_83405.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657891455 29/2/2020 2:27 compute-17-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.8d3eab46-b098-4843-a877-f80f6b4eaa94_31004.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657905465 29/2/2020 2:46 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.b4593765-63bf-4171-8dd6-d52bb714c355_77587.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657928390 29/2/2020 3:21 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.1038cc95-4d03-42a7-a443-1436c054bf19_23064.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657940271 29/2/2020 3:40 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.b748a67f-cf3e-4aa3-b6fc-fe4e5960a28c_78791.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657954281 29/2/2020 4:00 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.8d612d05-060d-49c0-9745-4b4b2e8030fe_92377.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657962597 29/2/2020 4:16 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.5353f998-0ee1-480e-994a-5b3857b8d701_96715.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657973783 29/2/2020 4:43 compute-19-14 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-14.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.11cead6f-54cd-4b7f-b6a0-73c62f896368_98064.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657984709 29/2/2020 4:59 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.d6fe4fc0-5fcf-4453-924b-26d334c8b04c_49268.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657999197 29/2/2020 5:17 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.08f49cea-d246-4f45-8a5c-dcfefe9a6186_38491.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658006391 29/2/2020 5:33 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.220b5963-9fca-4ac9-8f7f-7524555b3f1b_69668.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658014151 29/2/2020 6:02 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.f5365e76-bc64-4e5b-a146-5e275be45293_16302.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658024614 29/2/2020 6:26 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.0b48755a-c92d-4c7f-9748-ae2a3d1569b3_91899.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658031450 29/2/2020 6:42 compute-19-10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-10.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.d8511a6f-2446-40e6-83c5-90c75f83d316_3273.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658040102 29/2/2020 7:02 compute-19-10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-10.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.7e9d3d8e-69bf-41d4-8b12-6ea7be22dd6e_56387.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658052374 29/2/2020 7:17 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.6a85c9e4-7ea2-4728-8362-afebc40fdcb9_90247.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658066745 29/2/2020 7:37 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.548ad193-b194-4f3e-8ca0-d2f701b0d345_70649.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658085044 29/2/2020 7:51 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.cb346b54-6255-488c-be9d-b0f2708bec56_71979.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658094964 29/2/2020 8:11 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.d8724f5c-50f9-4cb6-b899-24eb49c573d7_17484.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658103711 29/2/2020 8:29 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.7519aae3-9ffc-42d1-8c00-d1364e75f29c_95994.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658111827 29/2/2020 8:48 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.22bc5272-e969-4085-ad03-681a1a829aac_47464.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658119876 29/2/2020 9:06 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.1ae97707-2aa5-4140-bad8-2252793e7fa6_51425.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658125784 29/2/2020 9:27 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.25f6c690-aac4-4e66-b8fb-8079e7dff45c_80342.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658136739 29/2/2020 9:53 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.39dbd0b2-52d3-48b9-b4f6-da7bb350d659_31778.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658146753 29/2/2020 10:19 compute-19-12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.727c542d-40d5-4782-9985-c0d0fe148964_28392.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658155426 29/2/2020 10:45 compute-19-12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.69cb17b8-1411-4763-ac9e-5fc64522d228_61026.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658174359 29/2/2020 11:22 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.58fa43d9-5346-4a1f-893a-2e205c1abf0d_12727.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658191334 29/2/2020 11:46 compute-19-13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-13.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.5e279ada-4c01-43bb-a14c-2d616b2f0ec7_87135.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658203035 29/2/2020 12:10 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.9ee0d904-a0ed-4807-90f5-b2536dac639b_35295.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658213632 29/2/2020 12:33 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: user.gangarbt:output.1.f30e32ca-fa81-4bf5-b17f-b4380da73f9a_73182.pool.root to GOEGRID_SCRATCHDISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658232863 29/2/2020 13:10 compute-19-2. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-2.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.350beb80-1038-439c-81b3-a078f187fbc6_76786.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658252407 29/2/2020 13:46 compute-19-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-11.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.6a9b7b25-98c4-465d-be90-95c7ec457590_80233.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658274432 29/2/2020 14:26 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4658290435 29/2/2020 14:58 compute-19-15 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-15.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4658318056 29/2/2020 15:42 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.faaed415-894d-45f7-88c9-648ebc967842_83499.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658333411 29/2/2020 16:06 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.52fff6fd-000b-4985-b54a-ed76b37c9d18_70253.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658346520 29/2/2020 16:44 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.a045b82b-9ff5-447c-8490-181c1d8b8daa_50779.pool.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658365776 29/2/2020 17:52 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.0db2fc90-78dd-49c3-a141-3f8c1c5c26af_30984.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657080886 28/2/2020 6:15 compute-19-8. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-8.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4657093752 28/2/2020 6:36 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656148833 27/2/2020 8:37 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: user.gangarbt:user.gangarbt.user.gangarbt.hc20163592.tid952.ANALY_GOEGRID.29.b1b4a626-d9ff-4cce-b142-8df75007083a_46431.hist-output.root to GOEGRID_SCRATCHDISK, GOEGRID_SCRATCHDISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools 4656159315 27/2/2020 8:51 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656168712 27/2/2020 9:25 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656198057 27/2/2020 9:48 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656207673 27/2/2020 10:10 compute-19-6. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-6.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656215900 27/2/2020 10:34 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656228726 27/2/2020 10:53 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656237970 27/2/2020 11:12 compute-11-11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-11-11.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4656250319 27/2/2020 11:37 compute-19-5. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from GOEGRID_DATADISK, Error on XrdCl:CopyProcess:Run(): [FATAL] Connection error',)]:failed to transfer files using copytools=[u'rucio'] 4655950978 27/2/2020 2:14 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local 4655957546 27/2/2020 2:25 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local 4655964862 27/2/2020 2:44 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local 4655993619 27/2/2020 3:29 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local 4656005419 27/2/2020 3:41 compute-19-4. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-4.local 4656012798 27/2/2020 3:59 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local 4649412078 18/2/2020 20:00 compute-19-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-9.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:user.gangarbt.99241777-5669-4f44-9593-48ec55bc984f_57883.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output user.gangarbt.99241777-5669-4f44-9593-48ec55bc984f_57883.job.log.tgz is missing in pilot XML 4649471832 18/2/2020 21:17 compute-7-6.l mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-7-6.local jobDispatcher:::100 lost heartbeat : 2020-02-18 21:17:52 4649621029 19/2/2020 0:29 compute-8-16. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-8-16.local 4648367901 17/2/2020 21:33 compute-11-14 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-11-14.local 4648250270 17/2/2020 18:21 compute-11-14 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-11-14.local 4646489930 14/2/2020 22:58 compute-17-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-17-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c7815806-60c3-4015-9fe1-f9a22542305c_44471.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4646555842 15/2/2020 1:01 compute-19-1. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-1.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:8badfb10-8c24-4337-b06c-affdb1e189c1_75157.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 8badfb10-8c24-4337-b06c-affdb1e189c1_75157.1.job.log.tgz is missing in pilot XML 4644502673 12/2/2020 19:43 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c41cf4b8-7de7-42f3-ab29-543df7ef011c_75451.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4642818956 11/2/2020 2:53 compute-14-10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-14-10.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.3c8dd4f8-2d51-4c88-8147-c5263c173b1e_40864.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4642918643 11/2/2020 5:23 compute-15-16 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-15-16.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:ebe59ccd-ba14-4b8d-98a9-051652cfb6b6_8324.1.job.log.tgz to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output ebe59ccd-ba14-4b8d-98a9-051652cfb6b6_8324.1.job.log.tgz is missing in pilot XML 4642990411 11/2/2020 7:02 compute-19-10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-10.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1ea4b01c-fe60-4bb3-b1d6-6de309638a5c_63834.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4642052823 10/2/2020 16:14 compute-18-1. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-18-1.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.3bd3d7b0-e560-4bbc-9fe8-4e765c1869e5_87932.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4640873134 9/2/2020 13:05 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4639840086 8/2/2020 8:21 compute-19-3. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.1e7173e2-a3fa-417f-9565-b88a1be9cd0b_98873.pool.root to GOEGRID_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4636352677 5/2/2020 4:18 compute-19-9. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: compute-19-9.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found GoeGrid ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4659011250 1/3/2020 18:08 compute-17-14 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-14.local pilot:::1152 File transfer timed out during stage-out: hc_test:ecbf13e7-d043-48fd-a4eb-95752946130a_62379.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output ecbf13e7-d043-48fd-a4eb-95752946130a_62379.1.job.log.tgz is missing in pilot XML 4659078980 1/3/2020 19:41 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4659158319 1/3/2020 21:32 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.42ebd4a4-c376-48ed-9287-dd3f4e977acb_81633.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659183755 1/3/2020 22:12 compute-19-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-10.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.6ac93f4f-95a0-4100-8626-b28b50b557f2_23600.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658374630 29/2/2020 18:06 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658448407 29/2/2020 19:52 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1324 Service not available at the moment: hc_test:output.1.e293f831-ce33-4323-90d2-61a06d251777_57158.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] TTL exceeded',)]:failed to transfer files using copytools=[u'rucio'] 4658473584 29/2/2020 20:41 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.eeb4c63d-64ab-4066-8650-65ff2e5ddbbb_27595.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658505807 29/2/2020 21:42 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:3e5e4fbd-7108-4f86-aeb6-97e9d498703d_69812.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 3e5e4fbd-7108-4f86-aeb6-97e9d498703d_69812.1.job.log.tgz is missing in pilot XML 4658535122 29/2/2020 22:45 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658604405 1/3/2020 0:34 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.65ff01b9-b50d-4fde-8137-b35d78e14f21_71136.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658629182 1/3/2020 1:26 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:71390f4b-d606-4bfa-a16a-cc5311579ccc_83877.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 71390f4b-d606-4bfa-a16a-cc5311579ccc_83877.1.job.log.tgz is missing in pilot XML 4658652003 1/3/2020 2:21 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:86ffef60-c656-4426-8017-94e5c43c134f_14776.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 86ffef60-c656-4426-8017-94e5c43c134f_14776.1.job.log.tgz is missing in pilot XML 4658691367 1/3/2020 3:34 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.f0818974-af27-446c-825f-0bdd61b1f8a4_3779.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658712886 1/3/2020 4:33 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.37b8d571-f0ee-4376-9e23-b38d81873c26_23016.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658746561 1/3/2020 6:21 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: hc_test:output.1.888bd8e4-b6e3-4772-82ea-afac5e3cf947_82240.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658770734 1/3/2020 7:48 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.0a68698c-df00-4cf9-8e63-6070c4177672_81029.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658787430 1/3/2020 8:36 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:6d2a20b3-901c-4f25-bdec-6ce0c853ac3e_52269.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 6d2a20b3-901c-4f25-bdec-6ce0c853ac3e_52269.1.job.log.tgz is missing in pilot XML 4658808100 1/3/2020 9:39 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:f8d657f0-f728-48f8-aa04-e99cabb3388a_29675.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output f8d657f0-f728-48f8-aa04-e99cabb3388a_29675.1.job.log.tgz is missing in pilot XML 4658832353 1/3/2020 10:40 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:e6f34d80-3e1a-4c00-846e-63e8f37a4c88_74012.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output e6f34d80-3e1a-4c00-846e-63e8f37a4c88_74012.1.job.log.tgz is missing in pilot XML 4658855000 1/3/2020 11:30 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658897278 1/3/2020 13:25 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: hc_test:8843ac32-8f6f-443f-83de-259c8ad5dabe_87533.1.job.log.tgz to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 8843ac32-8f6f-443f-83de-259c8ad5dabe_87533.1.job.log.tgz is missing in pilot XML 4658922348 1/3/2020 14:18 compute-19-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-7.local pilot:::1324 Service not available at the moment: hc_test:00a9c38f-120b-480f-8178-56ee4ee2f94f_96076.1.job.log.tgz to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 00a9c38f-120b-480f-8178-56ee4ee2f94f_96076.1.job.log.tgz is missing in pilot XML 4658938882 1/3/2020 15:10 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:52021454-5c98-428f-8ce0-ed604be09aa8_47216.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 52021454-5c98-428f-8ce0-ed604be09aa8_47216.1.job.log.tgz is missing in pilot XML 4657601315 28/2/2020 19:22 compute-17-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-9.local pilot:::1137 Failed to stage-out file: hc_test:Hits.hc_20163938.GoeGrid.f9509aac-d4a7-41e2-8087-81077f9a1bf4_13676.HITS.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657634125 28/2/2020 19:56 compute-17-13 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-13.local pilot:::1137 Failed to stage-out file: hc_test:output.1.450b2c77-b472-48a3-82c1-e28e1cc52e15_17176.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657659233 28/2/2020 20:20 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.d0cc35cb-41d8-4f61-9891-090b75b3c421_23514.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657686482 28/2/2020 20:52 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: hc_test:output.1.0741ee5e-7e7b-4873-995d-86fc725fde88_2561.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657704483 28/2/2020 21:29 compute-19-14 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-14.local pilot:::1137 Failed to stage-out file: hc_test:output.1.dcef3160-d04b-4b01-92d3-d073fd09ac84_16732.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657724546 28/2/2020 21:58 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local pilot:::1137 Failed to stage-out file: hc_test:output.1.36375796-c39c-4482-9386-6dd849ce2abd_61311.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657752315 28/2/2020 22:44 compute-17-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-6.local pilot:::1137 Failed to stage-out file: hc_test:output.1.1f72f880-a722-4527-9c4e-a8dd6a31898b_42312.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657765562 28/2/2020 23:12 compute-17-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.2e173437-d456-4975-b687-806e90ff939f_32811.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657794808 28/2/2020 23:42 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: hc_test:output.1.3c13bcf7-ac3b-4ad8-9555-a167a527916b_58357.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657828733 29/2/2020 0:28 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.08dbb4d3-1e33-471a-a3ec-8c6e9cce125c_35164.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657851338 29/2/2020 1:09 compute-17-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-4.local pilot:::1137 Failed to stage-out file: hc_test:output.1.5d0ddd55-ec4d-4864-aae4-5fa52bdc962f_94018.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657868503 29/2/2020 1:36 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.5d48a8aa-cf5e-49dc-b629-190a6100a1fe_4226.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657889188 29/2/2020 2:21 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.500b471e-66f6-45f9-9438-fe48c30f5437_5225.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657916473 29/2/2020 3:06 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: hc_test:output.1.da39d687-cb02-4412-ac2b-af65c8fba95d_12415.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657943651 29/2/2020 3:58 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658014886 29/2/2020 6:02 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.27b7991e-4231-4d7d-a31e-1d2590a3bce9_23290.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658031560 29/2/2020 6:42 compute-19-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-10.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658102284 29/2/2020 8:26 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.5495f656-7bf6-478e-823f-095630d21337_96119.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658121438 29/2/2020 9:14 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: hc_test:output.1.28f6c1bf-2b2f-472d-b7b2-c4f2d0ac4509_86384.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658140465 29/2/2020 10:00 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: hc_test:output.1.7442b1c7-c536-40eb-9f6c-f2a2126a610e_21473.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658159991 29/2/2020 10:56 compute-19-12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: hc_test:output.1.158d76be-d0a3-462f-9354-a6899b889a70_9551.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658183178 29/2/2020 11:38 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.ce965626-86d6-4797-ae91-f3296f23133f_33733.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658222088 29/2/2020 12:51 compute-19-13 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-13.local pilot:::1324 Service not available at the moment: hc_test:output.1.eff09277-65e2-400d-a5e7-668d2fd7dadb_41786.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658246143 29/2/2020 13:42 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: hc_test:output.1.d8121ffe-6d9f-4165-bffe-12f682a22b7b_44349.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658280162 29/2/2020 14:35 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000031.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658294380 29/2/2020 15:08 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000031.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658320151 29/2/2020 15:42 compute-17-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-9.local pilot:::1137 Failed to stage-out file: hc_test:output.1.ca126679-5c23-4f00-8406-d0e9afd57418_96144.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658340377 29/2/2020 16:21 compute-17-1. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-1.local pilot:::1137 Failed to stage-out file: hc_test:output.1.ed6d022f-900e-4629-9154-c3d802950e57_61349.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657208755 28/2/2020 10:00 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000040.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4654294277 25/2/2020 9:36 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4649528287 18/2/2020 22:50 compute-13-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-13-5.local pilot:::1137 Failed to stage-out file: hc_test:2e6b8486-50f0-4521-8319-1a05bf5e7f1b_6388.1.job.log.tgz to GOEGRID_DATADISK, no error information passed (http status code: 503 ('service_unavailable', 'unavailable'))",)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 2e6b8486-50f0-4521-8319-1a05bf5e7f1b_6388.1.job.log.tgz is missing in pilot XML 4648341490 17/2/2020 20:54 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648481112 18/2/2020 1:07 compute-19-2. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-2.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648665583 18/2/2020 4:50 compute-13-13 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-13-13.local taskBuffer:::300 The worker was finished while the job was running : None 4648786157 18/2/2020 8:38 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648858529 18/2/2020 10:35 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647622459 16/2/2020 17:46 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647659950 16/2/2020 19:37 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647703636 16/2/2020 21:37 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647764221 17/2/2020 0:18 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647877654 17/2/2020 5:11 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647942913 17/2/2020 8:33 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648059557 17/2/2020 12:52 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648154077 17/2/2020 15:50 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647377925 16/2/2020 9:21 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646768322 15/2/2020 10:13 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646873252 15/2/2020 13:05 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4645305819 13/2/2020 17:48 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4644355661 12/2/2020 17:32 compute-19-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-9.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.ebb2cdde-1d6e-4fac-986a-77fcc9c2c4ae_36933.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644734801 13/2/2020 0:08 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4644974932 13/2/2020 7:43 compute-19-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-7.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4643915598 12/2/2020 8:54 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4642655221 11/2/2020 0:44 compute-19-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4642804044 11/2/2020 2:48 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4641220249 9/2/2020 21:49 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4641441303 10/2/2020 6:15 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4640264552 8/2/2020 18:15 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4640390690 8/2/2020 21:53 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4640602696 9/2/2020 3:44 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4638752325 7/2/2020 16:43 compute-19-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-9.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4639262591 8/2/2020 0:02 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4640066891 8/2/2020 12:10 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637830959 6/2/2020 15:40 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4638138883 7/2/2020 1:54 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4638469074 7/2/2020 11:19 compute-19-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-10.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637129904 5/2/2020 20:36 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637672915 6/2/2020 10:36 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635317221 4/2/2020 13:26 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635696791 4/2/2020 19:27 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4636208177 5/2/2020 2:37 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4636374328 5/2/2020 4:47 compute-19-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4636722349 5/2/2020 11:19 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4634063665 3/2/2020 14:51 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4632714762 2/2/2020 15:00 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4631449437 1/2/2020 16:38 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658980441 1/3/2020 16:47 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4659095083 1/3/2020 20:27 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.9d50ae7c-6ce4-4742-b912-dccec7d9bc5d_43091.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659170361 1/3/2020 21:52 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658413477 29/2/2020 19:06 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658477164 29/2/2020 20:56 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.fff81635-9b2d-431e-a174-b4f0e359b5d8_75562.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658506920 29/2/2020 21:50 compute-19-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-10.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.084f237a-d951-4f89-b8e2-e96d654c05c0_20149.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658536022 29/2/2020 23:03 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658611403 1/3/2020 0:51 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: hc_test:dc3becd3-4703-4429-8593-a2963470e536_97386.1.job.log.tgz to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output dc3becd3-4703-4429-8593-a2963470e536_97386.1.job.log.tgz is missing in pilot XML 4658631739 1/3/2020 1:42 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658689163 1/3/2020 3:32 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.9c8f4974-f5d6-4730-9df5-92d25ececda2_81978.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658711691 1/3/2020 4:29 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.c0f162d6-5bc4-48a0-9731-5ba1b2b2deb5_19859.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658739928 1/3/2020 6:00 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.d983fa9d-e0ec-4ccb-b69f-53f9fd363067_734.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658752660 1/3/2020 6:44 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658761473 1/3/2020 7:16 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: hc_test:output.1.f745c40b-7b18-43b3-8854-325df7c4f838_76864.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658773960 1/3/2020 8:12 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.3e424b78-ba4c-4a03-b7e4-8d09766616ee_10613.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658809382 1/3/2020 9:41 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.1bbbb93a-0241-462b-8732-24c8778a9b12_38822.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658830725 1/3/2020 10:37 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658844198 1/3/2020 11:19 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.7c8d44ed-a287-4ca0-be67-402392bef703_54365.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658868208 1/3/2020 12:12 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: hc_test:ef998e9b-c63c-4a4d-ae1b-d34b63288f40_32558.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output ef998e9b-c63c-4a4d-ae1b-d34b63288f40_32558.1.job.log.tgz is missing in pilot XML 4658890630 1/3/2020 13:06 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658936297 1/3/2020 14:57 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658947571 1/3/2020 15:32 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: hc_test:bf49dc2b-0aac-4b85-b4f3-6449ca68bf99_37500.1.job.log.tgz to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output bf49dc2b-0aac-4b85-b4f3-6449ca68bf99_37500.1.job.log.tgz is missing in pilot XML 4657600638 28/2/2020 19:19 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:Hits.hc_20163937.GoeGrid.9f7d292f-65e7-48a3-885f-41280674dbc2_5052.HITS.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657636863 28/2/2020 19:56 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1137 Failed to stage-out file: hc_test:output.1.44157b35-fee7-4fc7-bfd7-08f77dbafb76_39458.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657668879 28/2/2020 20:29 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.02730044-d4b6-427a-97f4-b4e713ef0063_97193.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657694161 28/2/2020 21:05 compute-19-12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-12.local pilot:::1137 Failed to stage-out file: hc_test:output.1.b49aede6-1a52-454c-992d-ad4fb4c06e67_79196.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657707245 28/2/2020 21:31 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.59014e6c-b9df-455e-9e37-52b140c43b80_76930.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657734721 28/2/2020 22:13 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4657804676 28/2/2020 23:57 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.0b62e468-e5a9-48f9-85a8-ee18b6ff0bbe_42671.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657831947 29/2/2020 0:37 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4657895771 29/2/2020 2:29 compute-17-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.3ec3ad4a-90df-482c-85a3-2af740a23928_27330.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657913042 29/2/2020 2:57 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.5e66fca8-ed8a-4707-82b7-d2c17e764fa2_73530.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657938970 29/2/2020 3:39 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658002822 29/2/2020 5:43 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.c7d50e16-9d28-4001-9e1f-7b78e23be398_38125.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658022109 29/2/2020 6:19 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658095892 29/2/2020 8:12 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.a9fcc914-456b-49dd-8e29-0dc84dd1e363_9209.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658113958 29/2/2020 8:53 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.b812f3fb-3b88-4d5b-97b5-165134ec1e92_12572.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658128060 29/2/2020 9:39 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658180709 29/2/2020 11:22 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658233604 29/2/2020 13:15 compute-17-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-4.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.558d1e94-5df6-4366-92e8-d76b6ab9f992_4356.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=321 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658259615 29/2/2020 13:55 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: hc_test:output.1.ef4dbebc-1259-49e6-8de0-78a35bf090ca_44854.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] TTL exceeded',)]:failed to transfer files using copytools=[u'rucio'] 4658286040 29/2/2020 14:49 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658303784 29/2/2020 15:23 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658322013 29/2/2020 15:56 compute-17-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-10.local pilot:::1137 Failed to stage-out file: hc_test:output.1.d43d6ed6-f3ed-449f-b20d-85ff8ae6c69f_72501.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658346031 29/2/2020 17:51 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656980897 28/2/2020 4:11 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656996240 28/2/2020 4:35 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657013308 28/2/2020 5:11 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657043103 28/2/2020 5:30 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656104391 27/2/2020 7:02 compute-7-12. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-7-12.local pilot:::1137 Failed to stage-out file: hc_test:output.1.f86f617e-e475-4df9-9771-6f0d959a952e_56152.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4656121479 27/2/2020 7:33 compute-10-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-10-10.local pilot:::1137 Failed to stage-out file: hc_test:output.1.f8d03306-5a0b-4a2c-b51b-9680a1c31d5e_26450.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4656135608 27/2/2020 8:06 compute-13-13 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-13-13.local pilot:::1137 Failed to stage-out file: hc_test:output.1.5b9086c7-209f-4046-a6c5-3552c9f01496_26129.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4656149267 27/2/2020 8:29 compute-16-14 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-16-14.local pilot:::1137 Failed to stage-out file: hc_test:output.1.bc6f884a-2c06-403d-b816-8160fc2a6c5d_23787.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4656159034 27/2/2020 8:56 compute-14-15 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-14-15.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000033.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656170044 27/2/2020 9:16 compute-18-16 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-18-16.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000033.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656194594 27/2/2020 9:38 compute-13-1. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-13-1.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000033.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656203126 27/2/2020 9:56 compute-8-47. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-8-47.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000033.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656210639 27/2/2020 10:19 compute-8-22. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-8-22.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000033.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656220577 27/2/2020 10:38 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000033.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4651142610 20/2/2020 22:51 compute-14-15 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-14-15.local pilot:::1244 No release candidates found 4648412472 17/2/2020 22:55 compute-19-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-9.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648588891 18/2/2020 3:18 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648682463 18/2/2020 5:36 compute-19-2. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-2.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647734396 16/2/2020 23:29 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647793440 17/2/2020 1:57 compute-19-14 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-14.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647885251 17/2/2020 5:54 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647928306 17/2/2020 7:46 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647971886 17/2/2020 9:48 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648060659 17/2/2020 12:55 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found jobDispatcher:::100 lost heartbeat : 2020-02-17 12:56:09 sup:::2 worker already cancelled at 2020-02-17 14:30:08 with Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647167118 15/2/2020 23:29 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647226738 16/2/2020 2:19 compute-19-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647326309 16/2/2020 7:05 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647382531 16/2/2020 9:26 compute-19-15 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-15.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647428602 16/2/2020 11:45 compute-19-12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-12.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647551293 16/2/2020 15:54 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646554550 15/2/2020 1:15 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646681509 15/2/2020 6:40 compute-19-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646808980 15/2/2020 11:50 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4645118808 13/2/2020 12:49 compute-8-43. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-8-43.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4643899909 12/2/2020 8:21 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4643956956 12/2/2020 10:12 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4644284559 12/2/2020 16:37 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4644569089 12/2/2020 20:47 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4641438807 10/2/2020 5:24 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4641550870 10/2/2020 9:34 compute-17-13 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-13.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4641705670 10/2/2020 11:29 compute-19-15 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-15.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4642610604 11/2/2020 0:23 compute-18-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-18-7.local taskBuffer:::300 The worker was finished while the job was running : None 4640635543 9/2/2020 4:28 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4640687015 9/2/2020 6:28 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4641082677 9/2/2020 18:54 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4640156847 8/2/2020 15:32 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4640518966 9/2/2020 1:53 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4638341767 7/2/2020 8:01 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4638419042 7/2/2020 10:01 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637703176 6/2/2020 11:18 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637770489 6/2/2020 13:19 compute-18-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-18-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4636386029 5/2/2020 4:49 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4636640636 5/2/2020 8:59 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637152309 5/2/2020 21:26 compute-19-13 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-13.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635002910 4/2/2020 10:01 compute-17-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-10.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/tmp_local/54041093.pbs-goegrid.lo 4635251534 4/2/2020 13:28 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/tmp_local/54043667.pbs-goegrid.lo 4635413382 4/2/2020 15:12 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/tmp_local/54044663.pbs-goegrid.lo ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635549678 4/2/2020 16:56 compute-19-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-10.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635659102 4/2/2020 19:11 compute-19-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-7.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',), error code: 1099, message: Failed to stage-in file stacktrace: Traceback (most recent call last): File "/tmp_local/54047416.pbs-goegrid.lo 4635792109 4/2/2020 20:50 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635924402 4/2/2020 22:53 compute-19-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-7.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4632412177 2/2/2020 8:37 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4630669688 1/2/2020 6:32 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4630819437 1/2/2020 8:21 compute-19-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4631304873 1/2/2020 14:22 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4631587461 1/2/2020 18:42 compute-19-16 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-16.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4631769078 1/2/2020 20:38 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4631891759 1/2/2020 23:02 compute-19-15 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-15.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4632106758 2/2/2020 3:35 compute-15-1. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-15-1.local jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 03:49:34 sup:::2 worker already finished at 2020-02-02 04:02:08 with None 4658983856 1/3/2020 16:58 compute-19-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-9.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000030.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658992157 1/3/2020 17:39 compute-15-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-15-7.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000030.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659011140 1/3/2020 18:08 compute-16-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-16-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.09e6d84f-1b93-4d9a-8e37-ddb42af3730f_93560.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=315 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659073596 1/3/2020 19:46 compute-19-10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-10.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4659158647 1/3/2020 21:37 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000030.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4659174750 1/3/2020 22:05 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.ec95559f-f78b-48c0-b69c-a1276059b386_87705.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=315 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658414691 29/2/2020 19:18 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20164157.GoeGrid.ac3728d9-1055-49d3-9e62-0d99e3825aa9_35551.HITS.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658451319 29/2/2020 20:01 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.538d4297-3073-4d24-b85c-e4ba267e9733_8035.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658472252 29/2/2020 20:41 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.7dbaa4e8-055a-4c76-a8ea-0d0feee5686c_29379.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658492600 29/2/2020 21:16 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: hc_test:2f1b4a39-8355-49fd-857e-08c87fd2dd19_19459.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 2f1b4a39-8355-49fd-857e-08c87fd2dd19_19459.1.job.log.tgz is missing in pilot XML 4658514895 29/2/2020 21:56 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.393c6483-cae2-41b4-85d7-15dca4f39ff5_85230.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658531998 29/2/2020 22:44 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.e076eeb4-909d-43c3-bb7d-5df0725c50b9_51014.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658555640 29/2/2020 23:23 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.d5804c41-9267-4cea-9192-c1472fb2ed3c_89220.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658589355 1/3/2020 0:21 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.9a60141f-36b0-485e-abd4-e306bfef737b_31819.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658619859 1/3/2020 1:08 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.d27e5e98-2a00-4e72-974d-89d7d019c60d_56496.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658636407 1/3/2020 2:03 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:c770799d-cadf-4e98-becc-5fef87689b67_87426.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output c770799d-cadf-4e98-becc-5fef87689b67_87426.1.job.log.tgz is missing in pilot XML 4658662863 1/3/2020 2:44 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.c83079e0-ba9f-4913-8ab9-24037ed3021b_56940.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658688450 1/3/2020 3:26 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:e3d7aac3-0105-46cd-b9dc-4855ca079c5c_74979.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output e3d7aac3-0105-46cd-b9dc-4855ca079c5c_74979.1.job.log.tgz is missing in pilot XML 4658701585 1/3/2020 4:01 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.39cbb0f7-2c3c-4fbd-a2f2-21b02d6f8411_49239.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658743775 1/3/2020 6:09 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.5a4eedef-f9b7-4671-b9a7-ab950c58517d_31054.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658764084 1/3/2020 7:23 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.8ecec7d6-3f49-47f4-928a-4586ffb49168_30285.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658773411 1/3/2020 8:06 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.e8bd7d4c-01c4-4721-bb34-6d5bd22e2972_83496.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658791648 1/3/2020 8:50 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1324 Service not available at the moment: hc_test:output.1.d65842a5-b4ef-4885-8bd4-560199e79c08_77104.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658846078 1/3/2020 11:16 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:d9c25986-cca6-49ec-ae8d-fbb8f37b28cf_79166.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output d9c25986-cca6-49ec-ae8d-fbb8f37b28cf_79166.1.job.log.tgz is missing in pilot XML 4658900872 1/3/2020 13:32 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1324 Service not available at the moment: hc_test:output.1.a533dfd0-785d-4db3-806c-26d8a1030fab_49137.pool.root to GOEGRID_DATADISK, ((('stageOut: File existence verification failed with: The requested service is not available at the moment.\nDetails: An unknown exception occurred.\nDetails: Error reported from srm_ifce : 70 [SE] 4658920916 1/3/2020 14:18 compute-19-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-7.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.c8e70c2e-590a-437d-bf32-7587b448485b_47329.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658936233 1/3/2020 14:57 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.ecaa94af-893e-4acd-859f-406ea5cedfca_32097.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657600967 28/2/2020 19:22 compute-17-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-9.local pilot:::1137 Failed to stage-out file: hc_test:Hits.hc_20163934.GoeGrid.c320c9e8-8398-4394-96cf-199e32e1a603_32447.HITS.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657631470 28/2/2020 19:46 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.64bfa1eb-e396-4891-9e9d-87456a6b8ad7_308.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657658817 28/2/2020 20:20 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4657727653 28/2/2020 22:05 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4657801272 28/2/2020 23:51 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.03150f23-19e9-4a50-8602-2e3d3d3d07ea_91995.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657822447 29/2/2020 0:21 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.68c84a40-4891-4fe2-a61d-e003c799adb0_5942.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657841918 29/2/2020 0:47 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.1717e51a-070d-4723-9ad6-dbff8d4d933d_25599.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657858799 29/2/2020 1:22 compute-17-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-4.local pilot:::1137 Failed to stage-out file: hc_test:output.1.b74c4b97-e512-47a8-8b1f-af3bf32e2eb9_19347.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657875564 29/2/2020 1:50 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.4754aa73-efd4-478d-8bb5-b6cf91d99b9c_8926.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657893662 29/2/2020 2:22 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.5e6248e0-7d39-4869-879e-0bfe1922c31a_8561.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4657914749 29/2/2020 3:01 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4657982155 29/2/2020 4:53 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.b777f8c7-b744-4c12-bc38-8607e9e41855_28924.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658002509 29/2/2020 5:37 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.b5726bdb-2e2c-49e7-8aa5-bbb4ee5dc795_54229.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658020230 29/2/2020 6:10 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.3d493e3d-49f9-4b3b-850a-4f16b1b6eb2b_18774.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658033495 29/2/2020 7:02 compute-19-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-11.local pilot:::1137 Failed to stage-out file: hc_test:output.1.9b53c8da-cb67-4293-9d98-b56b74b5e5a7_89920.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658061497 29/2/2020 7:32 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1137 Failed to stage-out file: hc_test:output.1.6d3d4a34-ef0d-4508-9ebb-2660e2c91b1f_14522.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658094017 29/2/2020 8:06 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: hc_test:output.1.448c11f0-fc1a-4cec-bc5a-e5c04c89c6b8_69775.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658110272 29/2/2020 8:51 compute-19-6. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-6.local pilot:::1137 Failed to stage-out file: hc_test:output.1.8e74d0d1-414c-4b40-b545-b61d39f14921_90580.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658127574 29/2/2020 9:39 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1137 Failed to stage-out file: hc_test:output.1.5a69e97c-5b1e-43af-b836-e296f24797c8_44925.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658144189 29/2/2020 10:14 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658202208 29/2/2020 12:10 compute-19-12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-12.local pilot:::1324 Service not available at the moment: hc_test:output.1.a554ba8c-6964-4671-a934-1e07ce65c23e_25782.pool.root to GOEGRID_DATADISK, Error reported from srm_ifce : 70 [SE][Ls][SRM_INTERNAL_ERROR] Request to [>PnfsManager@local] timed out.',)]:failed to transfer files using copytools=[u'rucio'] 4658225348 29/2/2020 13:01 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4658284815 29/2/2020 14:48 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000028.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658302867 29/2/2020 15:23 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000028.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658327139 29/2/2020 16:05 compute-17-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-11.local pilot:::1137 Failed to stage-out file: hc_test:output.1.6eaa57ca-1950-44bf-acfd-0505a76f156a_52870.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4658345765 29/2/2020 17:02 compute-17-12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-12.local pilot:::1137 Failed to stage-out file: hc_test:output.1.456178d6-1437-4ebf-ab74-fd4a206b6ed9_78425.pool.root to GOEGRID_DATADISK, GOEGRID_DATADISK is blacklisted for writing. No actions have been taken',)]:failed to transfer files using copytools=[u'rucio'] 4656900988 28/2/2020 2:42 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000026.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656918250 28/2/2020 3:19 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000026.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656964166 28/2/2020 3:35 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000026.pool.root.1 from GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655232227 26/2/2020 11:23 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648668046 18/2/2020 4:56 compute-14-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-14-9.local pilot:::1346 Transform not found:/srv/container_script.sh: line 1: Sim_tf.py: command not found 4648671173 18/2/2020 5:05 compute-14-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-14-9.local taskBuffer:::300 The worker was finished while the job was starting : None 4648824863 18/2/2020 9:53 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648058073 17/2/2020 12:49 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648221451 17/2/2020 17:34 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4648389018 17/2/2020 22:27 compute-19-2. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-2.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647273959 16/2/2020 4:42 compute-17-1. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-1.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.983992a7-f927-4afb-89ef-6b68bf9bb2a3_53373.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647714252 16/2/2020 22:28 compute-19-4. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-4.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4646679061 15/2/2020 6:34 compute-18-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-18-9.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647049809 15/2/2020 17:42 compute-19-11 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-11.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4647176996 16/2/2020 0:22 compute-19-7. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-7.local pilot:::1201 Job killed by signal: SIGTERM 4646526419 15/2/2020 0:15 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.d8dc129a-28a7-4386-b4f0-511757019305_36879.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644908961 13/2/2020 5:28 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4645491627 13/2/2020 22:11 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4643927569 12/2/2020 9:06 compute-19-5. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-5.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4644540132 12/2/2020 20:23 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4641510456 10/2/2020 8:15 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4642227056 10/2/2020 19:20 compute-15-16 hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-15-16.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.7be3d871-80aa-4133-9fa4-ad25da58cfbe_17812.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4639636952 8/2/2020 5:29 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4639864914 8/2/2020 8:30 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4638373627 7/2/2020 8:25 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4638457652 7/2/2020 11:04 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637839075 6/2/2020 15:52 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637914881 6/2/2020 18:04 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:6eecd876-05ff-4542-8363-919aafc8dc1b_95699.1.job.log.tgz to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 6eecd876-05ff-4542-8363-919aafc8dc1b_95699.1.job.log.tgz is missing in pilot XML 4638007333 6/2/2020 20:40 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4636666794 5/2/2020 9:43 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4637019543 5/2/2020 18:04 compute-17-9. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-17-9.local taskBuffer:::300 The worker was finished while the job was running : None 4634877902 4/2/2020 7:49 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635060961 4/2/2020 10:37 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.7cd0d7ce-1047-45ef-affc-00bb094aabd1_94755.pool.root to GOEGRID_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635434318 4/2/2020 15:07 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4635716078 4/2/2020 19:34 compute-19-3. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-3.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found 4630605106 1/2/2020 5:25 compute-19-8. hc_test:mc15_13TeV.361106.PowhegP modificationHost: compute-19-8.local taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Job has been terminated (got SIGTERM) ; Worker canceled by harvester due to held too long or not found ====== HEPHY-UIBK HEPHY-UIBK ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4651088945 20/2/2020 21:20 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--hephy--1782--214634779719013-1 pilot:::1098 Not enough local space 4645275413 13/2/2020 16:48 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--hephy--1782--83012275680454-4 taskBuffer:::300 The worker was finished while the job was running : None 4644872138 13/2/2020 4:19 slot1_4@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@atlas-cern--hephy--1782--34709260371509-2 pilot:::1137 Failed to stage-out file: hc_test:741d0451-74ec-4bfe-91c3-76acda68f813_44479.1.job.log.tgz to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 741d0451-74ec-4bfe-91c3-76acda68f813_44479.1.job.log.tgz is missing in pilot XML 4645098341 13/2/2020 13:12 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--hephy--1782--165304206620403-3 pilot:::1137 Failed to stage-out file: hc_test:output.1.e5020dd7-a64e-4285-88f2-829bfb62486f_38692.pool.root to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4635317028 4/2/2020 13:50 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--hephy--1782--181949593163239-4 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635385888 4/2/2020 14:57 slot1_2@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@atlas-cern--hephy--1782--254198731771049-4 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-04 15:16:36 sup:::2 worker already finished at 2020-02-04 15:35:05 with None 4647471873 16/2/2020 13:19 slot1_6@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_6@atlas-cern--hephy--1782--63444985976258-5 pilot:::1137 Failed to stage-out file: hc_test:output.1.7f28fea4-a305-455a-817c-cda492e8b0f3_20374.pool.root to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4646793399 15/2/2020 11:00 slot1_3@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_3@atlas-cern--hephy--1782--201537352003445-3 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.bda9c186-72f0-4917-aab9-3dcdb0713edf_73859.pool.root to LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=319 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644860229 13/2/2020 4:19 slot1_5@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@atlas-cern--hephy--1782--34709260371509-2 pilot:::1137 Failed to stage-out file: hc_test:Hits.hc_20160894.HEPHY-UIBK.b791d609-5566-47a8-b8f7-808ffbb3a88a_21205.HITS.pool.root to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4635648923 4/2/2020 18:56 slot1_5@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@atlas-cern--hephy--1782--90059718644534-2 taskBuffer:::300 The worker was finished while the job was running : None 4658816129 1/3/2020 9:53 slot1_3@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_3@atlas-cern--hephy--1782--244001168376397-1 pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4655900421 27/2/2020 1:16 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--hephy--1782--193682513275096-2 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655928845 27/2/2020 1:35 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--hephy--1782--191186712773476-5 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655942362 27/2/2020 3:40 slot1_2@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@atlas-cern--hephy--1782--88939305466159-5 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644881807 13/2/2020 4:25 slot1_2@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@atlas-cern--hephy--1782--34709260371509-2 pilot:::1137 Failed to stage-out file: hc_test:Hits.hc_20160901.HEPHY-UIBK.b9abc5fd-07ab-4d8b-a9af-a819140a8ae0_74517.HITS.pool.root to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4643608481 11/2/2020 23:06 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--hephy--1782--218838811549932-2 exe:::65 Non-zero return code from EVNTtoHITS (1); Logfile error in log.EVNTtoHITS: "Segmentation fault: Event counter: 0; Run: unknown; Evt: unknown; Current algorithm: ISF_Kernel_FullG4; Current Function: unknown" 4635466531 4/2/2020 15:30 slot1_2@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@atlas-cern--hephy--1782--254198731771049-4 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' ====== IEPSAS-Kosice ANALY_IEPSAS-Kosice-CC7 ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4646724676 15/2/2020 7:47 n59-iep-grid. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n59-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4632461503 2/2/2020 9:42 n18-iep-grid. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n18-iep-grid.saske.sk jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 09:44:25 sup:::2 worker already finished at 2020-02-02 09:57:44 with None 4658928816 1/3/2020 14:34 n45-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n45-iep-grid.saske.sk pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4655008739 26/2/2020 6:39 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4653600107 24/2/2020 10:08 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4648166341 17/2/2020 16:31 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4647767765 17/2/2020 1:48 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4647882421 17/2/2020 5:52 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4646215020 14/2/2020 16:40 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4646593206 15/2/2020 2:56 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4646673428 15/2/2020 6:23 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4645113137 13/2/2020 13:19 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4645260408 13/2/2020 16:31 n10-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n10-iep-grid.saske.sk 4645293959 13/2/2020 17:52 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4642317747 10/2/2020 20:57 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4641264738 10/2/2020 2:31 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4637957330 6/2/2020 19:37 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4636814928 5/2/2020 13:25 n30-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n30-iep-grid.saske.sk pilot:::1201 Job killed by signal: SIGTERM sup:::9000 Diag from worker : Condor HoldReason: None ; Condor RemoveReason: removed by SYSTEM_PERIODIC_REMOVE due to job restarted undesirably. 4637000696 5/2/2020 18:05 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4635715912 4/2/2020 20:08 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4635924095 4/2/2020 23:47 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4636188683 5/2/2020 3:00 n44-iep-grid. data18_13TeV:data18_13TeV.0034926 modificationHost: n44-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4655951040 27/2/2020 2:12 n6-iep-grid.s mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n6-iep-grid.saske.sk 4655956530 27/2/2020 2:22 n5-iep-grid.s mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n5-iep-grid.saske.sk 4655962679 27/2/2020 2:44 n74-iep-grid. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n74-iep-grid.saske.sk 4646630946 15/2/2020 4:16 n22-iep-grid. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n22-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4646193871 14/2/2020 15:26 n33-iep-grid. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n33-iep-grid.saske.sk pilot:::1322 Failed to get replicas from Rucio: HTTPSConnectionPool(host='voatlasrucio-server-prod.cern.ch', port=443): Read timed out. (read timeout=600)",)]:failed to transfer files using copytools=[u'rucio'] 4634313622 3/2/2020 20:53 n11-iep-grid. mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n11-iep-grid.saske.sk jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-03 20:54:32 sup:::2 worker already finished at 2020-02-03 21:16:02 with None IEPSAS-Kosice-CC7 ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4658922301 1/3/2020 14:23 n37-iep-grid. hc_test:mc15_13TeV.361106.PowhegP modificationHost: n37-iep-grid.saske.sk pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4645329071 13/2/2020 18:16 n37-iep-grid. hc_test:mc15_13TeV.361106.PowhegP modificationHost: n37-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4638825441 7/2/2020 17:34 n30-iep-grid. hc_test:mc15_13TeV.361106.PowhegP modificationHost: n30-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4631811127 1/2/2020 21:24 n53-iep-grid. hc_test:mc15_13TeV.361106.PowhegP modificationHost: n53-iep-grid.saske.sk jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 21:38:10 sup:::2 worker already finished at 2020-02-01 21:51:44 with None 4645675666 14/2/2020 4:32 n1-iep-grid.s hc_test:mc15_13TeV.361106.PowhegP modificationHost: n1-iep-grid.saske.sk pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.e1c4b7a9-1a6b-4e1a-9d98-8c6a9a8a11c1_2136.pool.root to IEPSAS-KOSICE_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=319 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4643894374 12/2/2020 7:51 n38-iep-grid. hc_test:mc15_13TeV.361106.PowhegP modificationHost: n38-iep-grid.saske.sk pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4658994232 1/3/2020 17:25 n8-iep-grid.s hc_test:mc15_13TeV.361106.PowhegP modificationHost: n8-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4657896551 29/2/2020 2:29 n5-iep-grid.s hc_test:mc15_13TeV.361106.PowhegP modificationHost: n5-iep-grid.saske.sk taskBuffer:::300 The worker was finished while the job was running : None 4644137522 12/2/2020 14:23 n1-iep-grid.s hc_test:mc15_13TeV.361106.PowhegP modificationHost: n1-iep-grid.saske.sk pilot:::1152 File transfer timed out during stage-out: hc_test:49981338-370a-46a9-9272-d52bcb44ad09_64329.1.job.log.tgz to IEPSAS-KOSICE_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 49981338-370a-46a9-9272-d52bcb44ad09_64329.1.job.log.tgz is missing in pilot XML 4641935321 10/2/2020 14:33 n42-iep-grid. hc_test:mc15_13TeV.361106.PowhegP modificationHost: n42-iep-grid.saske.sk pilot:::1152 File transfer timed out during stage-out: hc_test:0c83dd2b-f1de-43c8-9f4b-d5fa4a8e5869_90149.1.job.log.tgz to IEPSAS-KOSICE_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 0c83dd2b-f1de-43c8-9f4b-d5fa4a8e5869_90149.1.job.log.tgz is missing in pilot XML ====== LRZ-LMU LRZ-LMU_UCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4658592582 1/3/2020 1:40 hlcgr1oc01s12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s12 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658646293 1/3/2020 5:27 lxe30 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe30 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658770735 1/3/2020 8:35 lxe37 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe37 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658797344 1/3/2020 9:44 hlcgr1nc01s03 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1nc01s03 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4656739740 27/2/2020 23:02 hlcgr1oc01s12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s12 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4650312164 19/2/2020 19:44 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650323376 19/2/2020 20:05 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650338196 19/2/2020 20:27 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650344893 19/2/2020 20:40 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4649240543 18/2/2020 17:36 hlcgr1oc01s05 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s05 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4649925364 19/2/2020 12:20 lxe35 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe35 pilot:::1137 Failed to stage-out file: hc_test:output.1.48857c84-26cc-4998-83c0-af67f16224f0_86880.pool.root to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648145186 17/2/2020 16:10 hlcgr1oc01s04 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s04 jobDispatcher:::100 lost heartbeat : 2020-02-17 16:10:46 4644810128 13/2/2020 6:06 hlcgr1oc01s06 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s06 pilot:::1137 Failed to stage-out file: hc_test:0b45ead8-79b0-480c-a1b0-c26beb6c88bb_4490.1.job.log.tgz to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 0b45ead8-79b0-480c-a1b0-c26beb6c88bb_4490.1.job.log.tgz is missing in pilot XML 4635317011 4/2/2020 14:05 hlcgr09c01s08 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr09c01s08 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636587450 5/2/2020 7:49 aipanda066.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda066.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4632661348 2/2/2020 13:57 aipanda066.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda066.cern.ch jobDispatcher:::100 lost heartbeat : 2020-02-02 14:18:57 4632817387 2/2/2020 17:47 lxe46 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe46 taskBuffer:::300 The worker was finished while the job was running : None 4658590808 1/3/2020 1:40 hlcgr1oc01s12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s12 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658648925 1/3/2020 5:27 lxe38 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe38 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658768694 1/3/2020 7:55 lxe37 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe37 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658799281 1/3/2020 9:11 hlcgr1nc01s03 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1nc01s03 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4656371377 27/2/2020 17:45 hlcgr1oc01s02 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s02 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4650272812 19/2/2020 18:20 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650281104 19/2/2020 18:36 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650288819 19/2/2020 18:52 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650294139 19/2/2020 19:05 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650301123 19/2/2020 19:21 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650307442 19/2/2020 19:35 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650315306 19/2/2020 19:48 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650323542 19/2/2020 20:05 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650338185 19/2/2020 20:27 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650346596 19/2/2020 20:43 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650356011 19/2/2020 20:59 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650361828 19/2/2020 21:14 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650369439 19/2/2020 21:29 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650397273 19/2/2020 22:22 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650405033 19/2/2020 22:36 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650413522 19/2/2020 22:53 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650423077 19/2/2020 23:16 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650435301 None aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650444003 19/2/2020 23:57 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650453625 20/2/2020 0:17 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650462214 20/2/2020 0:34 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650467803 20/2/2020 0:49 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4649630067 19/2/2020 5:59 hlcgr1nc01s12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1nc01s12 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (15) Job was cancelled by SLURM sup:::9000 Diag from worker : LRMS error: (15) Job was cancelled by SLURM 4646783894 15/2/2020 10:30 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4641354061 10/2/2020 2:21 lxe27 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe27 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4640635476 9/2/2020 4:41 lxe62 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe62 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4639704900 8/2/2020 6:31 lxe50 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe50 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4638341864 7/2/2020 7:58 hlcgr1nc01s10 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1nc01s10 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (15) Job was cancelled by SLURM sup:::9000 Diag from worker : LRMS error: (15) Job was cancelled by SLURM 4637608746 6/2/2020 8:44 lxe16 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe16 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636340807 5/2/2020 5:41 lxe49 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe49 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job timeout sup:::9000 Diag from worker : LRMS error: (-1) Job timeout 4636530158 5/2/2020 6:48 aipanda066.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda066.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4634770073 4/2/2020 6:59 hlcgr09c01s06 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr09c01s06 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658688444 1/3/2020 5:25 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed sup:::9000 Diag from worker : LRMS error: (1) Job failed 4658783631 1/3/2020 8:28 hlcgr1oc01s01 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s01 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4651536187 21/2/2020 10:09 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4648112496 17/2/2020 14:46 hlcgr09c01s04 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr09c01s04 pilot:::1137 Failed to stage-out file: hc_test:output.1.24ed5804-e816-4e6f-b1ee-ce85ade56205_8881.pool.root to LRZ-LMU_SCRATCHDISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4646912525 15/2/2020 13:40 hlcgr1oc01s05 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s05 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4645323959 13/2/2020 18:12 hlcgr1oc01s03 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s03 taskBuffer:::300 The worker was finished while the job was running : None 4645581364 14/2/2020 1:09 hlcgr1nc01s09 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s09 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (15) Job was cancelled by SLURM sup:::9000 Diag from worker : LRMS error: (15) Job was cancelled by SLURM 4641919264 10/2/2020 14:21 hlcgr1oc01s11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s11 taskBuffer:::300 The worker was finished while the job was running : None 4640196666 8/2/2020 16:42 hlcgr1nc01s09 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s09 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4637738076 6/2/2020 12:15 hlcgr1nc01s10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s10 pilot:::1137 Failed to stage-out file: hc_test:output.1.009a4a7d-415c-44b8-a5bd-de808b0e6363_15827.pool.root to LRZ-LMU_SCRATCHDISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4636802998 5/2/2020 12:53 aipanda066.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda066.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636813773 5/2/2020 13:09 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636836664 5/2/2020 13:42 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4630661723 1/2/2020 6:13 aipanda066.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda066.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658650695 1/3/2020 2:28 hlcgr1nc01s02 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s02 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658737737 1/3/2020 8:23 hlcgr1oc01s01 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s01 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658791997 1/3/2020 8:43 aipanda404.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (257) Node fail sup:::9000 Diag from worker : LRMS error: (257) Node fail 4656554630 27/2/2020 18:16 hlcgr09c02s05 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr09c02s05 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4656602802 27/2/2020 19:37 hlcgr1nc01s09 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s09 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4656633261 27/2/2020 20:04 hlcgr1oc01s03 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s03 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4656659326 27/2/2020 21:21 lxe30 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe30 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4656716062 27/2/2020 22:20 lxe49 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe49 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4656772052 27/2/2020 23:30 hlcgr09c01s11 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr09c01s11 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4656821534 28/2/2020 0:14 lxe44 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe44 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4656839918 28/2/2020 1:19 lxe50 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe50 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4656882274 28/2/2020 2:33 hlcgr1nc01s06 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s06 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4656932170 28/2/2020 3:47 lxe17 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe17 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4656997403 28/2/2020 5:02 hlcgr1oc01s08 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s08 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4657061347 28/2/2020 6:20 hlcgr1nc01s04 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s04 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4657109421 28/2/2020 7:35 hlcgr1oc01s03 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s03 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4657146017 28/2/2020 8:50 lxe46 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe46 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4657195623 28/2/2020 9:30 aipanda403.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4657208326 28/2/2020 9:53 hlcgr1nc01s06 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s06 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4657235319 28/2/2020 11:12 hlcgr1nc01s01 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s01 pilot:::1324 Service not available at the moment: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0162._0001.1 from LRZ-LMU_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Operation expired',)]:failed to transfer files using copytools=[u'rucio'] 4655752511 None aipanda403.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4655861199 26/2/2020 23:45 hlcgr1oc01s12 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s12 pilot:::1152 File transfer timed out during stage-out: hc_test:a97ae1d6-e4a5-4a2e-b9f6-e2a1cda32778_62852.1.job.log.tgz to LRZ-LMU_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output a97ae1d6-e4a5-4a2e-b9f6-e2a1cda32778_62852.1.job.log.tgz is missing in pilot XML 4650023121 19/2/2020 12:22 hlcgr09c01s06 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr09c01s06 pilot:::1137 Failed to stage-out file: hc_test:70510d37-c628-4276-9d77-77677eeaf941_94152.1.job.log.tgz to LRZ-LMU_SCRATCHDISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 70510d37-c628-4276-9d77-77677eeaf941_94152.1.job.log.tgz is missing in pilot XML 4648438294 17/2/2020 23:35 hlcgr1oc01s04 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s04 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4647623240 16/2/2020 17:48 hlcgr1nc01s09 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s09 pilot:::1152 File transfer timed out during stage-out: hc_test:29aa16a0-2a67-4af6-94b1-eecb4621df41_49984.1.job.log.tgz to LRZ-LMU_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 29aa16a0-2a67-4af6-94b1-eecb4621df41_49984.1.job.log.tgz is missing in pilot XML 4647191428 16/2/2020 0:50 lxe30 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe30 taskBuffer:::300 The worker was finished while the job was running : None 4646486432 14/2/2020 22:54 hlcgr1nc01s03 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s03 taskBuffer:::300 The worker was finished while the job was running : None 4645306811 13/2/2020 17:42 lxe32 data18_13TeV:data18_13TeV.0034926 modificationHost: lxe32 taskBuffer:::300 The worker was finished while the job was running : None 4645576616 14/2/2020 1:11 hlcgr1nc01s09 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s09 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (15) Job was cancelled by SLURM sup:::9000 Diag from worker : LRMS error: (15) Job was cancelled by SLURM 4644622217 12/2/2020 21:45 hlcgr1nc01s05 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s05 taskBuffer:::300 The worker was finished while the job was running : None 4642371643 10/2/2020 21:02 hlcgr1nc01s12 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s12 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4642899147 11/2/2020 5:02 hlcgr1oc01s01 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s01 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4641036496 9/2/2020 16:56 hlcgr1oc01s04 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s04 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4641188429 9/2/2020 20:57 hlcgr1oc01s08 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s08 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (15) Job was cancelled by SLURM sup:::9000 Diag from worker : LRMS error: (15) Job was cancelled by SLURM 4640247428 8/2/2020 17:37 aipanda404.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (15) Job was cancelled by SLURM sup:::9000 Diag from worker : LRMS error: (15) Job was cancelled by SLURM 4640509804 9/2/2020 1:49 hlcgr1nc01s08 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1nc01s08 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4640781655 9/2/2020 10:18 hlcgr1oc01s08 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr1oc01s08 taskBuffer:::300 The worker was finished while the job was running : None 4636802630 5/2/2020 12:50 aipanda403.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4633853876 3/2/2020 21:03 aipanda403.ce data18_13TeV:data18_13TeV.0034926 modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4631706959 1/2/2020 22:05 hlcgr09c01s11 data18_13TeV:data18_13TeV.0034926 modificationHost: hlcgr09c01s11 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 22:16:41 sup:::2 worker already finished at 2020-02-01 22:25:27 with None 4658783286 1/3/2020 8:28 hlcgr1oc01s01 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s01 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4657754136 28/2/2020 22:44 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch jobDispatcher:::100 lost heartbeat : 2020-02-28 22:44:50 4656148847 27/2/2020 8:28 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (257) Node fail sup:::9000 Diag from worker : LRMS error: (257) Node fail 4655951029 27/2/2020 2:15 hlcgr1nc01s03 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s03 4655957681 27/2/2020 2:28 hlcgr1nc01s03 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s03 4655969902 27/2/2020 2:44 hlcgr1nc01s03 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s03 4655975921 27/2/2020 2:56 hlcgr1nc01s03 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s03 4650304182 19/2/2020 19:28 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650312967 19/2/2020 19:45 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650320332 19/2/2020 19:59 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650331872 19/2/2020 20:17 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650343129 19/2/2020 20:37 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4649471864 None aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4648379313 17/2/2020 22:05 hlcgr1nc01s03 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s03 taskBuffer:::300 The worker was finished while the job was running : None 4646662850 15/2/2020 5:31 hlcgr1nc01s11 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1nc01s11 taskBuffer:::300 The worker was finished while the job was running : None 4645511085 13/2/2020 22:57 hlcgr1oc01s04 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s04 taskBuffer:::300 The worker was finished while the job was running : None 4645678814 14/2/2020 4:42 hlcgr1oc01s05 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s05 taskBuffer:::300 The worker was finished while the job was running : None 4645265701 13/2/2020 16:55 hlcgr1oc01s12 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s12 taskBuffer:::300 The worker was finished while the job was running : None 4640543022 9/2/2020 2:10 hlcgr1oc01s08 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s08 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4638873273 7/2/2020 18:16 hlcgr1oc01s06 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: hlcgr1oc01s06 taskBuffer:::300 The worker was finished while the job was running : None 4636840501 5/2/2020 13:47 aipanda066.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda066.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636851913 5/2/2020 14:03 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636802425 None aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658998558 1/3/2020 17:37 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658593082 1/3/2020 1:40 hlcgr1oc01s12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s12 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658648305 1/3/2020 5:27 lxe38 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe38 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658778261 1/3/2020 8:32 lxe38 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe38 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658801113 1/3/2020 9:12 hlcgr1nc01s03 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1nc01s03 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4658140699 29/2/2020 14:33 hlcgr1oc01s07 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s07 taskBuffer:::300 The worker was finished while the job was running : None 4656900911 28/2/2020 2:36 hlcgr09c02s02 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr09c02s02 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000036.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4656919098 28/2/2020 3:12 lxe17 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe17 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000036.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657137032 28/2/2020 8:00 hlcgr1nc01s03 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1nc01s03 taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4657238676 28/2/2020 10:46 hlcgr1oc01s06 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s06 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000036.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=387 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4651364402 21/2/2020 3:19 lxe31 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe31 pilot:::1137 Failed to stage-out file: hc_test:output.1.d55bdf3c-4de3-43a3-90f7-0ac81157e510_88716.pool.root to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4650599213 20/2/2020 5:26 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650617560 20/2/2020 6:15 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650623094 20/2/2020 6:30 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650629199 20/2/2020 6:46 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650637883 20/2/2020 7:03 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650664941 20/2/2020 7:24 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4649918549 19/2/2020 9:20 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4649992440 19/2/2020 12:27 hlcgr1oc01s03 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s03 pilot:::1137 Failed to stage-out file: hc_test:output.1.4a7491d3-f22a-4ed1-81ea-bc9fd650a09c_24150.pool.root to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] 4650333955 19/2/2020 20:20 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650342331 19/2/2020 20:36 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650355150 19/2/2020 20:58 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650362321 19/2/2020 21:15 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650372361 19/2/2020 21:37 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650384750 19/2/2020 21:59 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650390863 19/2/2020 22:12 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650403728 19/2/2020 22:31 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650410075 19/2/2020 22:44 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650417486 19/2/2020 23:02 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650425858 19/2/2020 23:21 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650435763 19/2/2020 23:40 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650448675 20/2/2020 0:08 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650461532 20/2/2020 0:33 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4650472019 20/2/2020 0:56 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Job submission to LRMS failed sup:::9000 Diag from worker : Job submission to LRMS failed 4648171362 17/2/2020 16:25 hlcgr1oc01s08 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s08 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Node fail jobDispatcher:::100 lost heartbeat : 2020-02-17 16:33:37 sup:::2 worker already failed at 2020-02-17 19:27:56 with LRMS error: (-1) Node fail 4645143045 13/2/2020 20:27 hlcgr1nc01s05 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1nc01s05 taskBuffer:::300 The worker was finished while the job was running : None 4637606307 6/2/2020 8:49 lxe62 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe62 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636545649 5/2/2020 7:07 aipanda066.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda066.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636802813 5/2/2020 12:51 aipanda403.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4636846867 5/2/2020 13:57 lxe18 hc_test:mc15_13TeV.361106.PowhegP modificationHost: lxe18 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM 4634799699 4/2/2020 7:25 hlcgr1oc01s01 hc_test:mc15_13TeV.361106.PowhegP modificationHost: hlcgr1oc01s01 taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) Job missing from SLURM sup:::9000 Diag from worker : LRMS error: (-1) Job missing from SLURM LRZ-LMU_CLOUD ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4657600910 28/2/2020 21:11 slot1_10@atla hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_10@atlas-cern--lrz--1782--199006905227852 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657704488 28/2/2020 21:28 slot1_3@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_3@atlas-cern--lrz--1782--199006905227852 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4649745348 19/2/2020 4:17 slot1_8@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_8@atlas-cern--lrz--1782--31724768791053 pilot:::1137 Failed to stage-out file: hc_test:99a4087c-9832-48ea-b57b-580afbe25ad4_66176.1.job.log.tgz to LRZ-LMU_DATADISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 99a4087c-9832-48ea-b57b-580afbe25ad4_66176.1.job.log.tgz is missing in pilot XML 4631560979 1/2/2020 18:15 slot1_3@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_3@atlas-cern--lrz--1782--104304363690332 taskBuffer:::300 The worker was finished while the job was running : None 4631736293 1/2/2020 20:25 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--lrz--1782--104304363690332 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 20:41:39 sup:::2 worker already finished at 2020-02-01 20:47:02 with None 4646516970 15/2/2020 7:18 slot1_2@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_2@atlas-cern--lrz--1782--56553304273193 pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20161287.LRZ-LMU_CLOUD.a79e09d5-828c-4a5f-90a5-459f15d574ef_94630.HITS.pool.root to LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=335 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4638024426 6/2/2020 21:42 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--lrz--1782--148569865442166 taskBuffer:::300 The worker was cancelled while the job was running : condor job submissionHost=LOCAL batchID=382134.1 not found. Skipped sup:::9000 Diag from worker : condor job submissionHost=LOCAL batchID=382134.1 not found. Skipped 4652128832 23/2/2020 1:52 slot1_1@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@atlas-cern--lrz--1782--269543549908233 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000026.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=384 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4651332691 21/2/2020 2:05 slot1_4@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@atlas-cern--lrz--1782--31724768791053 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=385 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4646780860 15/2/2020 10:23 slot1_5@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@atlas-cern--lrz--1782--56553304273193 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.0b48ee1d-9c42-4d18-906a-7d9095acbc5a_14738.pool.root to LRZ-LMU_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4636968035 5/2/2020 16:46 slot1_6@atlas hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_6@atlas-cern--lrz--1782--148569865442166 pilot:::1322 Failed to get replicas from Rucio: HTTPSConnectionPool(host='voatlasrucio-server-prod.cern.ch', port=443): Read timed out. (read timeout=600)",)]:failed to transfer files using copytools=[u'rucio'] ====== MPPMU ANALY_MPPMU ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4658820621 1/3/2020 10:19 bt09 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt09 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.916af889-6936-49da-b2a8-91137b12a854_57353.pool.root to MPPMU_SCRATCHDISK, ((('stageOut: File existence verification failed with: error code: 1137, message: Failed to stage-out file\ndetails: stageOut: Physical check after upload failed.',), {}), {})",)]:f 4658832880 1/3/2020 10:38 at13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at13 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.aa5894a1-0ede-4535-b3d4-016e40b1070d_84449.pool.root to MPPMU_SCRATCHDISK, ((('stageOut: File existence verification failed with: error code: 1137, message: Failed to stage-out file\ndetails: stageOut: Physical check after upload failed.',), {}), {})",)]:f 4645870768 14/2/2020 7:38 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645915074 14/2/2020 8:32 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645960757 14/2/2020 9:11 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645981450 14/2/2020 9:30 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645988373 14/2/2020 9:48 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645998703 14/2/2020 10:06 at48 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at48 4646010720 14/2/2020 10:26 at42 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at42 4646190191 14/2/2020 15:18 bt02 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt02 pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4645061333 13/2/2020 20:41 at48 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at48 4645453742 13/2/2020 21:01 bt25 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt25 4645469506 13/2/2020 21:34 at45 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at45 4645486852 13/2/2020 21:56 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645503752 13/2/2020 22:28 bt24 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt24 4645522482 13/2/2020 22:59 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645538655 13/2/2020 23:34 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645552764 14/2/2020 0:07 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645590731 14/2/2020 1:23 at44 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at44 4645614021 14/2/2020 1:56 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645628950 14/2/2020 2:31 at44 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at44 4645642197 14/2/2020 3:05 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645655628 14/2/2020 3:43 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645667508 14/2/2020 4:53 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645754285 14/2/2020 5:42 bt02 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt02 4645809819 14/2/2020 6:29 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4658819762 1/3/2020 10:01 at35 data18_13TeV:data18_13TeV.0034926 modificationHost: at35 pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4648159935 17/2/2020 15:53 kt01 data18_13TeV:data18_13TeV.0034926 modificationHost: kt01 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1: checksum (catalog)=770c7b63 != checksum (local)=bf0f931f',)]:failed to transfer files using copytools=[u'rucio'] 4645113047 13/2/2020 12:31 bt19 data18_13TeV:data18_13TeV.0034926 modificationHost: bt19 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645451954 13/2/2020 20:55 ft04 data18_13TeV:data18_13TeV.0034926 modificationHost: ft04 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645467684 13/2/2020 21:23 bt24 data18_13TeV:data18_13TeV.0034926 modificationHost: bt24 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645485352 13/2/2020 21:52 bt24 data18_13TeV:data18_13TeV.0034926 modificationHost: bt24 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645492263 13/2/2020 22:08 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645502079 13/2/2020 22:23 bt24 data18_13TeV:data18_13TeV.0034926 modificationHost: bt24 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645512382 13/2/2020 22:39 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645525356 13/2/2020 23:04 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645537014 13/2/2020 23:28 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645544328 13/2/2020 23:44 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645576606 14/2/2020 0:45 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645594838 14/2/2020 1:22 bt02 data18_13TeV:data18_13TeV.0034926 modificationHost: bt02 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645612331 14/2/2020 1:49 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645786086 14/2/2020 6:07 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645812701 14/2/2020 6:28 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645840771 14/2/2020 6:51 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645868243 14/2/2020 7:15 at42 data18_13TeV:data18_13TeV.0034926 modificationHost: at42 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645898460 14/2/2020 7:38 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645927250 14/2/2020 8:02 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645981369 14/2/2020 9:28 at42 data18_13TeV:data18_13TeV.0034926 modificationHost: at42 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645993450 14/2/2020 9:53 at26 data18_13TeV:data18_13TeV.0034926 modificationHost: at26 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4646006570 14/2/2020 10:18 at42 data18_13TeV:data18_13TeV.0034926 modificationHost: at42 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4644039311 12/2/2020 12:39 at28 data18_13TeV:data18_13TeV.0034926 modificationHost: at28 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4658826701 1/3/2020 10:24 at13 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at13 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.fcb629f2-e6b3-47a1-b0a5-b1e471770797_85498.pool.root to MPPMU_SCRATCHDISK, ((('stageOut: File existence verification failed with: error code: 1137, message: Failed to stage-out file\ndetails: stageOut: Physical check after upload failed.',), {}), {})",)]:f 4655950940 27/2/2020 4:48 kt05 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: kt05 4656053045 27/2/2020 5:22 at30 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at30 4656067104 27/2/2020 6:37 kt10 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: kt10 4645453675 13/2/2020 21:01 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645461145 13/2/2020 21:10 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645470132 13/2/2020 21:30 bt28 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt28 4645477530 13/2/2020 21:41 bt24 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt24 4645485313 13/2/2020 21:54 bt24 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: bt24 4645497691 13/2/2020 22:19 at42 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at42 4645511095 13/2/2020 22:38 at42 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at42 4645522711 13/2/2020 22:59 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645531258 13/2/2020 23:35 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645544341 14/2/2020 0:32 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645579371 14/2/2020 0:56 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645588423 14/2/2020 1:18 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645608234 14/2/2020 1:47 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645619335 14/2/2020 5:53 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645796948 14/2/2020 6:17 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645833335 14/2/2020 7:37 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645924836 14/2/2020 8:03 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645952862 14/2/2020 8:35 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645964065 14/2/2020 9:11 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645987290 14/2/2020 9:45 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4645999306 14/2/2020 10:09 at26 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at26 4646013711 14/2/2020 10:33 kt01 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: kt01 4645016812 13/2/2020 8:59 at23 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: at23 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.fa71d5ec-5780-4be6-8698-4816c8e4f8ac_46558.pool.root to MPPMU_SCRATCHDISK, None of the given files have been uploaded.',)]:failed to transfer files using copytools=[u'rucio'] MPPMU ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4659155755 1/3/2020 21:47 at18 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at18 pilot:::1322 Failed to get replicas from Rucio: HTTPSConnectionPool(host='voatlasrucio-server-prod.cern.ch', port=443): Read timed out. (read timeout=600)",)]:failed to transfer files using copytools=[u'rucio'] 4658822416 1/3/2020 10:17 at15 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at15 pilot:::1137 Failed to stage-out file: hc_test:output.1.bd1249a8-0637-4f7d-ba3c-2e634083b3e4_23116.pool.root to MPPMU_DATADISK, ((('stageOut: File existence verification failed with: error code: 1137, message: Failed to stage-out file\ndetails: stageOut: Physical check after upload failed.',), {}), {})",)]:failed to 4645275376 13/2/2020 20:41 bt09 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt09 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645454896 13/2/2020 21:04 bt27 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt27 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645468001 13/2/2020 21:25 at48 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at48 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645478423 13/2/2020 21:43 ft01 hc_test:mc15_13TeV.361106.PowhegP modificationHost: ft01 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645486319 13/2/2020 22:23 bt24 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt24 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645510148 13/2/2020 22:49 bt25 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt25 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645525597 13/2/2020 23:14 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645539316 13/2/2020 23:41 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645550194 14/2/2020 0:02 bt25 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt25 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645558600 14/2/2020 0:23 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645574656 14/2/2020 1:23 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645601170 14/2/2020 1:38 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645614516 14/2/2020 1:56 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645626919 14/2/2020 2:26 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645635082 14/2/2020 2:41 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645641265 14/2/2020 3:02 at48 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at48 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645646885 14/2/2020 3:20 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645654759 14/2/2020 3:56 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645668844 14/2/2020 4:23 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645684609 14/2/2020 4:56 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645723882 14/2/2020 5:18 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645753349 14/2/2020 5:47 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645784612 14/2/2020 6:09 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645817007 14/2/2020 6:33 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645849853 14/2/2020 7:02 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645886004 14/2/2020 7:48 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645933884 14/2/2020 8:17 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645955728 14/2/2020 8:39 kt01 hc_test:mc15_13TeV.361106.PowhegP modificationHost: kt01 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645967339 14/2/2020 9:15 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645984539 14/2/2020 9:36 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645997990 14/2/2020 10:09 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000025.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4644041237 12/2/2020 12:54 at12 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at12 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000021.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4642084750 11/2/2020 1:23 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 exe:::65 Non-zero return code from EVNTtoHITS (33); Logfile error in log.EVNTtoHITS: "IOVDbSvc FATAL Conditions database connection COOLOFL_INDET/OFLP200 cannot be opened - STOP" 4658822086 1/3/2020 10:14 at35 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at35 pilot:::1137 Failed to stage-out file: hc_test:output.1.65473e5c-6f5c-4eb2-b259-82e208a4a972_63210.pool.root to MPPMU_DATADISK, ((('stageOut: File existence verification failed with: error code: 1137, message: Failed to stage-out file\ndetails: stageOut: Physical check after upload failed.',), {}), {})",)]:failed to 4645607412 14/2/2020 1:56 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645619748 14/2/2020 2:26 bt21 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt21 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645634020 14/2/2020 2:41 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645639075 14/2/2020 3:02 at48 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at48 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645645272 14/2/2020 3:17 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645651851 14/2/2020 3:56 at44 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at44 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645666258 14/2/2020 4:23 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645677585 14/2/2020 4:41 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645700569 14/2/2020 5:18 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645745435 14/2/2020 5:33 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645767546 14/2/2020 5:51 at48 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at48 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645783437 14/2/2020 6:09 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645808622 14/2/2020 6:33 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645834761 14/2/2020 6:51 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645861717 14/2/2020 7:11 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645883136 14/2/2020 7:29 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645906342 14/2/2020 7:48 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645925275 14/2/2020 8:03 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645947327 14/2/2020 8:36 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645965118 14/2/2020 9:15 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645984479 14/2/2020 9:36 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645991829 14/2/2020 9:51 kt01 hc_test:mc15_13TeV.361106.PowhegP modificationHost: kt01 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645999779 14/2/2020 10:09 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645059112 13/2/2020 20:41 at20 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at20 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645455614 13/2/2020 21:22 bt25 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt25 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645475680 13/2/2020 21:43 ft03 hc_test:mc15_13TeV.361106.PowhegP modificationHost: ft03 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645489177 13/2/2020 22:23 bt24 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt24 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645516702 13/2/2020 23:14 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645539770 13/2/2020 23:41 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645551502 14/2/2020 0:02 bt25 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt25 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645567803 14/2/2020 0:40 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645588149 14/2/2020 1:23 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000032.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4658824758 1/3/2020 10:18 at15 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at15 pilot:::1137 Failed to stage-out file: hc_test:output.1.47ad0fb1-3074-4659-91a5-4e860ff76c15_65271.pool.root to MPPMU_DATADISK, ((('stageOut: File existence verification failed with: error code: 1137, message: Failed to stage-out file\ndetails: stageOut: Physical check after upload failed.',), {}), {})",)]:failed to 4645682335 14/2/2020 4:41 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645698281 14/2/2020 4:57 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645725385 14/2/2020 5:18 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645739292 14/2/2020 5:33 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645759107 14/2/2020 5:48 at48 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at48 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645781119 14/2/2020 6:09 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645804309 14/2/2020 6:33 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645833436 14/2/2020 6:51 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645857441 14/2/2020 7:11 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645885003 14/2/2020 7:30 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645913044 14/2/2020 7:49 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645928683 14/2/2020 8:03 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645940798 14/2/2020 8:19 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645953486 14/2/2020 8:36 kt01 hc_test:mc15_13TeV.361106.PowhegP modificationHost: kt01 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645961137 14/2/2020 9:15 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645985652 14/2/2020 9:51 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000038.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645057829 13/2/2020 20:41 at20 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at20 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645463889 13/2/2020 21:22 bt25 hc_test:mc15_13TeV.361106.PowhegP modificationHost: bt25 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645483491 13/2/2020 22:23 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645522611 13/2/2020 23:14 at42 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at42 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645547079 13/2/2020 23:52 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645565185 14/2/2020 0:40 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645593482 14/2/2020 1:23 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645617095 14/2/2020 2:08 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645632760 14/2/2020 2:41 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645645190 14/2/2020 3:17 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4645660302 14/2/2020 3:56 at26 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at26 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from MPPMU_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4642923854 11/2/2020 6:36 ft04 hc_test:mc15_13TeV.361106.PowhegP modificationHost: ft04 exe:::65 Non-zero return code from EVNTtoHITS (33); Logfile error in log.EVNTtoHITS: "IOVDbSvc FATAL Conditions database connection COOLOFL_TRT/OFLP200 cannot be opened - STOP" 4641784174 10/2/2020 12:36 at44 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at44 taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: Unspecified gridmanager error ; Worker canceled by harvester due to held too long or not found sup:::9000 Diag from worker : Condor HoldReason: Unspecified gridmanager error ; Worker canceled by harvester due to held too long or not found 4636486332 5/2/2020 6:06 at28 hc_test:mc15_13TeV.361106.PowhegP modificationHost: at28 exe:::65 Non-zero return code from EVNTtoHITS (33); Logfile error in log.EVNTtoHITS: "IOVDbSvc FATAL Conditions database connection COOLOFL_TRT/OFLP200 cannot be opened - STOP" ====== praguelcg2 ANALY_FZU ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4635268637 4/2/2020 12:44 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : LRMS error: (1) Job failed with exit code 1 sup:::9000 Diag from worker : LRMS error: (1) Job failed with exit code 1 4635346662 4/2/2020 13:45 slot1_27@maha mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_27@mahagon01.farm.particle.cz jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-04 13:48:14 sup:::2 worker already finished at 2020-02-04 13:54:52 with None 4657598782 28/2/2020 19:19 slot1_4@ib09. data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_4@ib09.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: user.gangarbt:user.gangarbt.f56d2488-6d5b-4b6c-9021-3b6ba93b81d9_42747.DAOD_EXOT12.test.pool.root to PRAGUELCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4657955292 29/2/2020 4:03 slot1_12@ib21 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_12@ib21.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c3040a93-3173-489b-842b-3de60de6f432_36318.pool.root to PRAGUELCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4655124489 26/2/2020 7:48 slot1_31@magi data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_31@magic35.farm.particle.cz taskBuffer:::300 The worker was failed while the job was running : LRMS error: (-1) PeriodicRemove evaluated to TRUE sup:::9000 Diag from worker : LRMS error: (-1) PeriodicRemove evaluated to TRUE 4648102444 17/2/2020 14:30 slot1_1@mikan data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_1@mikan03.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.26bfe357-b060-476f-be9b-274e428bb590_83831.pool.root to PRAGUELCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4648629057 18/2/2020 4:01 slot1_17@magi data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_17@magic12.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.c2542f1c-fd47-4732-b48c-c4d1cfcf3a1f_26928.pool.root to PRAGUELCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647989547 17/2/2020 10:29 slot1_7@mikan data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_7@mikan27.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.eab94d92-471a-4cce-aefe-dcdc40e59b78_10404.pool.root to PRAGUELCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635565789 4/2/2020 16:56 slot1_1@ibis2 data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_1@ibis20.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: user.gangarbt:output.1.0cadded3-f9cb-4381-a0e3-49be1ad3ce9d_11150.pool.root to PRAGUELCG2_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=314 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4631625363 1/2/2020 19:15 slot1_13@ibis data18_13TeV:data18_13TeV.0034926 modificationHost: slot1_13@ibis21.farm.particle.cz jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 19:30:17 sup:::2 worker already finished at 2020-02-01 19:40:58 with None 4635315629 4/2/2020 13:27 slot1_18@ib01 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: slot1_18@ib01.farm.particle.cz ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' praguelcg2_fzu_UCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4655198088 26/2/2020 10:31 slot1_8@mff46 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_8@mff46.farm.particle.cz pilot:::1322 Failed to get replicas from Rucio: An unknown exception occurred. Details: no error information passed (http status code: 503 ('service_unavailable', 'unavailable'))",)]:failed to transfer files using copytools=[u'rucio'] 4640853642 9/2/2020 11:45 slot1_13@ibis hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_13@ibis31.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.05c880a6-2300-4908-8eaa-6a02ed13162c_17036.pool.root to PRAGUELCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4639184920 7/2/2020 23:00 slot1_15@aple hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_15@aplex11.farm.particle.cz pilot:::1322 Failed to get replicas from Rucio: An unknown exception occurred. Details: no error information passed (http status code: 503 ('service_unavailable', 'unavailable'))",)]:failed to transfer files using copytools=[u'rucio'] 4658732142 1/3/2020 5:35 slot1_5@mff25 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_5@mff25.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.9f2d4bea-2228-4361-9f72-f81fd00ab5f0_7692.pool.root to PRAGUELCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=328 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647832957 17/2/2020 3:59 slot1_31@magi hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_31@magic05.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.bcb4ceab-3188-49ba-816d-efbb229fec4a_21840.pool.root to PRAGUELCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=318 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647443748 16/2/2020 12:09 slot1_13@maha hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_13@mahagon14.farm.particle.cz taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: Unspecified gridmanager error ; Worker canceled by harvester due to held too long or not found jobDispatcher:::100 lost heartbeat : 2020-02-16 12:09:48 sup:::2 worker already cancelled at 2020-02-16 14:12:26 with Condor HoldReason: Unspecified gridmanager error ; Worker canceled by harvester due to held too long or not found 4646279805 14/2/2020 17:26 slot1_8@mff48 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_8@mff48.farm.particle.cz pilot:::1322 Failed to get replicas from Rucio: HTTPSConnectionPool(host='voatlasrucio-server-prod.cern.ch', port=443): Read timed out. (read timeout=600)",)]:failed to transfer files using copytools=[u'rucio'] 4645304548 13/2/2020 17:37 slot1_8@mikan hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_8@mikan03.farm.particle.cz taskBuffer:::300 The worker was finished while the job was running : None 4635281959 4/2/2020 12:57 slot1_28@maha hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_28@mahagon03.farm.particle.cz ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4647382585 16/2/2020 9:26 slot1_1@mff21 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@mff21.farm.particle.cz pilot:::1098 Not enough local space 4638417658 7/2/2020 9:42 slot1_1@mff74 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_1@mff74.farm.particle.cz pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.cac1e19e-33f8-41e9-8d99-ab5d72723ce5_86303.pool.root to PRAGUELCG2_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4635294785 4/2/2020 13:11 slot1_4@mff25 hc_test:mc15_13TeV.361106.PowhegP modificationHost: slot1_4@mff25.farm.particle.cz ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' ====== PSNC ANALY_PSNC ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4653554210 24/2/2020 8:19 e0144 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0144 jobDispatcher:::100 lost heartbeat : 2020-02-24 08:19:32 4648793469 18/2/2020 9:04 e0402 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0402 pilot:::1324 Service not available at the moment: user.gangarbt:output.1.029c92a2-b474-4c4f-906c-f8599e1dd150_29668.pool.root to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0402 reports could not open connection to se.reef.man.poznan.p 4648156095 17/2/2020 15:58 e0543 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0543 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.9b70f337-ef7e-49a6-ac63-64e0f538f1cf_1059.pool.root to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4648194250 17/2/2020 16:58 e0459 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0459 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.a3fcfcf7-9e27-4e2c-b88b-cd7d8b1e2e23_56890.pool.root to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4648250828 17/2/2020 18:36 e0477 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0477 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.9018833c-0762-4f98-9e28-6a0edf01f5ac_59264.pool.root to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4648337260 17/2/2020 20:52 e0184 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0184 pilot:::1137 Failed to stage-out file: user.gangarbt:e650ab07-0963-4209-819f-79153dd10f26_45245.1.job.log.tgz to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output e650ab07-0963-4209-819f-79153dd10f26_45245.1.job.log.tgz is missing in pilot XML 4648705637 18/2/2020 6:35 e0488 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0488 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647398463 16/2/2020 10:07 e0125 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0125 pilot:::1137 Failed to stage-out file: user.gangarbt:02b340a5-71c2-48ee-bada-876f19f1be48_90513.1.job.log.tgz to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 02b340a5-71c2-48ee-bada-876f19f1be48_90513.1.job.log.tgz is missing in pilot XML 4647663449 16/2/2020 19:50 e0168 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0168 pilot:::1324 Service not available at the moment: user.gangarbt:output.1.4f029e90-4767-496e-9e9b-48935b6a8e16_57037.pool.root to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0168 reports could not open connection to se.reef.man.poznan.p 4647679927 16/2/2020 20:22 e0483 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0483 pilot:::1099 Failed to stage-in file: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4647693776 16/2/2020 21:03 e0345 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0345 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647709040 16/2/2020 21:44 e0668 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0668 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647726822 16/2/2020 22:43 e0015 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0015 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647746183 16/2/2020 23:25 e0020 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0020 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4646786006 15/2/2020 10:49 e0157 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0157 pilot:::1137 Failed to stage-out file: user.gangarbt:46212235-b63e-4350-8c5b-8bcb8b39fe24_66088.1.job.log.tgz to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 46212235-b63e-4350-8c5b-8bcb8b39fe24_66088.1.job.log.tgz is missing in pilot XML 4647003955 15/2/2020 15:34 e0310 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0310 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4643882946 12/2/2020 7:26 e0047 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0047 taskBuffer:::300 The worker was finished while the job was running : None 4644272558 12/2/2020 16:12 e0010 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0010 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.8bfb6e6f-1f38-4623-9895-b518ecb9ae1d_68134.pool.root to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4644370016 12/2/2020 17:49 e0010 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0010 pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] 4644654015 12/2/2020 22:14 e0031 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0031 pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output a582542b-2286-40d3-a42c-dafb052b45ba_74496.1.job.log.tgz is missing in pilot XML 4640282137 8/2/2020 18:52 e0587 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0587 pilot:::1324 Service not available at the moment: user.gangarbt:output.1.fbaa700b-d49e-44e9-8aae-55e2858cf820_24961.pool.root to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0587 reports could not open connection to se.reef.man.poznan.p 4635744382 4/2/2020 21:54 e0184 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0184 pilot:::1099 Failed to stage-in file: [ImportError('No module named rucio.client',)]:failed to transfer files using copytools=[u'rucio'] 4634233404 3/2/2020 19:22 e0051 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0051 taskBuffer:::300 The worker was finished while the job was running : None 4630828987 1/2/2020 8:28 e0145 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0145 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630867109 1/2/2020 8:56 e0032 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0032 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630891509 1/2/2020 9:24 e0049 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0049 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630915522 1/2/2020 9:50 e0662 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0662 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630943249 1/2/2020 10:18 e0769 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0769 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630977124 1/2/2020 10:55 e0007 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0007 jobDispatcher:::100 lost heartbeat : 2020-02-01 10:55:48 4631178669 1/2/2020 13:27 e0451 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0451 jobDispatcher:::100 lost heartbeat : 2020-02-01 13:40:17 4631432760 1/2/2020 17:44 e0038 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0038 jobDispatcher:::100 lost heartbeat : 2020-02-01 17:44:14 4631767766 1/2/2020 21:54 e0057 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0057 jobDispatcher:::100 lost heartbeat : 2020-02-01 23:42:14 4632076963 2/2/2020 2:43 e0005 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0005 jobDispatcher:::100 lost heartbeat : 2020-02-02 06:01:11 4653576180 24/2/2020 8:20 e0144 data18_13TeV:data18_13TeV.0034926 modificationHost: e0144 jobDispatcher:::100 lost heartbeat : 2020-02-24 09:00:04 4653656805 24/2/2020 11:51 e1084 data18_13TeV:data18_13TeV.0034926 modificationHost: e1084 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4653695148 24/2/2020 14:30 e0591 data18_13TeV:data18_13TeV.0034926 modificationHost: e0591 jobDispatcher:::100 lost heartbeat : 2020-02-24 14:31:01 4654075292 25/2/2020 0:25 e0144 data18_13TeV:data18_13TeV.0034926 modificationHost: e0144 jobDispatcher:::100 lost heartbeat : 2020-02-25 02:34:48 4654198841 25/2/2020 5:42 e0018 data18_13TeV:data18_13TeV.0034926 modificationHost: e0018 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4648159789 17/2/2020 15:57 e0543 data18_13TeV:data18_13TeV.0034926 modificationHost: e0543 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648171942 17/2/2020 16:27 e0467 data18_13TeV:data18_13TeV.0034926 modificationHost: e0467 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648191791 17/2/2020 16:42 e0578 data18_13TeV:data18_13TeV.0034926 modificationHost: e0578 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648200239 17/2/2020 16:59 e0163 data18_13TeV:data18_13TeV.0034926 modificationHost: e0163 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648210638 17/2/2020 17:20 e0475 data18_13TeV:data18_13TeV.0034926 modificationHost: e0475 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648224683 17/2/2020 17:57 e0176 data18_13TeV:data18_13TeV.0034926 modificationHost: e0176 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648248182 17/2/2020 18:17 e0463 data18_13TeV:data18_13TeV.0034926 modificationHost: e0463 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648261710 17/2/2020 18:41 e0849 data18_13TeV:data18_13TeV.0034926 modificationHost: e0849 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648279762 17/2/2020 19:03 e0155 data18_13TeV:data18_13TeV.0034926 modificationHost: e0155 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648295049 17/2/2020 19:34 e0824 data18_13TeV:data18_13TeV.0034926 modificationHost: e0824 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648313821 17/2/2020 20:14 e0611 data18_13TeV:data18_13TeV.0034926 modificationHost: e0611 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648336613 17/2/2020 20:51 e0184 data18_13TeV:data18_13TeV.0034926 modificationHost: e0184 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648357399 17/2/2020 21:20 e0474 data18_13TeV:data18_13TeV.0034926 modificationHost: e0474 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648478313 18/2/2020 1:32 e0824 data18_13TeV:data18_13TeV.0034926 modificationHost: e0824 pilot:::1151 File transfer timed out during stage-in: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=4690 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4648556917 18/2/2020 3:17 e0494 data18_13TeV:data18_13TeV.0034926 modificationHost: e0494 pilot:::1151 File transfer timed out during stage-in: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=4690 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4648661196 18/2/2020 5:08 e0165 data18_13TeV:data18_13TeV.0034926 modificationHost: e0165 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4648701279 18/2/2020 6:52 e0176 data18_13TeV:data18_13TeV.0034926 modificationHost: e0176 pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0149._0001.1 from PSNC_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4647667085 16/2/2020 20:13 e0020 data18_13TeV:data18_13TeV.0034926 modificationHost: e0020 pilot:::1151 File transfer timed out during stage-in: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=6160 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647709142 16/2/2020 22:13 e0529 data18_13TeV:data18_13TeV.0034926 modificationHost: e0529 pilot:::1151 File transfer timed out during stage-in: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0150._0003.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=6160 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4648000494 17/2/2020 11:08 e0849 data18_13TeV:data18_13TeV.0034926 modificationHost: e0849 pilot:::1324 Service not available at the moment: user.gangarbt:269a439e-99da-4c89-8e42-bfb309250fc7_36654.1.job.log.tgz to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0849 reports could not open connection to se.reef.man.poznan.pl:844 ddm:::200 expected output 269a439e-99da-4c89-8e42-bfb309250fc7_36654.1.job.log.tgz is missing in pilot XML 4646827078 15/2/2020 12:58 e0020 data18_13TeV:data18_13TeV.0034926 modificationHost: e0020 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0159._0001.1: checksum (catalog)=b3e8431e != checksum (local)=db2582fe',)]:failed to transfer files using copytools=[u'rucio'] 4646983033 15/2/2020 15:10 e0034 data18_13TeV:data18_13TeV.0034926 modificationHost: e0034 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0159._0001.1: checksum (catalog)=b3e8431e != checksum (local)=9094ddc3',)]:failed to transfer files using copytools=[u'rucio'] 4647199450 16/2/2020 1:42 e0265 data18_13TeV:data18_13TeV.0034926 modificationHost: e0265 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0159._0001.1: checksum (catalog)=b3e8431e != checksum (local)=ea0a7d73',)]:failed to transfer files using copytools=[u'rucio'] 4646746134 15/2/2020 9:19 e0573 data18_13TeV:data18_13TeV.0034926 modificationHost: e0573 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0159._0001.1: checksum (catalog)=b3e8431e != checksum (local)=6a18ab17',)]:failed to transfer files using copytools=[u'rucio'] 4642544216 10/2/2020 23:42 e0672 data18_13TeV:data18_13TeV.0034926 modificationHost: e0672 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1: checksum (catalog)=ad59aca0 != checksum (local)=c4b33848',)]:failed to transfer files using copytools=[u'rucio'] 4642732035 11/2/2020 1:58 e0605 data18_13TeV:data18_13TeV.0034926 modificationHost: e0605 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1: checksum (catalog)=ad59aca0 != checksum (local)=53879ff4',)]:failed to transfer files using copytools=[u'rucio'] 4642859096 11/2/2020 4:18 e0320 data18_13TeV:data18_13TeV.0034926 modificationHost: e0320 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1: checksum (catalog)=ad59aca0 != checksum (local)=0a4bc41b',)]:failed to transfer files using copytools=[u'rucio'] 4642950251 11/2/2020 6:29 e0605 data18_13TeV:data18_13TeV.0034926 modificationHost: e0605 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1: checksum (catalog)=ad59aca0 != checksum (local)=30fba9d4',)]:failed to transfer files using copytools=[u'rucio'] 4643058965 11/2/2020 8:53 e0025 data18_13TeV:data18_13TeV.0034926 modificationHost: e0025 taskBuffer:::300 The worker was finished while the job was running : None 4643177397 11/2/2020 11:17 e0126 data18_13TeV:data18_13TeV.0034926 modificationHost: e0126 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4643217114 11/2/2020 12:30 e0341 data18_13TeV:data18_13TeV.0034926 modificationHost: e0341 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4640229617 8/2/2020 17:43 e0552 data18_13TeV:data18_13TeV.0034926 modificationHost: e0552 pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] 4640565177 9/2/2020 3:05 e1013 data18_13TeV:data18_13TeV.0034926 modificationHost: e1013 pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0154._0001.1: checksum (catalog)=ad59aca0 != checksum (local)=1db6dbbe',)]:failed to transfer files using copytools=[u'rucio'] 4633853997 3/2/2020 10:03 e0824 data18_13TeV:data18_13TeV.0034926 modificationHost: e0824 jobDispatcher:::100 lost heartbeat : 2020-02-03 11:59:48 4633298968 2/2/2020 23:39 e0334 data18_13TeV:data18_13TeV.0034926 modificationHost: e0334 jobDispatcher:::100 lost heartbeat : 2020-02-02 23:39:14 4633517229 3/2/2020 2:26 e0300 data18_13TeV:data18_13TeV.0034926 modificationHost: e0300 jobDispatcher:::100 lost heartbeat : 2020-02-03 02:26:01 4633651453 3/2/2020 6:13 e0702 data18_13TeV:data18_13TeV.0034926 modificationHost: e0702 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4630936976 1/2/2020 10:38 e0145 data18_13TeV:data18_13TeV.0034926 modificationHost: e0145 jobDispatcher:::100 lost heartbeat : 2020-02-01 11:09:37 4631176115 1/2/2020 13:19 e0006 data18_13TeV:data18_13TeV.0034926 modificationHost: e0006 jobDispatcher:::100 lost heartbeat : 2020-02-01 13:19:58 4631433258 1/2/2020 17:43 e0038 data18_13TeV:data18_13TeV.0034926 modificationHost: e0038 jobDispatcher:::100 lost heartbeat : 2020-02-01 17:44:02 4631769079 1/2/2020 22:01 e0057 data18_13TeV:data18_13TeV.0034926 modificationHost: e0057 jobDispatcher:::100 lost heartbeat : 2020-02-02 00:56:21 4653698821 24/2/2020 14:30 e0591 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0591 jobDispatcher:::100 lost heartbeat : 2020-02-24 14:31:01 4653893874 24/2/2020 17:56 e0317 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0317 jobDispatcher:::100 lost heartbeat : 2020-02-24 17:56:20 4654111522 25/2/2020 2:01 e0317 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0317 jobDispatcher:::100 lost heartbeat : 2020-02-25 02:01:31 4648289265 17/2/2020 19:35 e0824 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0824 pilot:::1137 Failed to stage-out file: user.gangarbt:user.gangarbt.39d67891-6a53-4f5f-84d7-f360e878db25_13382.job.log.tgz to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output user.gangarbt.39d67891-6a53-4f5f-84d7-f360e878db25_13382.job.log.tgz is missing in pilot XML 4648313747 17/2/2020 20:15 e0611 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0611 pilot:::1137 Failed to stage-out file: user.gangarbt:output.1.04620aaa-3b4f-41f3-83e6-b8c88236258d_28546.pool.root to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4648351986 17/2/2020 21:14 e0179 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0179 pilot:::1137 Failed to stage-out file: user.gangarbt:bac4336d-d86b-4c0d-a6bd-2ffcd73b3868_76260.1.job.log.tgz to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output bac4336d-d86b-4c0d-a6bd-2ffcd73b3868_76260.1.job.log.tgz is missing in pilot XML 4648548315 18/2/2020 2:46 e0507 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0507 pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] 4647673738 16/2/2020 20:05 e0185 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0185 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647687204 16/2/2020 20:39 e0072 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0072 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647696331 16/2/2020 21:07 e0125 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0125 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647707053 16/2/2020 21:37 e0265 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0265 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647714800 16/2/2020 22:04 e0345 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0345 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647724977 16/2/2020 22:38 e0038 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0038 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4647195156 16/2/2020 1:08 e0197 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0197 pilot:::1324 Service not available at the moment: user.gangarbt:output.1.7e401d2e-8dae-4041-9640-c368aa24655e_47770.pool.root to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0197 reports could not open connection to se.reef.man.poznan.p 4647315369 16/2/2020 6:34 e0020 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0020 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4646756992 15/2/2020 9:33 e0746 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0746 pilot:::1324 Service not available at the moment: user.gangarbt:fe871e11-c60f-4831-bc39-8789f3f05da0_67888.1.job.log.tgz to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0746 reports could not open connection to se.reef.man.poznan.pl:844 ddm:::200 expected output fe871e11-c60f-4831-bc39-8789f3f05da0_67888.1.job.log.tgz is missing in pilot XML 4646825655 15/2/2020 12:41 e0521 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0521 pilot:::1137 Failed to stage-out file: user.gangarbt:02666c8d-80e1-4cc6-9b28-f0b20ce442fb_54008.1.job.log.tgz to PSNC_SCRATCHDISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 02666c8d-80e1-4cc6-9b28-f0b20ce442fb_54008.1.job.log.tgz is missing in pilot XML 4647010112 15/2/2020 15:46 e0612 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0612 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:AOD.05536542._000001.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=1052 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4644783751 13/2/2020 1:12 e0047 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0047 pilot:::1324 Service not available at the moment: user.gangarbt:cd0676dc-4d87-45f2-a254-1139db51883e_31743.1.job.log.tgz to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0047 reports could not open connection to se.reef.man.poznan.pl:844 ddm:::200 expected output cd0676dc-4d87-45f2-a254-1139db51883e_31743.1.job.log.tgz is missing in pilot XML 4644265882 12/2/2020 16:12 e0010 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0010 pilot:::1324 Service not available at the moment: user.gangarbt:ff70afd1-1188-40fb-9ac7-31dce2640200_5148.1.job.log.tgz to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0010 reports could not open connection to se.reef.man.poznan.pl:8446 ddm:::200 expected output ff70afd1-1188-40fb-9ac7-31dce2640200_5148.1.job.log.tgz is missing in pilot XML 4642804362 11/2/2020 2:48 e0352 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0352 pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 3ee8978c-aee9-4b43-8952-de7c04efb143_93778.1.job.log.tgz is missing in pilot XML 4642925948 11/2/2020 6:07 e0420 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0420 jobDispatcher:::100 lost heartbeat : 2020-02-11 07:52:58 4643136279 11/2/2020 10:40 e0632 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0632 jobDispatcher:::100 lost heartbeat : 2020-02-11 12:16:06 4640244841 8/2/2020 17:41 e0948 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0948 pilot:::1324 Service not available at the moment: user.gangarbt:output.1.d31658e0-5bb6-4858-b07d-d2f66ea1fa98_26373.pool.root to PSNC_SCRATCHDISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0948 reports could not open connection to se.reef.man.poznan.p 4635230908 4/2/2020 12:28 e0504 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0504 ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4635765746 4/2/2020 21:54 e0184 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0184 pilot:::1099 Failed to stage-in file: [ImportError('No module named rucio.client',)]:failed to transfer files using copytools=[u'rucio'] 4636716015 5/2/2020 10:59 e0025 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0025 taskBuffer:::300 The worker was finished while the job was running : None 4634038168 3/2/2020 15:13 e0124 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0124 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4631332707 1/2/2020 14:45 e0273 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0273 jobDispatcher:::100 lost heartbeat : 2020-02-01 14:45:45 4631497012 1/2/2020 18:20 e0306 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0306 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4632045394 2/2/2020 1:47 e0005 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0005 jobDispatcher:::100 lost heartbeat : 2020-02-02 02:50:00 4632251474 2/2/2020 6:03 e0032 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: e0032 jobDispatcher:::100 lost heartbeat : 2020-02-02 06:03:21 PSNC ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4653772056 24/2/2020 14:31 e0591 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0591 jobDispatcher:::100 lost heartbeat : 2020-02-24 14:31:33 4653897401 24/2/2020 17:56 e0317 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0317 jobDispatcher:::100 lost heartbeat : 2020-02-24 17:56:03 4654113867 25/2/2020 1:56 e0144 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0144 jobDispatcher:::100 lost heartbeat : 2020-02-25 04:14:48 4648287480 17/2/2020 20:22 e0184 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0184 pilot:::1137 Failed to stage-out file: hc_test:Hits.hc_20161881.PSNC.b975e6d3-d510-4e3e-a70e-73594119dbf4_69058.HITS.pool.root to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4647647900 16/2/2020 19:01 e0505 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0505 pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output c483140d-9767-4b3b-b787-65de617acc92_5742.1.job.log.tgz is missing in pilot XML 4648170045 17/2/2020 16:20 e0474 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0474 pilot:::1137 Failed to stage-out file: hc_test:5f1801ee-bd43-40ef-b96a-3eb445824d2b_9552.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 5f1801ee-bd43-40ef-b96a-3eb445824d2b_9552.1.job.log.tgz is missing in pilot XML 4648203476 17/2/2020 17:05 e0163 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0163 pilot:::1137 Failed to stage-out file: hc_test:b71047e3-d661-4bec-ab59-2c731d355e40_45187.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output b71047e3-d661-4bec-ab59-2c731d355e40_45187.1.job.log.tgz is missing in pilot XML 4648223683 17/2/2020 17:43 e0459 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0459 pilot:::1137 Failed to stage-out file: hc_test:output.1.f29ea869-55a9-4787-a96a-ff72b138ffaa_10192.pool.root to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4643323872 11/2/2020 14:54 e0352 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0352 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4642186084 10/2/2020 18:34 e0274 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0274 pilot:::1137 Failed to stage-out file: hc_test:output.1.a100b6f2-08fa-425b-b54f-f8a4d46340ff_462.pool.root to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4636689993 5/2/2020 10:55 e0025 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0025 taskBuffer:::300 The worker was finished while the job was running : None 4633892942 3/2/2020 11:42 e0041 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0041 jobDispatcher:::100 lost heartbeat : 2020-02-03 11:42:41 4631324165 1/2/2020 14:45 e0363 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0363 jobDispatcher:::100 lost heartbeat : 2020-02-01 14:47:36 4631497013 1/2/2020 17:44 e0038 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0038 jobDispatcher:::100 lost heartbeat : 2020-02-01 17:44:05 4631770813 1/2/2020 21:16 e0058 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0058 jobDispatcher:::100 lost heartbeat : 2020-02-01 21:16:18 4653654362 24/2/2020 11:55 e0565 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0565 jobDispatcher:::100 lost heartbeat : 2020-02-24 11:55:29 4653790410 24/2/2020 14:32 e0591 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0591 jobDispatcher:::100 lost heartbeat : 2020-02-24 14:32:24 4653896431 24/2/2020 17:55 e0317 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0317 jobDispatcher:::100 lost heartbeat : 2020-02-24 17:55:42 4654091642 25/2/2020 0:54 e0411 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0411 jobDispatcher:::100 lost heartbeat : 2020-02-25 02:23:07 4648460275 18/2/2020 1:27 e0584 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0584 pilot:::1324 Service not available at the moment: hc_test:output.1.43bf8742-79a4-4305-b814-fa2b873a8dc6_16508.pool.root to PSNC_DATADISK, srm-ifce err: Communication error on send, err: [SE][Ls][] httpg://se.reef.man.poznan.pl:8446/srm/managerv2: CGSI-gSOAP running on e0584 reports could not open connection to se.reef.man.poznan.pl:8446',) 4648597157 18/2/2020 4:33 e0824 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0824 pilot:::1137 Failed to stage-out file: hc_test:5a0da208-48a6-495a-8296-f55359a8dcec_21956.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 5a0da208-48a6-495a-8296-f55359a8dcec_21956.1.job.log.tgz is missing in pilot XML 4648149131 17/2/2020 15:57 e0984 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0984 pilot:::1137 Failed to stage-out file: hc_test:265c8511-32c0-40ee-b6a4-1a688ddca03d_513.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 265c8511-32c0-40ee-b6a4-1a688ddca03d_513.1.job.log.tgz is missing in pilot XML 4648237989 17/2/2020 20:15 e0611 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0611 pilot:::1137 Failed to stage-out file: hc_test:output.1.d58063b0-62cd-4dbd-9e22-25711955c196_23860.pool.root to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4647713930 16/2/2020 22:13 e0272 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0272 pilot:::1137 Failed to stage-out file: hc_test:1b4b0abe-8955-494a-88ef-a551462fb3df_23481.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 1b4b0abe-8955-494a-88ef-a551462fb3df_23481.1.job.log.tgz is missing in pilot XML 4642976202 11/2/2020 6:51 e0389 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0389 jobDispatcher:::100 lost heartbeat : 2020-02-11 09:00:24 4640705810 9/2/2020 6:53 e0182 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0182 pilot:::1137 Failed to stage-out file: hc_test:output.1.e7ca7365-c32d-4b0a-8f02-9da24b7f1ff6_48025.pool.root to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4636699285 5/2/2020 10:55 e0025 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0025 jobDispatcher:::100 lost heartbeat : 2020-02-05 10:55:50 4635750743 4/2/2020 21:54 e0184 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0184 pilot:::1099 Failed to stage-in file: [ImportError('No module named rucio.client',)]:failed to transfer files using copytools=[u'rucio'] 4633714392 3/2/2020 7:19 e0041 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0041 jobDispatcher:::100 lost heartbeat : 2020-02-03 07:19:59 4633862514 3/2/2020 11:01 e0300 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0300 jobDispatcher:::100 lost heartbeat : 2020-02-03 11:01:58 4634006369 3/2/2020 13:45 e0281 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0281 jobDispatcher:::100 lost heartbeat : 2020-02-03 13:45:44 4633212703 2/2/2020 22:27 e0315 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0315 jobDispatcher:::100 lost heartbeat : 2020-02-02 22:57:03 4633482330 3/2/2020 1:43 e0329 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0329 jobDispatcher:::100 lost heartbeat : 2020-02-03 01:43:34 4630281732 1/2/2020 1:56 e0402 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0402 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000028.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630463078 1/2/2020 3:55 e0018 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0018 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.6ce0fd57-b4f2-4e55-a75f-a68e9c66503c_27177.pool.root to PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=324 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630633378 1/2/2020 5:48 e0409 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0409 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000028.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=386 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4648192695 17/2/2020 16:50 e0163 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0163 pilot:::1137 Failed to stage-out file: hc_test:c07ecd09-5e99-4746-850f-431d8667f4e9_7435.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output c07ecd09-5e99-4746-850f-431d8667f4e9_7435.1.job.log.tgz is missing in pilot XML 4648221447 17/2/2020 17:35 e0459 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0459 pilot:::1137 Failed to stage-out file: hc_test:12d2a06e-1247-47ed-962b-2ed52c6a05c4_95465.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 12d2a06e-1247-47ed-962b-2ed52c6a05c4_95465.1.job.log.tgz is missing in pilot XML 4648341728 17/2/2020 20:52 e1017 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e1017 pilot:::1137 Failed to stage-out file: hc_test:output.1.3302cc9d-356d-44b0-aa92-c983c476a537_25812.pool.root to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] 4647191275 16/2/2020 1:30 e0181 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0181 pilot:::1137 Failed to stage-out file: hc_test:3efa2150-4e1e-405c-8443-45d01cb46fff_41562.1.job.log.tgz to PSNC_DATADISK, Checksum not validated',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 3efa2150-4e1e-405c-8443-45d01cb46fff_41562.1.job.log.tgz is missing in pilot XML 4642923871 11/2/2020 5:35 e0815 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0815 jobDispatcher:::100 lost heartbeat : 2020-02-11 05:35:25 4643054328 11/2/2020 8:29 e0352 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0352 jobDispatcher:::100 lost heartbeat : 2020-02-11 14:02:31 4640160223 8/2/2020 16:14 e0136 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0136 pilot:::1137 Failed to stage-out file: [TypeError('not all arguments converted during string formatting',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 4ab92fdd-05b6-4597-a3c0-0dde8db06e0b_34393.1.job.log.tgz is missing in pilot XML 4636676632 5/2/2020 10:50 e0025 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0025 taskBuffer:::300 The worker was finished while the job was running : None 4633699902 3/2/2020 6:46 e0936 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0936 jobDispatcher:::100 lost heartbeat : 2020-02-03 06:46:31 4633832114 3/2/2020 9:25 e0632 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0632 jobDispatcher:::100 lost heartbeat : 2020-02-03 10:03:41 4633958309 3/2/2020 12:29 e0124 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0124 jobDispatcher:::100 lost heartbeat : 2020-02-03 12:31:15 4632264300 2/2/2020 6:09 e0032 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0032 taskBuffer:::300 The worker was finished while the job was running : None 4632285649 2/2/2020 7:03 e0032 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0032 jobDispatcher:::100 lost heartbeat : 2020-02-02 07:03:57 4630523558 1/2/2020 4:26 e0823 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0823 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=383 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630555374 1/2/2020 4:53 e0659 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0659 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=383 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630598953 1/2/2020 5:19 e0989 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0989 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=383 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630628423 1/2/2020 5:48 e0351 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0351 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=383 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630657980 1/2/2020 6:08 e0273 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0273 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=383 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630686863 1/2/2020 6:32 e0065 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0065 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=383 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630878565 1/2/2020 9:33 e0433 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0433 pilot:::1151 File transfer timed out during stage-in: u' mc15_13TeV:EVNT.04972714._000039.pool.root.1 from PSNC_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=383 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4630925595 1/2/2020 10:06 e0189 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0189 jobDispatcher:::100 lost heartbeat : 2020-02-01 10:38:35 4631184847 1/2/2020 13:24 e0124 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0124 jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4631308992 1/2/2020 14:45 e0363 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0363 jobDispatcher:::100 lost heartbeat : 2020-02-01 14:47:43 4631499300 1/2/2020 17:44 e0038 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0038 jobDispatcher:::100 lost heartbeat : 2020-02-01 17:44:06 4631773581 1/2/2020 21:16 e0058 hc_test:mc15_13TeV.361106.PowhegP modificationHost: e0058 jobDispatcher:::100 lost heartbeat : 2020-02-01 21:16:17 ====== UNI-FREIBURG ANALY_FREIBURG ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4658236860 29/2/2020 13:18 n4562.bfg.uni data18_13TeV:data18_13TeV.0034926 modificationHost: n4562.bfg.uni-freiburg.de taskBuffer:::300 The worker was finished while the job was running : None 4658354945 29/2/2020 16:49 n240108.bfg.u data18_13TeV:data18_13TeV.0034926 modificationHost: n240108.bfg.uni-freiburg.de pilot:::1180 Globus system error during stage-in: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0159._0001.1 from UNI-FREIBURG_DATADISK, Could not open source: an end-of-file was reached globus_xio: An end of file occurred',)]:failed to transfer files using copytools=[u'rucio'] 4645293881 13/2/2020 17:15 n240106.bfg.u data18_13TeV:data18_13TeV.0034926 modificationHost: n240106.bfg.uni-freiburg.de 4644852042 13/2/2020 3:20 n241107.bfg.u data18_13TeV:data18_13TeV.0034926 modificationHost: n241107.bfg.uni-freiburg.de pilot:::1322 Failed to get replicas from Rucio: HTTPSConnectionPool(host='voatlasrucio-server-prod.cern.ch', port=443): Read timed out. (read timeout=600)",)]:failed to transfer files using copytools=[u'rucio'] 4635125642 4/2/2020 11:58 n231116.bfg.u data18_13TeV:data18_13TeV.0034926 modificationHost: n231116.bfg.uni-freiburg.de ddm:::200 Could not add files to DDM: Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' Details: Problem validating attachments : u'NULL' does not match '^(\{){0,1}[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}(\}){0,1}$' 4632911991 2/2/2020 18:04 n241105.bfg.u data18_13TeV:data18_13TeV.0034926 modificationHost: n241105.bfg.uni-freiburg.de pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0159._0001.1: checksum (catalog)=b3e8431e != checksum (local)=7bb56852',)]:failed to transfer files using copytools=[u'rucio'] 4633473995 3/2/2020 1:28 n241102.bfg.u data18_13TeV:data18_13TeV.0034926 modificationHost: n241102.bfg.uni-freiburg.de jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-03 01:40:13 sup:::2 worker already finished at 2020-02-03 01:53:20 with None 4633749493 3/2/2020 7:33 n4587.bfg.uni data18_13TeV:data18_13TeV.0034926 modificationHost: n4587.bfg.uni-freiburg.de pilot:::1171 adler32 mismatch on input file: checksum verification failed for LFN=data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0159._0001.1: checksum (catalog)=b3e8431e != checksum (local)=2a7649d4',)]:failed to transfer files using copytools=[u'rucio'] 4631032231 1/2/2020 11:38 n241104.bfg.u data18_13TeV:data18_13TeV.0034926 modificationHost: n241104.bfg.uni-freiburg.de pilot:::1099 Failed to stage-in file: u' data18_13TeV:data18_13TeV.00349263.physics_Main.merge.AOD.f937_m1972._lb0151._0001.1 from UNI-FREIBURG_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4658633884 1/3/2020 1:35 n240106.bfg.u mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n240106.bfg.uni-freiburg.de pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4655951014 27/2/2020 3:01 n231114.bfg.u mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n231114.bfg.uni-freiburg.de 4647809488 17/2/2020 2:41 n4572.bfg.uni mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n4572.bfg.uni-freiburg.de taskBuffer:::300 The worker was cancelled while the job was starting : Condor HoldReason: CREAM error: Transfer failed: globus_ftp_client: the server responded with an error 500 500-Command failed. : globus_l_gfs_file_open failed. 500-globus_xio: Unable to open file /var/cream_sandbox/atlpl/CN_Robot__ATLAS_Pilot1_C sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Transfer failed: globus_ftp_client: the server responded with an error 500 500-Command failed. : globus_l_gfs_file_open failed. 500-globus_xio: Unable to open file /var/cream_sandbox/atlpl/CN_Robot_ 4644618794 12/2/2020 21:30 n4563.bfg.uni mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n4563.bfg.uni-freiburg.de pilot:::1322 Failed to get replicas from Rucio:Database exception. Details: QueuePool limit of size 3 overflow 3 reached, connection timed out, timeout 600 (Background on this error at: http://sqlalche.me/e/3o7r)',)]:failed to transfer files using copytools=[u'rucio'] 4645279949 13/2/2020 16:54 n231103.bfg.u mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n231103.bfg.uni-freiburg.de 4631801705 1/2/2020 21:15 n230109.bfg.u mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: n230109.bfg.uni-freiburg.de jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-01 21:16:33 sup:::2 worker already finished at 2020-02-01 21:27:36 with None UNI-FREIBURG ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4658497519 29/2/2020 21:21 n241105.bfg.u hc_test:mc15_13TeV.361106.PowhegP modificationHost: n241105.bfg.uni-freiburg.de pilot:::1180 Globus system error during stage-in: u' mc15_13TeV:EVNT.04972714._000027.pool.root.1 from UNI-FREIBURG_DATADISK, Could not open source: an end-of-file was reached globus_xio: An end of file occurred',)]:failed to transfer files using copytools=[u'rucio'] 4650390518 19/2/2020 22:19 n241112.bfg.u hc_test:mc15_13TeV.361106.PowhegP modificationHost: n241112.bfg.uni-freiburg.de pilot:::1137 Failed to stage-out file: hc_test:output.1.4efd6c1c-f0cc-4edd-ac4a-1710cc3e5312_91836.pool.root to UNI-FREIBURG_DATADISK, no error information passed (http status code: 503 ('service_unavailable', 'unavailable'))",)]:failed to transfer files using copytools=[u'rucio'] 4649413237 18/2/2020 20:16 n4563.bfg.uni hc_test:mc15_13TeV.361106.PowhegP modificationHost: n4563.bfg.uni-freiburg.de pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000022.pool.root.1 from UNI-FREIBURG_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4643636103 11/2/2020 22:39 n4575.bfg.uni hc_test:mc15_13TeV.361106.PowhegP modificationHost: n4575.bfg.uni-freiburg.de pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000021.pool.root.1 from UNI-FREIBURG_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4639138884 7/2/2020 22:15 n4586.bfg.uni hc_test:mc15_13TeV.361106.PowhegP modificationHost: n4586.bfg.uni-freiburg.de taskBuffer:::300 The worker was finished while the job was running : None 4631018048 1/2/2020 11:42 n4566.bfg.uni hc_test:mc15_13TeV.361106.PowhegP modificationHost: n4566.bfg.uni-freiburg.de pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000024.pool.root.1 from UNI-FREIBURG_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4658482263 29/2/2020 20:58 n241104.bfg.u hc_test:mc15_13TeV.361106.PowhegP modificationHost: n241104.bfg.uni-freiburg.de pilot:::1180 Globus system error during stage-in: u' mc15_13TeV:EVNT.04972714._000026.pool.root.1 from UNI-FREIBURG_DATADISK, Could not open source: an end-of-file was reached globus_xio: An end of file occurred',)]:failed to transfer files using copytools=[u'rucio'] 4644898927 13/2/2020 4:51 n240103.bfg.u hc_test:mc15_13TeV.361106.PowhegP modificationHost: n240103.bfg.uni-freiburg.de pilot:::1099 Failed to stage-in file: u' mc15_13TeV:EVNT.04972714._000029.pool.root.1 from UNI-FREIBURG_DATADISK, None of the requested files have been downloaded.',)]:failed to transfer files using copytools=[u'rucio'] 4632922455 2/2/2020 18:15 n4581.bfg.uni hc_test:mc15_13TeV.361106.PowhegP modificationHost: n4581.bfg.uni-freiburg.de taskBuffer:::300 The worker was finished while the job was running : None 4633045609 2/2/2020 20:03 n231106.bfg.u hc_test:mc15_13TeV.361106.PowhegP modificationHost: n231106.bfg.uni-freiburg.de jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 20:40:50 sup:::2 worker already finished at 2020-02-02 20:53:28 with None UNI-FREIBURG_MCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4643434604 11/2/2020 17:38 n4577.bfg.uni hc_test:mc15_13TeV.361106.PowhegP modificationHost: n4577.bfg.uni-freiburg.de taskBuffer:::300 The worker was cancelled while the job was running : Condor HoldReason: CREAM error: Transfer failed: globus_ftp_client: the server responded with an error 500 500-Command failed. : globus_l_gfs_file_open failed. 500-globus_xio: Unable to open file /var/cream_sandbox/atlpr/CN_Robot__ATLAS_Pilot1_CN sup:::9000 Diag from worker : Condor HoldReason: CREAM error: Transfer failed: globus_ftp_client: the server responded with an error 500 500-Command failed. : globus_l_gfs_file_open failed. 500-globus_xio: Unable to open file /var/cream_sandbox/atlpr/CN_Robot_ ====== wuppertalprod ANALY_wuppertalprod ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4658998840 1/3/2020 17:34 wnfg005 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: wnfg005 taskBuffer:::300 The worker was finished while the job was running : None 4641597248 10/2/2020 10:00 wn049 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: wn049 taskBuffer:::300 The worker was finished while the job was running : None 4636481777 5/2/2020 6:04 aipanda404.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda404.cern.ch jobDispatcher:::102 Sent job didn't receive reply from pilot within 30 min 4658547651 29/2/2020 22:57 wnfg044 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: wnfg044 pilot:::1099 Failed to stage-in file: [CannotAuthenticate("x509 authentication failed for account=pilot with identity={'client_proxy': '/tmp/pltatlas099/TRNLDm1EQSwnsBfJnpimw0EmABFKDmABFKDmY5FKDmABFKDmCxNckn/user.proxy'}",)]:failed to transfer files using copytools=[u'rucio'] 4653172565 23/2/2020 11:44 aipanda403.ce mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: aipanda403.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Unknown error sup:::9000 Diag from worker : Unknown error 4645522649 13/2/2020 22:57 wnfg046 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: wnfg046 pilot:::1099 Failed to stage-in file: [CannotAuthenticate("x509 authentication failed for account=pilot with identity={'client_proxy': '/tmp/pltatlas099/0nPMDm4ccMwnsBfJnpimw0EmABFKDmABFKDmYuGKDmABFKDmB1tasm/user.proxy'}",)]:failed to transfer files using copytools=[u'rucio'] 4645293420 13/2/2020 17:15 wn127 mc15_13TeV:mc15_13TeV.423202.Pyth modificationHost: wn127 taskBuffer:::300 The worker was finished while the job was running : None wuppertalprod_UCORE ----- panda ID Error date Node Input dataset Error message ---------------------------------------------------------------------------------------------------- 4653172717 23/2/2020 11:45 aipanda404.ce hc_test:mc15_13TeV.361106.PowhegP modificationHost: aipanda404.cern.ch taskBuffer:::300 The worker was failed while the job was starting : Unknown error sup:::9000 Diag from worker : Unknown error 4645333375 13/2/2020 18:25 wn053 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wn053 taskBuffer:::300 The worker was finished while the job was running : None 4643888392 12/2/2020 7:35 wnfg045 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wnfg045 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.2e2b3b12-bae2-4454-bee6-b0413a1f9acb_79473.pool.root to WUPPERTALPROD_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=313 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4640127051 8/2/2020 14:31 wnfg029 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wnfg029 pilot:::1152 File transfer timed out during stage-out: hc_test:2e273b54-a43f-445b-a255-4a12404e44f4_28754.1.job.log.tgz to WUPPERTALPROD_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=310 seconds',)]:failed to transfer files using copytools=[u'rucio'] ddm:::200 expected output 2e273b54-a43f-445b-a255-4a12404e44f4_28754.1.job.log.tgz is missing in pilot XML 4648274509 17/2/2020 18:56 wnfg060 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wnfg060 jobDispatcher:::100 lost heartbeat : 2020-02-17 18:57:10 4645300960 13/2/2020 17:28 wn125 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wn125 taskBuffer:::300 The worker was finished while the job was running : None 4639331930 8/2/2020 1:00 wnfg053 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wnfg053 pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.362d4b6a-9bea-479b-beef-1f565eeab628_85195.pool.root to WUPPERTALPROD_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=323 seconds',)]:failed to transfer files using copytools=[u'rucio'] 4658992045 1/3/2020 17:24 wnfg063 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wnfg063 taskBuffer:::300 The worker was finished while the job was running : None 4632915959 2/2/2020 18:07 wnfg041 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wnfg041 taskBuffer:::300 The worker was finished while the job was running : None 4633011266 2/2/2020 19:37 wnfg051 hc_test:mc15_13TeV.361106.PowhegP modificationHost: wnfg051 jobDispatcher:::103 timeout in holding : last heartbeat at 2020-02-02 20:02:49 sup:::2 worker already finished at 2020-02-02 20:16:45 with None ======