Jobsub ID 33931.0@dunegpschedd02.fnal.gov

Jobsub ID	33931.0@dunegpschedd02.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 05:57:08
Site	CH_UNIBE-LHEP
Entry	OSG_CH_UNIBE_LHEP_ce02
Last heartbeat	2025-09-05 06:05:25
From worker node	Hostname	wn-1-10.local
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 05:58:45
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_310_20231028T032912Z_gen_g4_detsim_hitreco__20240503T050227Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (378s)
	CPU time	1m (62s = 16%)
	Max RSS bytes	1028595712 (980 MiB)
Outputting started	2025-09-05 06:05:04
Output files
Finished	2025-09-05 06:05:25
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

128 at 05-Sep-2025 08:02:55 CEST
Begin processing the 129th record. run: 1500 subRun: 1 event: 62129 at 05-Sep-2025 08:02:57 CEST
Begin processing the 130th record. run: 1500 subRun: 1 event: 62130 at 05-Sep-2025 08:02:58 CEST
Begin processing the 131st record. run: 1500 subRun: 1 event: 62131 at 05-Sep-2025 08:02:59 CEST
Begin processing the 132nd record. run: 1500 subRun: 1 event: 62132 at 05-Sep-2025 08:03:01 CEST
Begin processing the 133rd record. run: 1500 subRun: 1 event: 62133 at 05-Sep-2025 08:03:03 CEST
Begin processing the 134th record. run: 1500 subRun: 1 event: 62134 at 05-Sep-2025 08:03:04 CEST
Begin processing the 135th record. run: 1500 subRun: 1 event: 62135 at 05-Sep-2025 08:03:06 CEST
Begin processing the 136th record. run: 1500 subRun: 1 event: 62136 at 05-Sep-2025 08:03:07 CEST
Begin processing the 137th record. run: 1500 subRun: 1 event: 62137 at 05-Sep-2025 08:03:11 CEST
Begin processing the 138th record. run: 1500 subRun: 1 event: 62138 at 05-Sep-2025 08:03:12 CEST
Begin processing the 139th record. run: 1500 subRun: 1 event: 62139 at 05-Sep-2025 08:03:14 CEST
Begin processing the 140th record. run: 1500 subRun: 1 event: 62140 at 05-Sep-2025 08:03:15 CEST
Begin processing the 141st record. run: 1500 subRun: 1 event: 62141 at 05-Sep-2025 08:03:16 CEST
Begin processing the 142nd record. run: 1500 subRun: 1 event: 62142 at 05-Sep-2025 08:03:18 CEST
Begin processing the 143rd record. run: 1500 subRun: 1 event: 62143 at 05-Sep-2025 08:03:18 CEST
Begin processing the 144th record. run: 1500 subRun: 1 event: 62144 at 05-Sep-2025 08:03:20 CEST
Begin processing the 145th record. run: 1500 subRun: 1 event: 62145 at 05-Sep-2025 08:03:21 CEST
Begin processing the 146th record. run: 1500 subRun: 1 event: 62146 at 05-Sep-2025 08:03:23 CEST
Begin processing the 147th record. run: 1500 subRun: 1 event: 62147 at 05-Sep-2025 08:03:24 CEST
Begin processing the 148th record. run: 1500 subRun: 1 event: 62148 at 05-Sep-2025 08:03:26 CEST
Begin processing the 149th record. run: 1500 subRun: 1 event: 62149 at 05-Sep-2025 08:03:27 CEST
Begin processing the 150th record. run: 1500 subRun: 1 event: 62150 at 05-Sep-2025 08:03:29 CEST
Begin processing the 151st record. run: 1500 subRun: 1 event: 62151 at 05-Sep-2025 08:03:34 CEST
Begin processing the 152nd record. run: 1500 subRun: 1 event: 62152 at 05-Sep-2025 08:03:36 CEST
Begin processing the 153rd record. run: 1500 subRun: 1 event: 62153 at 05-Sep-2025 08:03:38 CEST
Begin processing the 154th record. run: 1500 subRun: 1 event: 62154 at 05-Sep-2025 08:03:40 CEST
Begin processing the 155th record. run: 1500 subRun: 1 event: 62155 at 05-Sep-2025 08:03:42 CEST
Begin processing the 156th record. run: 1500 subRun: 1 event: 62156 at 05-Sep-2025 08:03:43 CEST
Begin processing the 157th record. run: 1500 subRun: 1 event: 62157 at 05-Sep-2025 08:03:45 CEST
Begin processing the 158th record. run: 1500 subRun: 1 event: 62158 at 05-Sep-2025 08:03:46 CEST
Begin processing the 159th record. run: 1500 subRun: 1 event: 62159 at 05-Sep-2025 08:03:47 CEST
Begin processing the 160th record. run: 1500 subRun: 1 event: 62160 at 05-Sep-2025 08:03:53 CEST
Begin processing the 161st record. run: 1500 subRun: 1 event: 62161 at 05-Sep-2025 08:03:55 CEST
Begin processing the 162nd record. run: 1500 subRun: 1 event: 62162 at 05-Sep-2025 08:03:56 CEST
Begin processing the 163rd record. run: 1500 subRun: 1 event: 62163 at 05-Sep-2025 08:03:58 CEST
Begin processing the 164th record. run: 1500 subRun: 1 event: 62164 at 05-Sep-2025 08:03:59 CEST
Begin processing the 165th record. run: 1500 subRun: 1 event: 62165 at 05-Sep-2025 08:04:00 CEST
Begin processing the 166th record. run: 1500 subRun: 1 event: 62166 at 05-Sep-2025 08:04:03 CEST
Begin processing the 167th record. run: 1500 subRun: 1 event: 62167 at 05-Sep-2025 08:04:05 CEST
Begin processing the 168th record. run: 1500 subRun: 1 event: 62168 at 05-Sep-2025 08:04:06 CEST
Begin processing the 169th record. run: 1500 subRun: 1 event: 62169 at 05-Sep-2025 08:04:08 CEST
Begin processing the 170th record. run: 1500 subRun: 1 event: 62170 at 05-Sep-2025 08:04:09 CEST
Begin processing the 171st record. run: 1500 subRun: 1 event: 62171 at 05-Sep-2025 08:04:10 CEST
Begin processing the 172nd record. run: 1500 subRun: 1 event: 62172 at 05-Sep-2025 08:04:12 CEST
Begin processing the 173rd record. run: 1500 subRun: 1 event: 62173 at 05-Sep-2025 08:04:14 CEST
Begin processing the 174th record. run: 1500 subRun: 1 event: 62174 at 05-Sep-2025 08:04:15 CEST
Begin processing the 175th record. run: 1500 subRun: 1 event: 62175 at 05-Sep-2025 08:04:17 CEST
Begin processing the 176th record. run: 1500 subRun: 1 event: 62176 at 05-Sep-2025 08:04:18 CEST
Begin processing the 177th record. run: 1500 subRun: 1 event: 62177 at 05-Sep-2025 08:04:19 CEST
Begin processing the 178th record. run: 1500 subRun: 1 event: 62178 at 05-Sep-2025 08:04:21 CEST
Begin processing the 179th record. run: 1500 subRun: 1 event: 62179 at 05-Sep-2025 08:04:22 CEST
Begin processing the 180th record. run: 1500 subRun: 1 event: 62180 at 05-Sep-2025 08:04:24 CEST
Begin processing the 181st record. run: 1500 subRun: 1 event: 62181 at 05-Sep-2025 08:04:25 CEST
Begin processing the 182nd record. run: 1500 subRun: 1 event: 62182 at 05-Sep-2025 08:04:27 CEST
Begin processing the 183rd record. run: 1500 subRun: 1 event: 62183 at 05-Sep-2025 08:04:33 CEST
Begin processing the 184th record. run: 1500 subRun: 1 event: 62184 at 05-Sep-2025 08:04:35 CEST
Begin processing the 185th record. run: 1500 subRun: 1 event: 62185 at 05-Sep-2025 08:04:35 CEST
Begin processing the 186th record. run: 1500 subRun: 1 event: 62186 at 05-Sep-2025 08:04:36 CEST
Begin processing the 187th record. run: 1500 subRun: 1 event: 62187 at 05-Sep-2025 08:04:38 CEST
Begin processing the 188th record. run: 1500 subRun: 1 event: 62188 at 05-Sep-2025 08:04:39 CEST
Begin processing the 189th record. run: 1500 subRun: 1 event: 62189 at 05-Sep-2025 08:04:41 CEST
Begin processing the 190th record. run: 1500 subRun: 1 event: 62190 at 05-Sep-2025 08:04:42 CEST
Begin processing the 191st record. run: 1500 subRun: 1 event: 62191 at 05-Sep-2025 08:04:46 CEST
Begin processing the 192nd record. run: 1500 subRun: 1 event: 62192 at 05-Sep-2025 08:04:47 CEST
Begin processing the 193rd record. run: 1500 subRun: 1 event: 62193 at 05-Sep-2025 08:04:49 CEST
Begin processing the 194th record. run: 1500 subRun: 1 event: 62194 at 05-Sep-2025 08:04:51 CEST
Begin processing the 195th record. run: 1500 subRun: 1 event: 62195 at 05-Sep-2025 08:04:51 CEST
Begin processing the 196th record. run: 1500 subRun: 1 event: 62196 at 05-Sep-2025 08:04:53 CEST
Begin processing the 197th record. run: 1500 subRun: 1 event: 62197 at 05-Sep-2025 08:04:54 CEST
Begin processing the 198th record. run: 1500 subRun: 1 event: 62198 at 05-Sep-2025 08:04:57 CEST
Begin processing the 199th record. run: 1500 subRun: 1 event: 62199 at 05-Sep-2025 08:04:59 CEST
Begin processing the 200th record. run: 1500 subRun: 1 event: 62200 at 05-Sep-2025 08:05:01 CEST
05-Sep-2025 08:05:02 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9a/10/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_310_20231028T032912Z_gen_g4_detsim_hitreco__20240503T050227Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.337616       1.31297       5.98429       1.17701      0.686372        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0179048     0.0385559     0.0757124     0.0389124     0.0133448       200    
prod:pandorapid:Chi2ParticleID                 0.105143      0.166372       0.35688      0.169414      0.0312374       200    
prod:energyreconue:EnergyReco                  0.185883      0.416635      0.762233      0.392416      0.100952        200    
prod:energyreconumu:EnergyReco                0.00178594     0.100823       1.97881      0.0659214     0.188559        200    
prod:energyreconc:EnergyReco                  0.00288401     0.0444185     0.401968      0.0310196     0.0554076       200    
[art]:TriggerResults:TriggerResultInserter     9.707e-06    1.49097e-05   4.8251e-05    1.3723e-05    4.5542e-06       200    
end_path:ana:PIDASelectionAna                  0.0208885      0.54583       3.69572      0.467437      0.422908        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7793.14 MB
  Peak resident set size usage (VmHWM): 1028.6 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 61
-rw-r--r-- 1 dune002 dune 28445 Sep  5 08:05 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_310_20231028T032912Z_gen_g4_detsim_hitreco__20240503T050227Z_reco2_pida_2025-09-05T_055849Z.root
-rw-r--r-- 1 dune002 dune 23608 Sep  5 08:05 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_310_20231028T032912Z_gen_g4_detsim_hitreco__20240503T050227Z_reco2_reco_2025-09-05T_055849Z.log
-rw-r--r-- 1 dune002 dune 11810 Sep  5 08:05 jobscript.log
-rw-r--r-- 1 dune002 dune   183 Sep  5 08:05 justin-processed-pfns.txt
-rw-r--r-- 1 dune002 dune   134 Sep  5 07:58 all-input-dids.txt
-rw-r--r-- 1 dune002 dune     0 Sep  5 07:58 debugprod.log