Jobsub ID 41578.10@dunegpschedd01.fnal.gov

Jobsub ID	41578.10@dunegpschedd01.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-06 13:30:45
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce03
Last heartbeat	2025-09-06 13:39:25
From worker node	Hostname	ccwcondor0727
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-06 13:32:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_485_20231122T052748Z_gen_g4_detsim_hitreco__20240507T172930Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (373s)
	CPU time	1m (67s = 17%)
	Max RSS bytes	1319878656 (1258 MiB)
Outputting started	2025-09-06 13:38:44
Output files
Finished	2025-09-06 13:39:25
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

CEST
Begin processing the 132nd record. run: 74254738 subRun: 1 event: 97132 at 06-Sep-2025 15:36:34 CEST
Begin processing the 133rd record. run: 74254738 subRun: 1 event: 97133 at 06-Sep-2025 15:36:37 CEST
Begin processing the 134th record. run: 74254738 subRun: 1 event: 97134 at 06-Sep-2025 15:36:38 CEST
Begin processing the 135th record. run: 74254738 subRun: 1 event: 97135 at 06-Sep-2025 15:36:40 CEST
Begin processing the 136th record. run: 74254738 subRun: 1 event: 97136 at 06-Sep-2025 15:36:41 CEST
Begin processing the 137th record. run: 74254738 subRun: 1 event: 97137 at 06-Sep-2025 15:36:47 CEST
Begin processing the 138th record. run: 74254738 subRun: 1 event: 97138 at 06-Sep-2025 15:36:48 CEST
Begin processing the 139th record. run: 74254738 subRun: 1 event: 97139 at 06-Sep-2025 15:36:50 CEST
Begin processing the 140th record. run: 74254738 subRun: 1 event: 97140 at 06-Sep-2025 15:36:53 CEST
Begin processing the 141st record. run: 74254738 subRun: 1 event: 97141 at 06-Sep-2025 15:36:54 CEST
Begin processing the 142nd record. run: 74254738 subRun: 1 event: 97142 at 06-Sep-2025 15:36:56 CEST
Begin processing the 143rd record. run: 74254738 subRun: 1 event: 97143 at 06-Sep-2025 15:36:57 CEST
Begin processing the 144th record. run: 74254738 subRun: 1 event: 97144 at 06-Sep-2025 15:36:59 CEST
Begin processing the 145th record. run: 74254738 subRun: 1 event: 97145 at 06-Sep-2025 15:37:01 CEST
Begin processing the 146th record. run: 74254738 subRun: 1 event: 97146 at 06-Sep-2025 15:37:02 CEST
Begin processing the 147th record. run: 74254738 subRun: 1 event: 97147 at 06-Sep-2025 15:37:04 CEST
Begin processing the 148th record. run: 74254738 subRun: 1 event: 97148 at 06-Sep-2025 15:37:06 CEST
Begin processing the 149th record. run: 74254738 subRun: 1 event: 97149 at 06-Sep-2025 15:37:07 CEST
Begin processing the 150th record. run: 74254738 subRun: 1 event: 97150 at 06-Sep-2025 15:37:09 CEST
Begin processing the 151st record. run: 74254738 subRun: 1 event: 97151 at 06-Sep-2025 15:37:10 CEST
Begin processing the 152nd record. run: 74254738 subRun: 1 event: 97152 at 06-Sep-2025 15:37:12 CEST
Begin processing the 153rd record. run: 74254738 subRun: 1 event: 97153 at 06-Sep-2025 15:37:14 CEST
Begin processing the 154th record. run: 74254738 subRun: 1 event: 97154 at 06-Sep-2025 15:37:15 CEST
Begin processing the 155th record. run: 74254738 subRun: 1 event: 97155 at 06-Sep-2025 15:37:17 CEST
Begin processing the 156th record. run: 74254738 subRun: 1 event: 97156 at 06-Sep-2025 15:37:22 CEST
Begin processing the 157th record. run: 74254738 subRun: 1 event: 97157 at 06-Sep-2025 15:37:23 CEST
Begin processing the 158th record. run: 74254738 subRun: 1 event: 97158 at 06-Sep-2025 15:37:25 CEST
Begin processing the 159th record. run: 74254738 subRun: 1 event: 97159 at 06-Sep-2025 15:37:27 CEST
Begin processing the 160th record. run: 74254738 subRun: 1 event: 97160 at 06-Sep-2025 15:37:29 CEST
Begin processing the 161st record. run: 74254738 subRun: 1 event: 97161 at 06-Sep-2025 15:37:31 CEST
Begin processing the 162nd record. run: 74254738 subRun: 1 event: 97162 at 06-Sep-2025 15:37:33 CEST
Begin processing the 163rd record. run: 74254738 subRun: 1 event: 97163 at 06-Sep-2025 15:37:34 CEST
Begin processing the 164th record. run: 74254738 subRun: 1 event: 97164 at 06-Sep-2025 15:37:35 CEST
Begin processing the 165th record. run: 74254738 subRun: 1 event: 97165 at 06-Sep-2025 15:37:37 CEST
Begin processing the 166th record. run: 74254738 subRun: 1 event: 97166 at 06-Sep-2025 15:37:39 CEST
Begin processing the 167th record. run: 74254738 subRun: 1 event: 97167 at 06-Sep-2025 15:37:41 CEST
Begin processing the 168th record. run: 74254738 subRun: 1 event: 97168 at 06-Sep-2025 15:37:43 CEST
Begin processing the 169th record. run: 74254738 subRun: 1 event: 97169 at 06-Sep-2025 15:37:52 CEST
Begin processing the 170th record. run: 74254738 subRun: 1 event: 97170 at 06-Sep-2025 15:37:53 CEST
Begin processing the 171st record. run: 74254738 subRun: 1 event: 97171 at 06-Sep-2025 15:37:54 CEST
Begin processing the 172nd record. run: 74254738 subRun: 1 event: 97172 at 06-Sep-2025 15:37:55 CEST
Begin processing the 173rd record. run: 74254738 subRun: 1 event: 97173 at 06-Sep-2025 15:37:56 CEST
Begin processing the 174th record. run: 74254738 subRun: 1 event: 97174 at 06-Sep-2025 15:37:57 CEST
Begin processing the 175th record. run: 74254738 subRun: 1 event: 97175 at 06-Sep-2025 15:37:58 CEST
Begin processing the 176th record. run: 74254738 subRun: 1 event: 97176 at 06-Sep-2025 15:38:00 CEST
Begin processing the 177th record. run: 74254738 subRun: 1 event: 97177 at 06-Sep-2025 15:38:02 CEST
Begin processing the 178th record. run: 74254738 subRun: 1 event: 97178 at 06-Sep-2025 15:38:04 CEST
Begin processing the 179th record. run: 74254738 subRun: 1 event: 97179 at 06-Sep-2025 15:38:06 CEST
Begin processing the 180th record. run: 74254738 subRun: 1 event: 97180 at 06-Sep-2025 15:38:07 CEST
Begin processing the 181st record. run: 74254738 subRun: 1 event: 97181 at 06-Sep-2025 15:38:08 CEST
Begin processing the 182nd record. run: 74254738 subRun: 1 event: 97182 at 06-Sep-2025 15:38:11 CEST
Begin processing the 183rd record. run: 74254738 subRun: 1 event: 97183 at 06-Sep-2025 15:38:12 CEST
Begin processing the 184th record. run: 74254738 subRun: 1 event: 97184 at 06-Sep-2025 15:38:14 CEST
Begin processing the 185th record. run: 74254738 subRun: 1 event: 97185 at 06-Sep-2025 15:38:16 CEST
Begin processing the 186th record. run: 74254738 subRun: 1 event: 97186 at 06-Sep-2025 15:38:17 CEST
Begin processing the 187th record. run: 74254738 subRun: 1 event: 97187 at 06-Sep-2025 15:38:19 CEST
Begin processing the 188th record. run: 74254738 subRun: 1 event: 97188 at 06-Sep-2025 15:38:20 CEST
Begin processing the 189th record. run: 74254738 subRun: 1 event: 97189 at 06-Sep-2025 15:38:21 CEST
Begin processing the 190th record. run: 74254738 subRun: 1 event: 97190 at 06-Sep-2025 15:38:23 CEST
Begin processing the 191st record. run: 74254738 subRun: 1 event: 97191 at 06-Sep-2025 15:38:25 CEST
Begin processing the 192nd record. run: 74254738 subRun: 1 event: 97192 at 06-Sep-2025 15:38:27 CEST
Begin processing the 193rd record. run: 74254738 subRun: 1 event: 97193 at 06-Sep-2025 15:38:29 CEST
Begin processing the 194th record. run: 74254738 subRun: 1 event: 97194 at 06-Sep-2025 15:38:30 CEST
Begin processing the 195th record. run: 74254738 subRun: 1 event: 97195 at 06-Sep-2025 15:38:32 CEST
Begin processing the 196th record. run: 74254738 subRun: 1 event: 97196 at 06-Sep-2025 15:38:33 CEST
Begin processing the 197th record. run: 74254738 subRun: 1 event: 97197 at 06-Sep-2025 15:38:34 CEST
Begin processing the 198th record. run: 74254738 subRun: 1 event: 97198 at 06-Sep-2025 15:38:36 CEST
Begin processing the 199th record. run: 74254738 subRun: 1 event: 97199 at 06-Sep-2025 15:38:37 CEST
Begin processing the 200th record. run: 74254738 subRun: 1 event: 97200 at 06-Sep-2025 15:38:39 CEST
06-Sep-2025 15:38:43 CEST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5f/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_485_20231122T052748Z_gen_g4_detsim_hitreco__20240507T172930Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.371412       1.32391       5.65623       1.2296       0.516982        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0196048     0.0298305     0.0412604     0.0215307    0.00971677       200    
prod:pandorapid:Chi2ParticleID                 0.115266      0.169919      0.252868      0.175001      0.021352        200    
prod:energyreconue:EnergyReco                  0.204582       0.44472      0.768681      0.415183      0.0888068       200    
prod:energyreconumu:EnergyReco                0.00159658     0.088474      0.790823      0.0678264     0.0987555       200    
prod:energyreconc:EnergyReco                  0.00287709     0.0367849     0.289849      0.033029      0.0381129       200    
[art]:TriggerResults:TriggerResultInserter     1.098e-05    1.75515e-05   0.000128553   1.5925e-05    1.15612e-05      200    
end_path:ana:PIDASelectionAna                  0.0227135     0.553562       4.21716      0.498223      0.375579        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11103.8 MB
  Peak resident set size usage (VmHWM): 1319.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 76
-rw-r--r-- 1 dune001 lbno 28987 Sep  6 15:38 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_485_20231122T052748Z_gen_g4_detsim_hitreco__20240507T172930Z_reco2_pida_2025-09-06T_133236Z.root
-rw-r--r-- 1 dune001 lbno 24511 Sep  6 15:38 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_485_20231122T052748Z_gen_g4_detsim_hitreco__20240507T172930Z_reco2_reco_2025-09-06T_133236Z.log
-rw-r--r-- 1 dune001 lbno 12187 Sep  6 15:38 jobscript.log
-rw-r--r-- 1 dune001 lbno   217 Sep  6 15:38 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno   138 Sep  6 15:32 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno     0 Sep  6 15:32 debugprod.log