Jobsub ID 29295.166@dunegpschedd01.fnal.gov

Jobsub ID	29295.166@dunegpschedd01.fnal.gov
Workflow ID	1657
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-19 23:46:32
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2025-08-19 23:51:42
From worker node	Hostname	wi09.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-19 23:47:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_381_20231121T154433Z_gen_g4_detsim_hitreco__20240507T173720Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (227s)
	CPU time	1m (116s = 51%)
	Max RSS bytes	1530208256 (1459 MiB)
Outputting started	2025-08-19 23:51:20
Output files
Finished	2025-08-19 23:51:42
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

131st record. run: 50258005 subRun: 1 event: 76331 at 20-Aug-2025 00:50:37 BST
Begin processing the 132nd record. run: 50258005 subRun: 1 event: 76332 at 20-Aug-2025 00:50:37 BST
Begin processing the 133rd record. run: 50258005 subRun: 1 event: 76333 at 20-Aug-2025 00:50:39 BST
Begin processing the 134th record. run: 50258005 subRun: 1 event: 76334 at 20-Aug-2025 00:50:40 BST
Begin processing the 135th record. run: 50258005 subRun: 1 event: 76335 at 20-Aug-2025 00:50:41 BST
Begin processing the 136th record. run: 50258005 subRun: 1 event: 76336 at 20-Aug-2025 00:50:41 BST
Begin processing the 137th record. run: 50258005 subRun: 1 event: 76337 at 20-Aug-2025 00:50:42 BST
Begin processing the 138th record. run: 50258005 subRun: 1 event: 76338 at 20-Aug-2025 00:50:44 BST
Begin processing the 139th record. run: 50258005 subRun: 1 event: 76339 at 20-Aug-2025 00:50:45 BST
Begin processing the 140th record. run: 50258005 subRun: 1 event: 76340 at 20-Aug-2025 00:50:45 BST
Begin processing the 141st record. run: 50258005 subRun: 1 event: 76341 at 20-Aug-2025 00:50:45 BST
Begin processing the 142nd record. run: 50258005 subRun: 1 event: 76342 at 20-Aug-2025 00:50:46 BST
Begin processing the 143rd record. run: 50258005 subRun: 1 event: 76343 at 20-Aug-2025 00:50:47 BST
Begin processing the 144th record. run: 50258005 subRun: 1 event: 76344 at 20-Aug-2025 00:50:48 BST
Begin processing the 145th record. run: 50258005 subRun: 1 event: 76345 at 20-Aug-2025 00:50:48 BST
Begin processing the 146th record. run: 50258005 subRun: 1 event: 76346 at 20-Aug-2025 00:50:48 BST
Begin processing the 147th record. run: 50258005 subRun: 1 event: 76347 at 20-Aug-2025 00:50:49 BST
Begin processing the 148th record. run: 50258005 subRun: 1 event: 76348 at 20-Aug-2025 00:50:50 BST
Begin processing the 149th record. run: 50258005 subRun: 1 event: 76349 at 20-Aug-2025 00:50:50 BST
Begin processing the 150th record. run: 50258005 subRun: 1 event: 76350 at 20-Aug-2025 00:50:51 BST
Begin processing the 151st record. run: 50258005 subRun: 1 event: 76351 at 20-Aug-2025 00:50:51 BST
Begin processing the 152nd record. run: 50258005 subRun: 1 event: 76352 at 20-Aug-2025 00:50:51 BST
Begin processing the 153rd record. run: 50258005 subRun: 1 event: 76353 at 20-Aug-2025 00:50:52 BST
Begin processing the 154th record. run: 50258005 subRun: 1 event: 76354 at 20-Aug-2025 00:50:52 BST
Begin processing the 155th record. run: 50258005 subRun: 1 event: 76355 at 20-Aug-2025 00:50:54 BST
Begin processing the 156th record. run: 50258005 subRun: 1 event: 76356 at 20-Aug-2025 00:50:55 BST
Begin processing the 157th record. run: 50258005 subRun: 1 event: 76357 at 20-Aug-2025 00:50:56 BST
Begin processing the 158th record. run: 50258005 subRun: 1 event: 76358 at 20-Aug-2025 00:50:56 BST
Begin processing the 159th record. run: 50258005 subRun: 1 event: 76359 at 20-Aug-2025 00:50:57 BST
Begin processing the 160th record. run: 50258005 subRun: 1 event: 76360 at 20-Aug-2025 00:50:57 BST
Begin processing the 161st record. run: 50258005 subRun: 1 event: 76361 at 20-Aug-2025 00:50:57 BST
Begin processing the 162nd record. run: 50258005 subRun: 1 event: 76362 at 20-Aug-2025 00:50:58 BST
Begin processing the 163rd record. run: 50258005 subRun: 1 event: 76363 at 20-Aug-2025 00:50:58 BST
Begin processing the 164th record. run: 50258005 subRun: 1 event: 76364 at 20-Aug-2025 00:50:58 BST
Begin processing the 165th record. run: 50258005 subRun: 1 event: 76365 at 20-Aug-2025 00:50:59 BST
Begin processing the 166th record. run: 50258005 subRun: 1 event: 76366 at 20-Aug-2025 00:50:59 BST
Begin processing the 167th record. run: 50258005 subRun: 1 event: 76367 at 20-Aug-2025 00:50:59 BST
Begin processing the 168th record. run: 50258005 subRun: 1 event: 76368 at 20-Aug-2025 00:50:59 BST
Begin processing the 169th record. run: 50258005 subRun: 1 event: 76369 at 20-Aug-2025 00:51:00 BST
Begin processing the 170th record. run: 50258005 subRun: 1 event: 76370 at 20-Aug-2025 00:51:00 BST
Begin processing the 171st record. run: 50258005 subRun: 1 event: 76371 at 20-Aug-2025 00:51:00 BST
Begin processing the 172nd record. run: 50258005 subRun: 1 event: 76372 at 20-Aug-2025 00:51:00 BST
Begin processing the 173rd record. run: 50258005 subRun: 1 event: 76373 at 20-Aug-2025 00:51:01 BST
Begin processing the 174th record. run: 50258005 subRun: 1 event: 76374 at 20-Aug-2025 00:51:01 BST
Begin processing the 175th record. run: 50258005 subRun: 1 event: 76375 at 20-Aug-2025 00:51:02 BST
Begin processing the 176th record. run: 50258005 subRun: 1 event: 76376 at 20-Aug-2025 00:51:02 BST
Begin processing the 177th record. run: 50258005 subRun: 1 event: 76377 at 20-Aug-2025 00:51:02 BST
Begin processing the 178th record. run: 50258005 subRun: 1 event: 76378 at 20-Aug-2025 00:51:03 BST
Begin processing the 179th record. run: 50258005 subRun: 1 event: 76379 at 20-Aug-2025 00:51:03 BST
Begin processing the 180th record. run: 50258005 subRun: 1 event: 76380 at 20-Aug-2025 00:51:04 BST
Begin processing the 181st record. run: 50258005 subRun: 1 event: 76381 at 20-Aug-2025 00:51:04 BST
Begin processing the 182nd record. run: 50258005 subRun: 1 event: 76382 at 20-Aug-2025 00:51:04 BST
Begin processing the 183rd record. run: 50258005 subRun: 1 event: 76383 at 20-Aug-2025 00:51:06 BST
Begin processing the 184th record. run: 50258005 subRun: 1 event: 76384 at 20-Aug-2025 00:51:08 BST
Begin processing the 185th record. run: 50258005 subRun: 1 event: 76385 at 20-Aug-2025 00:51:08 BST
Begin processing the 186th record. run: 50258005 subRun: 1 event: 76386 at 20-Aug-2025 00:51:09 BST
Begin processing the 187th record. run: 50258005 subRun: 1 event: 76387 at 20-Aug-2025 00:51:12 BST
Begin processing the 188th record. run: 50258005 subRun: 1 event: 76388 at 20-Aug-2025 00:51:13 BST
Begin processing the 189th record. run: 50258005 subRun: 1 event: 76389 at 20-Aug-2025 00:51:13 BST
Begin processing the 190th record. run: 50258005 subRun: 1 event: 76390 at 20-Aug-2025 00:51:14 BST
Begin processing the 191st record. run: 50258005 subRun: 1 event: 76391 at 20-Aug-2025 00:51:14 BST
Begin processing the 192nd record. run: 50258005 subRun: 1 event: 76392 at 20-Aug-2025 00:51:14 BST
Begin processing the 193rd record. run: 50258005 subRun: 1 event: 76393 at 20-Aug-2025 00:51:15 BST
Begin processing the 194th record. run: 50258005 subRun: 1 event: 76394 at 20-Aug-2025 00:51:15 BST
Begin processing the 195th record. run: 50258005 subRun: 1 event: 76395 at 20-Aug-2025 00:51:16 BST
Begin processing the 196th record. run: 50258005 subRun: 1 event: 76396 at 20-Aug-2025 00:51:16 BST
Begin processing the 197th record. run: 50258005 subRun: 1 event: 76397 at 20-Aug-2025 00:51:17 BST
Begin processing the 198th record. run: 50258005 subRun: 1 event: 76398 at 20-Aug-2025 00:51:17 BST
Begin processing the 199th record. run: 50258005 subRun: 1 event: 76399 at 20-Aug-2025 00:51:17 BST
Begin processing the 200th record. run: 50258005 subRun: 1 event: 76400 at 20-Aug-2025 00:51:17 BST
20-Aug-2025 00:51:18 BST  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/a5/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_381_20231121T154433Z_gen_g4_detsim_hitreco__20240507T173720Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0952014     0.542714       11.8345      0.292529      0.981238        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00248893    0.00930538     0.0466494    0.00553691     0.0087931       200    
prod:pandorapid:Chi2ParticleID                 0.0118047     0.0459092     0.196082      0.0233891     0.0410496       200    
prod:energyreconue:EnergyReco                  0.0404326     0.093611       1.01892      0.0663611     0.0961044       200    
prod:energyreconumu:EnergyReco                0.00253785     0.110107       6.57657      0.0235028     0.491472        200    
prod:energyreconc:EnergyReco                  0.00393174     0.0515533     0.978406      0.0413043     0.0801955       200    
[art]:TriggerResults:TriggerResultInserter    1.2719e-05    2.12239e-05   8.7179e-05    2.03165e-05   5.82777e-06      200    
end_path:ana:PIDASelectionAna                  0.011025      0.231667       3.09453      0.104213      0.406106        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8108.31 MB
  Peak resident set size usage (VmHWM): 1530.21 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 76
-rw-r--r--. 1 job0006 job0006 28433 Aug 20 00:51 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_381_20231121T154433Z_gen_g4_detsim_hitreco__20240507T173720Z_reco2_pida_2025-08-19T_234737Z.root
-rw-r--r--. 1 job0006 job0006 24224 Aug 20 00:51 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_381_20231121T154433Z_gen_g4_detsim_hitreco__20240507T173720Z_reco2_reco_2025-08-19T_234737Z.log
-rw-r--r--. 1 job0006 job0006 12053 Aug 20 00:51 jobscript.log
-rw-r--r--. 1 job0006 job0006   189 Aug 20 00:51 justin-processed-pfns.txt
-rw-r--r--. 1 job0006 job0006   138 Aug 20 00:47 all-input-dids.txt
-rw-r--r--. 1 job0006 job0006     0 Aug 20 00:47 debugprod.log