Jobsub ID 29303.104@dunegpschedd01.fnal.gov

Jobsub ID	29303.104@dunegpschedd01.fnal.gov
Workflow ID	1657
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-20 00:00:33
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2025-08-20 00:06:24
From worker node	Hostname	wg62.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-20 00:02:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_132_20231118T173336Z_gen_g4_detsim_hitreco__20240503T051652Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (196s)
	CPU time	2m (121s = 61%)
	Max RSS bytes	1259802624 (1201 MiB)
Outputting started	2025-08-20 00:05:59
Output files
Finished	2025-08-20 00:06:24
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

131st record. run: 50231162 subRun: 1 event: 26531 at 20-Aug-2025 01:05:01 BST
Begin processing the 132nd record. run: 50231162 subRun: 1 event: 26532 at 20-Aug-2025 01:05:02 BST
Begin processing the 133rd record. run: 50231162 subRun: 1 event: 26533 at 20-Aug-2025 01:05:02 BST
Begin processing the 134th record. run: 50231162 subRun: 1 event: 26534 at 20-Aug-2025 01:05:03 BST
Begin processing the 135th record. run: 50231162 subRun: 1 event: 26535 at 20-Aug-2025 01:05:03 BST
Begin processing the 136th record. run: 50231162 subRun: 1 event: 26536 at 20-Aug-2025 01:05:08 BST
Begin processing the 137th record. run: 50231162 subRun: 1 event: 26537 at 20-Aug-2025 01:05:08 BST
Begin processing the 138th record. run: 50231162 subRun: 1 event: 26538 at 20-Aug-2025 01:05:08 BST
Begin processing the 139th record. run: 50231162 subRun: 1 event: 26539 at 20-Aug-2025 01:05:08 BST
Begin processing the 140th record. run: 50231162 subRun: 1 event: 26540 at 20-Aug-2025 01:05:09 BST
Begin processing the 141st record. run: 50231162 subRun: 1 event: 26541 at 20-Aug-2025 01:05:09 BST
Begin processing the 142nd record. run: 50231162 subRun: 1 event: 26542 at 20-Aug-2025 01:05:11 BST
Begin processing the 143rd record. run: 50231162 subRun: 1 event: 26543 at 20-Aug-2025 01:05:11 BST
Begin processing the 144th record. run: 50231162 subRun: 1 event: 26544 at 20-Aug-2025 01:05:11 BST
Begin processing the 145th record. run: 50231162 subRun: 1 event: 26545 at 20-Aug-2025 01:05:13 BST
Begin processing the 146th record. run: 50231162 subRun: 1 event: 26546 at 20-Aug-2025 01:05:14 BST
Begin processing the 147th record. run: 50231162 subRun: 1 event: 26547 at 20-Aug-2025 01:05:14 BST
Begin processing the 148th record. run: 50231162 subRun: 1 event: 26548 at 20-Aug-2025 01:05:15 BST
Begin processing the 149th record. run: 50231162 subRun: 1 event: 26549 at 20-Aug-2025 01:05:18 BST
Begin processing the 150th record. run: 50231162 subRun: 1 event: 26550 at 20-Aug-2025 01:05:18 BST
Begin processing the 151st record. run: 50231162 subRun: 1 event: 26551 at 20-Aug-2025 01:05:18 BST
Begin processing the 152nd record. run: 50231162 subRun: 1 event: 26552 at 20-Aug-2025 01:05:19 BST
Begin processing the 153rd record. run: 50231162 subRun: 1 event: 26553 at 20-Aug-2025 01:05:20 BST
Begin processing the 154th record. run: 50231162 subRun: 1 event: 26554 at 20-Aug-2025 01:05:20 BST
Begin processing the 155th record. run: 50231162 subRun: 1 event: 26555 at 20-Aug-2025 01:05:21 BST
Begin processing the 156th record. run: 50231162 subRun: 1 event: 26556 at 20-Aug-2025 01:05:21 BST
Begin processing the 157th record. run: 50231162 subRun: 1 event: 26557 at 20-Aug-2025 01:05:22 BST
Begin processing the 158th record. run: 50231162 subRun: 1 event: 26558 at 20-Aug-2025 01:05:23 BST
Begin processing the 159th record. run: 50231162 subRun: 1 event: 26559 at 20-Aug-2025 01:05:23 BST
Begin processing the 160th record. run: 50231162 subRun: 1 event: 26560 at 20-Aug-2025 01:05:24 BST
Begin processing the 161st record. run: 50231162 subRun: 1 event: 26561 at 20-Aug-2025 01:05:25 BST
Begin processing the 162nd record. run: 50231162 subRun: 1 event: 26562 at 20-Aug-2025 01:05:25 BST
Begin processing the 163rd record. run: 50231162 subRun: 1 event: 26563 at 20-Aug-2025 01:05:25 BST
Begin processing the 164th record. run: 50231162 subRun: 1 event: 26564 at 20-Aug-2025 01:05:26 BST
Begin processing the 165th record. run: 50231162 subRun: 1 event: 26565 at 20-Aug-2025 01:05:26 BST
Begin processing the 166th record. run: 50231162 subRun: 1 event: 26566 at 20-Aug-2025 01:05:26 BST
Begin processing the 167th record. run: 50231162 subRun: 1 event: 26567 at 20-Aug-2025 01:05:27 BST
Begin processing the 168th record. run: 50231162 subRun: 1 event: 26568 at 20-Aug-2025 01:05:28 BST
Begin processing the 169th record. run: 50231162 subRun: 1 event: 26569 at 20-Aug-2025 01:05:28 BST
Begin processing the 170th record. run: 50231162 subRun: 1 event: 26570 at 20-Aug-2025 01:05:28 BST
Begin processing the 171st record. run: 50231162 subRun: 1 event: 26571 at 20-Aug-2025 01:05:28 BST
Begin processing the 172nd record. run: 50231162 subRun: 1 event: 26572 at 20-Aug-2025 01:05:29 BST
Begin processing the 173rd record. run: 50231162 subRun: 1 event: 26573 at 20-Aug-2025 01:05:29 BST
Begin processing the 174th record. run: 50231162 subRun: 1 event: 26574 at 20-Aug-2025 01:05:30 BST
Begin processing the 175th record. run: 50231162 subRun: 1 event: 26575 at 20-Aug-2025 01:05:31 BST
Begin processing the 176th record. run: 50231162 subRun: 1 event: 26576 at 20-Aug-2025 01:05:31 BST
Begin processing the 177th record. run: 50231162 subRun: 1 event: 26577 at 20-Aug-2025 01:05:31 BST
Begin processing the 178th record. run: 50231162 subRun: 1 event: 26578 at 20-Aug-2025 01:05:31 BST
Begin processing the 179th record. run: 50231162 subRun: 1 event: 26579 at 20-Aug-2025 01:05:32 BST
Begin processing the 180th record. run: 50231162 subRun: 1 event: 26580 at 20-Aug-2025 01:05:32 BST
Begin processing the 181st record. run: 50231162 subRun: 1 event: 26581 at 20-Aug-2025 01:05:33 BST
Begin processing the 182nd record. run: 50231162 subRun: 1 event: 26582 at 20-Aug-2025 01:05:33 BST
Begin processing the 183rd record. run: 50231162 subRun: 1 event: 26583 at 20-Aug-2025 01:05:35 BST
Begin processing the 184th record. run: 50231162 subRun: 1 event: 26584 at 20-Aug-2025 01:05:39 BST
Begin processing the 185th record. run: 50231162 subRun: 1 event: 26585 at 20-Aug-2025 01:05:40 BST
Begin processing the 186th record. run: 50231162 subRun: 1 event: 26586 at 20-Aug-2025 01:05:40 BST
Begin processing the 187th record. run: 50231162 subRun: 1 event: 26587 at 20-Aug-2025 01:05:41 BST
Begin processing the 188th record. run: 50231162 subRun: 1 event: 26588 at 20-Aug-2025 01:05:47 BST
Begin processing the 189th record. run: 50231162 subRun: 1 event: 26589 at 20-Aug-2025 01:05:48 BST
Begin processing the 190th record. run: 50231162 subRun: 1 event: 26590 at 20-Aug-2025 01:05:48 BST
Begin processing the 191st record. run: 50231162 subRun: 1 event: 26591 at 20-Aug-2025 01:05:49 BST
Begin processing the 192nd record. run: 50231162 subRun: 1 event: 26592 at 20-Aug-2025 01:05:52 BST
Begin processing the 193rd record. run: 50231162 subRun: 1 event: 26593 at 20-Aug-2025 01:05:54 BST
Begin processing the 194th record. run: 50231162 subRun: 1 event: 26594 at 20-Aug-2025 01:05:55 BST
Begin processing the 195th record. run: 50231162 subRun: 1 event: 26595 at 20-Aug-2025 01:05:55 BST
Begin processing the 196th record. run: 50231162 subRun: 1 event: 26596 at 20-Aug-2025 01:05:56 BST
Begin processing the 197th record. run: 50231162 subRun: 1 event: 26597 at 20-Aug-2025 01:05:56 BST
Begin processing the 198th record. run: 50231162 subRun: 1 event: 26598 at 20-Aug-2025 01:05:56 BST
Begin processing the 199th record. run: 50231162 subRun: 1 event: 26599 at 20-Aug-2025 01:05:57 BST
Begin processing the 200th record. run: 50231162 subRun: 1 event: 26600 at 20-Aug-2025 01:05:57 BST
20-Aug-2025 01:05:57 BST  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/8b/43/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_132_20231118T173336Z_gen_g4_detsim_hitreco__20240503T051652Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0875683     0.480228       5.5665        0.27388      0.678401        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00257099     0.0076625     0.106685     0.00546263    0.00968271       200    
prod:pandorapid:Chi2ParticleID                 0.0125991     0.0390409     0.444112      0.0236945     0.0440172       200    
prod:energyreconue:EnergyReco                  0.0434768     0.0928722     0.623291      0.0680428     0.0821731       200    
prod:energyreconumu:EnergyReco                0.00316659     0.0929544      4.60015      0.0252785     0.362294        200    
prod:energyreconc:EnergyReco                  0.00653493     0.0487682     0.431743      0.0362913     0.0595522       200    
[art]:TriggerResults:TriggerResultInserter    1.8158e-05    2.92482e-05   0.000118233   2.7512e-05    8.39964e-06      200    
end_path:ana:PIDASelectionAna                 0.00921251     0.198262       3.15084      0.100054      0.331788        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5663.29 MB
  Peak resident set size usage (VmHWM): 1259.8 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 76
-rw-r--r--. 1 job0004 job0004 28277 Aug 20 01:05 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_132_20231118T173336Z_gen_g4_detsim_hitreco__20240503T051652Z_reco2_pida_2025-08-20T_000250Z.root
-rw-r--r--. 1 job0004 job0004 24223 Aug 20 01:05 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_132_20231118T173336Z_gen_g4_detsim_hitreco__20240503T051652Z_reco2_reco_2025-08-20T_000250Z.log
-rw-r--r--. 1 job0004 job0004 12052 Aug 20 01:05 jobscript.log
-rw-r--r--. 1 job0004 job0004   189 Aug 20 01:05 justin-processed-pfns.txt
-rw-r--r--. 1 job0004 job0004   138 Aug 20 01:02 all-input-dids.txt
-rw-r--r--. 1 job0004 job0004     0 Aug 20 01:03 debugprod.log