Jobsub ID 40189.5@dunegpschedd02.fnal.gov

Jobsub ID	40189.5@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 17:13:54
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2025-09-15 18:47:20
From worker node	Hostname	wn-lot-033.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 17:14:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_488_20231207T025259Z_gen_g4_detsim_hitreco__20240510T050752Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_401_20231122T124933Z_gen_g4_detsim_hitreco__20240507T181914Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_188_20231207T152427Z_gen_g4_detsim_hitreco__20240510T043250Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_195_20231203T143909Z_gen_g4_detsim_hitreco__20240508T052700Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_927_20231201T210957Z_gen_g4_detsim_hitreco__20240508T023226Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_766_20231207T075258Z_gen_g4_detsim_hitreco__20240510T024406Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_96_20231201T113838Z_gen_g4_detsim_hitreco__20240507T185709Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_395_20231204T065540Z_gen_g4_detsim_hitreco__20240509T214943Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_398_20231201T115920Z_gen_g4_detsim_hitreco__20240507T194443Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_136_20231203T125208Z_gen_g4_detsim_hitreco__20240508T073923Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5533s)
	CPU time	1h (4561s = 82%)
	Max RSS bytes	2127757312 (2029 MiB)
Outputting started	2025-09-15 18:47:00
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T171454Z.root
Finished	2025-09-15 18:47:20
Saved logs	justin-logs:40189.5-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

5473      1.09947      0.0940486     0.0918602      1100    
reco:energyrecnumurange:EnergyReco                    0.000194785    0.0391572      2.93001     0.00240909     0.190876       1100    
reco:energyrecnumumcs:EnergyReco                      0.000460423    0.0398518      2.84063     0.00278162      0.18843       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00032003     0.0397974      2.88783     0.00276543     0.190655       1100    
reco:opdec:Deconvolution                              0.00398403     0.231893      0.652426      0.224875      0.0936013      1100    
reco:ophitspe:OpHitFinderDeco                          0.0024059     0.125131      0.546116      0.133881      0.0537894      1100    
reco:opflash:OpFlashFinder                            0.000118673   0.00114691     0.0177495    0.00106655    0.000735478     1100    
reco:opslicer:OpSlicer                                3.5978e-05     0.230526       1.04925      0.179634      0.191987       1100    
reco:rns:RandomNumberSaver                            1.5298e-05    2.45968e-05   0.000360906   2.4166e-05    1.35269e-05     1100    
reco:anglereconue:NuAngularReco                       0.000109706   0.00178238     0.133336     0.000316263   0.00699042      1100    
reco:anglereconumu:NuAngularReco                      7.1063e-05    0.00167816      0.1328      0.000191148   0.00705331      1100    
reco:anglereconuepfps:NuAngularReco                   0.000109385   0.00176425     0.131855     0.000274429   0.00697992      1100    
reco:anglereconumupfps:NuAngularReco                  0.000104225   0.00176619     0.140207     0.000247945   0.00715568      1100    
reco:anglerecohits:NuAngularReco                      8.1703e-05    0.00226216     0.136507     0.000588709   0.00750507      1100    
[art]:TriggerResults:TriggerResultInserter             8.827e-06    1.07485e-05   4.9233e-05    1.0389e-05    1.8219e-06      1100    
end_path:cafmaker:CAFMaker                            0.00113118     0.0220983      1.41354     0.00357391     0.0869588      1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3790.05 MB
  Peak resident set size usage (VmHWM): 2127.76 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Found hits: 3197!
Begin processing the 1092nd record. run: 74511302 subRun: 1 event: 13692 at 15-Sep-2025 20:46:34 CEST
Boundary wire vector sizes: 745, 698, 687
minwire 0: 584
minwire 1: 2344
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996274, 0.000755272, 0.00297113, 
Output 1: 0.331788, 0.526165, 0.0988735, 0.043173, 
Output 2: 0.783933, 0.185535, 0.026864, 0.00366745, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1830!
Begin processing the 1093rd record. run: 74511302 subRun: 1 event: 13693 at 15-Sep-2025 20:46:38 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 505!
Begin processing the 1094th record. run: 74511302 subRun: 1 event: 13694 at 15-Sep-2025 20:46:39 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 719!
Begin processing the 1095th record. run: 74511302 subRun: 1 event: 13695 at 15-Sep-2025 20:46:42 CEST
Boundary wire vector sizes: 1289, 1297, 735
minwire 0: 633
minwire 1: 1395
minwire 2: 256
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000720476, 0.996284, 0.00299564, 
Output 1: 0.0450846, 0.366541, 0.458492, 0.129882, 
Output 2: 0.0864659, 0.482353, 0.384626, 0.0465551, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2941!
Begin processing the 1096th record. run: 74511302 subRun: 1 event: 13696 at 15-Sep-2025 20:46:46 CEST
Boundary wire vector sizes: 59, 66, 44
minwire 0: 969
minwire 1: 2733
minwire 2: 331
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00457817, 0.945919, 0.0495032, 
Output 1: 0.995656, 0.00388179, 0.00024262, 0.000220026, 
Output 2: 0.991492, 0.00826046, 0.000187439, 6.01338e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1006!
Begin processing the 1097th record. run: 74511302 subRun: 1 event: 13697 at 15-Sep-2025 20:46:48 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1030!
Begin processing the 1098th record. run: 74511302 subRun: 1 event: 13698 at 15-Sep-2025 20:46:50 CEST
Boundary wire vector sizes: 65, 55, 58
minwire 0: 853
minwire 1: 1821
minwire 2: 354
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00670362, 0.00178216, 0.991514, 
Output 1: 0.0210645, 0.055977, 0.903637, 0.0193216, 
Output 2: 0.979428, 0.019242, 0.00130486, 2.4768e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1649!
Begin processing the 1099th record. run: 74511302 subRun: 1 event: 13699 at 15-Sep-2025 20:46:53 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 567!
Begin processing the 1100th record. run: 74511302 subRun: 1 event: 13700 at 15-Sep-2025 20:46:55 CEST
Boundary wire vector sizes: 378, 444, 375
minwire 0: 2063
minwire 1: 211
minwire 2: 2319
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000362798, 0.996848, 0.00278881, 
Output 1: 0.902692, 0.0964038, 0.000827995, 7.62002e-05, 
Output 2: 0.983531, 0.0161486, 0.000304069, 1.61238e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1803!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/59/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_488_20231207T025259Z_gen_g4_detsim_hitreco__20240510T050752Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/c0/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_401_20231122T124933Z_gen_g4_detsim_hitreco__20240507T181914Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/57/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_188_20231207T152427Z_gen_g4_detsim_hitreco__20240510T043250Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/8d/d1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_195_20231203T143909Z_gen_g4_detsim_hitreco__20240508T052700Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/52/0d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_927_20231201T210957Z_gen_g4_detsim_hitreco__20240508T023226Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/58/a6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_766_20231207T075258Z_gen_g4_detsim_hitreco__20240510T024406Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/cc/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_96_20231201T113838Z_gen_g4_detsim_hitreco__20240507T185709Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/89/93/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_395_20231204T065540Z_gen_g4_detsim_hitreco__20240509T214943Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3a/ae/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_398_20231201T115920Z_gen_g4_detsim_hitreco__20240507T194443Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_136_20231203T125208Z_gen_g4_detsim_hitreco__20240508T073923Z_reco2.root
.:
total 53264
-rw-r--r--. 1 dune012 dune 20015861 Sep 15 20:46 flatcaf.root
-rw-r--r--. 1 dune012 dune 16781297 Sep 15 20:46 caf.root
-rw-r--r--. 1 dune012 dune 16781297 Sep 15 20:46 caf_fd_hd_atmo_2501_20250915T171454Z.root
-rw-r--r--. 1 dune012 dune   655680 Sep 15 20:46 caf_20250915T171454Z.log
-rw-r--r--. 1 dune012 dune   257877 Sep 15 20:46 jobscript.log
-rw-r--r--. 1 dune012 dune     1948 Sep 15 20:46 caf_20250915T171454Z.file
-rw-r--r--. 1 dune012 dune     1948 Sep 15 20:46 caf_20250915T171454Z.pfns
-rw-r--r--. 1 dune012 dune     1948 Sep 15 19:14 file.list
-rw-r--r--. 1 dune012 dune     1948 Sep 15 20:46 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     1376 Sep 15 19:14 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     1376 Sep 15 20:46 caf_20250915T171454Z.did
-rw-r--r--. 1 dune012 dune     1376 Sep 15 19:14 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 15 19:15 debugprod.log