Jobsub ID 46679.9@dunegpschedd01.fnal.gov

Jobsub ID	46679.9@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 14:46:48
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-15 16:58:18
From worker node	Hostname	wn-lot-032.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 14:47:29
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_750_20231208T031504Z_gen_g4_detsim_hitreco__20240510T041620Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_892_20231202T161015Z_gen_g4_detsim_hitreco__20240508T043821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_255_20231202T200400Z_gen_g4_detsim_hitreco__20240508T065440Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_640_20231202T064421Z_gen_g4_detsim_hitreco__20240507T203234Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_296_20231207T023413Z_gen_g4_detsim_hitreco__20240510T043536Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_366_20231204T044142Z_gen_g4_detsim_hitreco__20240509T204129Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_368_20231208T045201Z_gen_g4_detsim_hitreco__20240510T065417Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_392_20231207T125115Z_gen_g4_detsim_hitreco__20240510T055022Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_376_20231122T052634Z_gen_g4_detsim_hitreco__20240503T070756Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_981_20231202T161024Z_gen_g4_detsim_hitreco__20240508T044411Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 16:58:18
Saved logs	justin-logs:46679.9-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eco:energyrecnumumcsllhd:EnergyReco                  0.00034666     0.0321096      3.20612     0.00276144      0.15776       1000    
reco:opdec:Deconvolution                               0.0458613     0.235066      0.490807      0.230959      0.0774698      1000    
reco:ophitspe:OpHitFinderDeco                          0.0779685     0.136875      0.322188      0.134269      0.0244816      1000    
reco:opflash:OpFlashFinder                            0.000193332   0.00119349    0.00303733    0.00115515    0.000482061     1000    
reco:opslicer:OpSlicer                                4.6968e-05     0.232854       1.08963      0.176943      0.187918       1000    
reco:rns:RandomNumberSaver                            1.9877e-05    3.82321e-05   0.00833523    3.0287e-05    0.000262694     1000    
reco:anglereconue:NuAngularReco                       0.000156664   0.00211931     0.191605     0.000371721    0.0104323      1000    
reco:anglereconumu:NuAngularReco                      7.1434e-05    0.00184332     0.147144     0.000197755   0.00907062      1000    
reco:anglereconuepfps:NuAngularReco                   0.000121107   0.00194864     0.154613     0.000304846   0.00914767      1000    
reco:anglereconumupfps:NuAngularReco                  0.000102613    0.0019343     0.153634     0.000259676   0.00926104      1000    
reco:anglerecohits:NuAngularReco                      0.000116478    0.002493      0.161203     0.000634123   0.00995743      1000    
[art]:TriggerResults:TriggerResultInserter            1.0019e-05    1.25068e-05   4.5285e-05    1.22575e-05   2.06242e-06     1000    
end_path:cafmaker:CAFMaker                            0.00140497     0.0257572       1.905      0.00418496     0.118753       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3898.02 MB
  Peak resident set size usage (VmHWM): 2238.01 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 18:57:58 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/20/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_981_20231202T161024Z_gen_g4_detsim_hitreco__20240508T044411Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000265673, 0.998406, 0.00132871, 
Output 1: 0.559289, 0.430267, 0.00985004, 0.000594408, 
Output 2: 0.998742, 0.00122907, 2.76318e-05, 1.09778e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1777!
Begin processing the 994th record. run: 74254738 subRun: 1 event: 75394 at 15-Sep-2025 18:57:35 CEST
Boundary wire vector sizes: 171, 278, 157
minwire 0: 1787
minwire 1: 541
minwire 2: 2061
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.989385, 0.00265933, 0.00795527, 
Output 1: 0.0842335, 0.894509, 0.0204374, 0.0008196, 
Output 2: 0.703729, 0.289561, 0.00661979, 9.00999e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1729!
Begin processing the 995th record. run: 74254738 subRun: 1 event: 75395 at 15-Sep-2025 18:57:38 CEST
Boundary wire vector sizes: 157, 169, 117
minwire 0: 763
minwire 1: 1265
minwire 2: 609
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00640333, 0.149402, 0.844195, 
Output 1: 0.903364, 0.09398, 0.00255458, 0.000101264, 
Output 2: 0.93341, 0.065816, 0.000751289, 2.22088e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1906!
Begin processing the 996th record. run: 74254738 subRun: 1 event: 75396 at 15-Sep-2025 18:57:41 CEST
Boundary wire vector sizes: 681, 640, 565
minwire 0: 645
minwire 1: 2615
minwire 2: 101
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000682911, 0.994418, 0.0048993, 
Output 1: 0.997085, 0.0027041, 0.000116872, 9.38459e-05, 
Output 2: 0.995441, 0.00442241, 0.000102972, 3.35959e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2050!
Begin processing the 997th record. run: 74254738 subRun: 1 event: 75397 at 15-Sep-2025 18:57:45 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 181!
Begin processing the 998th record. run: 74254738 subRun: 1 event: 75398 at 15-Sep-2025 18:57:47 CEST
Boundary wire vector sizes: 838, 537, 677
minwire 0: 430
minwire 1: 1740
minwire 2: 259
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000685545, 0.997061, 0.00225299, 
Output 1: 0.0249508, 0.480035, 0.470295, 0.0247198, 
Output 2: 0.975376, 0.0233936, 0.00119727, 3.2732e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2074!
Begin processing the 999th record. run: 74254738 subRun: 1 event: 75399 at 15-Sep-2025 18:57:51 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 542!
Begin processing the 1000th record. run: 74254738 subRun: 1 event: 75400 at 15-Sep-2025 18:57:54 CEST
Boundary wire vector sizes: 293, 517, 360
minwire 0: 545
minwire 1: 2638
minwire 2: 56
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0161535, 0.969378, 0.0144681, 
Output 1: 0.0050013, 0.102466, 0.859622, 0.0329105, 
Output 2: 0.94277, 0.0543257, 0.00285193, 5.28142e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1526!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/62/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_750_20231208T031504Z_gen_g4_detsim_hitreco__20240510T041620Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/94/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_892_20231202T161015Z_gen_g4_detsim_hitreco__20240508T043821Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_255_20231202T200400Z_gen_g4_detsim_hitreco__20240508T065440Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/38/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_640_20231202T064421Z_gen_g4_detsim_hitreco__20240507T203234Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/13/08/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_296_20231207T023413Z_gen_g4_detsim_hitreco__20240510T043536Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c0/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_366_20231204T044142Z_gen_g4_detsim_hitreco__20240509T204129Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_368_20231208T045201Z_gen_g4_detsim_hitreco__20240510T065417Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/e6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_392_20231207T125115Z_gen_g4_detsim_hitreco__20240510T055022Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/4d/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_376_20231122T052634Z_gen_g4_detsim_hitreco__20240503T070756Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/20/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_981_20231202T161024Z_gen_g4_detsim_hitreco__20240508T044411Z_reco2.root
.:
total 51896
-rw-r--r--. 1 dune012 dune 19482080 Sep 15 18:57 flatcaf.root
-rw-r--r--. 1 dune012 dune 16389656 Sep 15 18:57 caf.root
-rw-r--r--. 1 dune012 dune 16389656 Sep 15 18:57 caf_fd_hd_atmo_2501_20250915T144738Z.root
-rw-r--r--. 1 dune012 dune   600545 Sep 15 18:57 caf_20250915T144738Z.log
-rw-r--r--. 1 dune012 dune   224173 Sep 15 18:57 jobscript.log
-rw-r--r--. 1 dune012 dune     1901 Sep 15 18:57 caf_20250915T144738Z.file
-rw-r--r--. 1 dune012 dune     1901 Sep 15 18:57 caf_20250915T144738Z.pfns
-rw-r--r--. 1 dune012 dune     1901 Sep 15 16:47 file.list
-rw-r--r--. 1 dune012 dune     1901 Sep 15 18:57 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     1377 Sep 15 16:47 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     1377 Sep 15 18:57 caf_20250915T144738Z.did
-rw-r--r--. 1 dune012 dune     1377 Sep 15 16:47 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 15 16:47 debugprod.log