Jobsub ID 46694.13@dunegpschedd01.fnal.gov

Jobsub ID	46694.13@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:18:49
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-15 16:12:41
From worker node	Hostname	wn-lot-023.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 15:20:04
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_105_20231204T120339Z_gen_g4_detsim_hitreco__20240509T214623Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_727_20231203T233659Z_gen_g4_detsim_hitreco__20240509T210759Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_2_20231202T150324Z_gen_g4_detsim_hitreco__20240507T223637Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_706_20231202T115046Z_gen_g4_detsim_hitreco__20240507T220450Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_610_20231202T060615Z_gen_g4_detsim_hitreco__20240507T205647Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_214_20231202T075352Z_gen_g4_detsim_hitreco__20240507T205640Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_106_20231204T140301Z_gen_g4_detsim_hitreco__20240509T214537Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_890_20231204T140420Z_gen_g4_detsim_hitreco__20240509T204024Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_268_20231208T041613Z_gen_g4_detsim_hitreco__20240510T034743Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_1105_20231203T025450Z_gen_g4_detsim_hitreco__20240508T065303Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 16:12:41
Saved logs	justin-logs:46694.13-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.335674      0.154066      0.0632996       800    
reco:ophitspe:OpHitFinderDeco                         0.00214082    0.00782227     0.0297263    0.00783679    0.00221669       800    
reco:opflash:OpFlashFinder                            0.000156584   0.00115256     0.0027997    0.00110316    0.000511315      800    
reco:opslicer:OpSlicer                                3.6308e-05     0.223011       0.91088      0.176045       0.18009        800    
reco:rns:RandomNumberSaver                             1.593e-05    2.72925e-05   0.000256371   2.75115e-05   9.01717e-06      800    
reco:anglereconue:NuAngularReco                       0.000103124   0.00150721     0.0523968    0.00037055    0.00410482       800    
reco:anglereconumu:NuAngularReco                      7.0242e-05    0.00134913     0.0518289    0.00019711    0.00415303       800    
reco:anglereconuepfps:NuAngularReco                   0.000112841   0.00143736     0.0521116    0.000300083   0.00412323       800    
reco:anglereconumupfps:NuAngularReco                  9.9466e-05    0.00142019     0.0528923    0.000256456   0.00425651       800    
reco:anglerecohits:NuAngularReco                      8.1664e-05    0.00197281     0.0577837    0.000622206   0.00472006       800    
[art]:TriggerResults:TriggerResultInserter             9.318e-06    1.18548e-05   4.0686e-05    1.16165e-05   1.62171e-06      800    
end_path:cafmaker:CAFMaker                            0.00137158     0.019477      0.824883     0.00423971     0.0563087       800    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3807.08 MB
  Peak resident set size usage (VmHWM): 2144.38 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 18:12:24 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_268_20231208T041613Z_gen_g4_detsim_hitreco__20240510T034743Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.988796, 0.00161419, 0.00958962, 
Output 1: 0.0892806, 0.210622, 0.251219, 0.448878, 
Output 2: 0.0128151, 0.033274, 0.23059, 0.723321, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2945!
Begin processing the 794th record. run: 74517882 subRun: 1 event: 89094 at 15-Sep-2025 18:12:09 CEST
Boundary wire vector sizes: 1759, 1767, 1607
minwire 0: 962
minwire 1: 2071
minwire 2: 546
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00647823, 0.786371, 0.207151, 
Output 1: 0.170102, 0.822028, 0.00742248, 0.000447859, 
Output 2: 0.930282, 0.068315, 0.00136043, 4.28047e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2906!
Begin processing the 795th record. run: 74517882 subRun: 1 event: 89095 at 15-Sep-2025 18:12:13 CEST
Boundary wire vector sizes: 147, 228, 114
minwire 0: 1616
minwire 1: 849
minwire 2: 1988
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000542602, 0.997387, 0.00207083, 
Output 1: 0.00720661, 0.981192, 0.011163, 0.000438552, 
Output 2: 0.994766, 0.00509021, 0.000140341, 4.01458e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1794!
Begin processing the 796th record. run: 74517882 subRun: 1 event: 89096 at 15-Sep-2025 18:12:15 CEST
Boundary wire vector sizes: 63, 114, 96
minwire 0: 2140
minwire 1: 861
minwire 2: 2144
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00114758, 0.986609, 0.0122434, 
Output 1: 0.969786, 0.0299366, 0.000191452, 8.57263e-05, 
Output 2: 0.999215, 0.000755148, 2.2338e-05, 7.87558e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1271!
Begin processing the 797th record. run: 74517882 subRun: 1 event: 89097 at 15-Sep-2025 18:12:17 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 514!
Begin processing the 798th record. run: 74517882 subRun: 1 event: 89098 at 15-Sep-2025 18:12:18 CEST
Boundary wire vector sizes: 240, 275, 221
minwire 0: 1877
minwire 1: 817
minwire 2: 2051
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.984081, 0.000636729, 0.0152819, 
Output 1: 0.00179557, 0.994858, 0.00326828, 7.85183e-05, 
Output 2: 0.979228, 0.0203575, 0.000407842, 6.92532e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1594!
Begin processing the 799th record. run: 74517882 subRun: 1 event: 89099 at 15-Sep-2025 18:12:20 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 495!
Begin processing the 800th record. run: 74517882 subRun: 1 event: 89100 at 15-Sep-2025 18:12:22 CEST
Boundary wire vector sizes: 190, 242, 182
minwire 0: 716
minwire 1: 2098
minwire 2: 472
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000344197, 0.998746, 0.000909673, 
Output 1: 0.00384079, 0.992094, 0.00393138, 0.000133925, 
Output 2: 0.998368, 0.00158637, 4.38692e-05, 1.42145e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1428!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8d/74/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_105_20231204T120339Z_gen_g4_detsim_hitreco__20240509T214623Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/15/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_727_20231203T233659Z_gen_g4_detsim_hitreco__20240509T210759Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/18/6c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_2_20231202T150324Z_gen_g4_detsim_hitreco__20240507T223637Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e1/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_706_20231202T115046Z_gen_g4_detsim_hitreco__20240507T220450Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/87/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_610_20231202T060615Z_gen_g4_detsim_hitreco__20240507T205647Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/38/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_214_20231202T075352Z_gen_g4_detsim_hitreco__20240507T205640Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/75/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_106_20231204T140301Z_gen_g4_detsim_hitreco__20240509T214537Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/64/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_890_20231204T140420Z_gen_g4_detsim_hitreco__20240509T204024Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_268_20231208T041613Z_gen_g4_detsim_hitreco__20240510T034743Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/72/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_1105_20231203T025450Z_gen_g4_detsim_hitreco__20240508T065303Z_reco2.root
.:
total 38944
-rw-r--r--. 1 dune012 dune 14653876 Sep 15 18:12 flatcaf.root
-rw-r--r--. 1 dune012 dune 12246919 Sep 15 18:12 caf.root
-rw-r--r--. 1 dune012 dune 12246919 Sep 15 18:12 caf_fd_hd_atmo_2501_20250915T152012Z.root
-rw-r--r--. 1 dune012 dune   493637 Sep 15 18:12 caf_20250915T152012Z.log
-rw-r--r--. 1 dune012 dune   187040 Sep 15 18:12 jobscript.log
-rw-r--r--. 1 dune012 dune     2106 Sep 15 18:12 caf_20250915T152012Z.file
-rw-r--r--. 1 dune012 dune     2106 Sep 15 18:12 caf_20250915T152012Z.pfns
-rw-r--r--. 1 dune012 dune     2106 Sep 15 17:20 file.list
-rw-r--r--. 1 dune012 dune     2106 Sep 15 18:12 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     1374 Sep 15 17:20 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     1374 Sep 15 18:12 caf_20250915T152012Z.did
-rw-r--r--. 1 dune012 dune     1374 Sep 15 17:20 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 15 17:20 debugprod.log