Jobsub ID 106434.5@dunegpschedd01.fnal.gov

Jobsub ID	106434.5@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:57:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 11:44:26
From worker node	Hostname	wn76
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:00:41
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6447068_380_20231205T111536Z_gen_g4_detsim_hitreco__20240510T060746Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_922_20231201T230716Z_gen_g4_detsim_hitreco__20240507T204830Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_725_20231207T115556Z_gen_g4_detsim_hitreco__20240510T054448Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_949_20231204T073556Z_gen_g4_detsim_hitreco__20240509T212048Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_522_20231203T062123Z_gen_g4_detsim_hitreco__20240508T075126Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_194_20231201T121430Z_gen_g4_detsim_hitreco__20240507T194359Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_464_20231201T170120Z_gen_g4_detsim_hitreco__20240507T195153Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_471_20231202T160902Z_gen_g4_detsim_hitreco__20240508T043348Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_41_20231204T005724Z_gen_g4_detsim_hitreco__20240509T214802Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_126_20231122T051707Z_gen_g4_detsim_hitreco__20240503T063011Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 11:44:26
Saved logs	justin-logs:106434.5-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

yrecnumumcsllhd:EnergyReco                  0.000550103    0.0744721      4.02177     0.00454631     0.319781        400    
reco:opdec:Deconvolution                               0.593341       1.01739       3.30063       1.00447      0.251627        400    
reco:ophitspe:OpHitFinderDeco                           1.76857       1.8141        1.88753       1.81639      0.0354148       400    
reco:opflash:OpFlashFinder                            0.000199768   0.00170691    0.00529496    0.00164893    0.000822366      400    
reco:opslicer:OpSlicer                                6.9043e-05     0.367021       1.50622      0.293633      0.314723        400    
reco:rns:RandomNumberSaver                            2.4591e-05     3.92e-05     0.000523851   3.70815e-05   2.57068e-05      400    
reco:anglereconue:NuAngularReco                       0.000210139    0.0025793      0.10093     0.000478705   0.00832145       400    
reco:anglereconumu:NuAngularReco                      0.000112834    0.0024189     0.0992448    0.000324388   0.00825958       400    
reco:anglereconuepfps:NuAngularReco                   0.000161687   0.00253266     0.100234     0.000442798   0.00828359       400    
reco:anglereconumupfps:NuAngularReco                  0.000155747   0.00251328     0.100397     0.000411878   0.00830966       400    
reco:anglerecohits:NuAngularReco                      0.000140536   0.00337308     0.108061     0.000918528   0.00923443       400    
[art]:TriggerResults:TriggerResultInserter             1.088e-05    1.8117e-05    8.4933e-05    1.69255e-05   7.50329e-06      400    
end_path:cafmaker:CAFMaker                            0.00141335     0.0319629      1.08643     0.00504099     0.0987532       400    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3699.85 MB
  Peak resident set size usage (VmHWM): 2002.95 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:40:11 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_522_20231203T062123Z_gen_g4_detsim_hitreco__20240508T075126Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 2309
minwire 1: 752
minwire 2: 2228
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997569, 0.000609703, 0.00182089, 
Output 1: 0.318365, 0.521121, 0.112128, 0.0483865, 
Output 2: 0.867989, 0.118215, 0.0123832, 0.00141266, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1993!
Begin processing the 395th record. run: 74517585 subRun: 1 event: 94995 at 23-Sep-2025 08:39:02 -03
Boundary wire vector sizes: 249, 261, 192
minwire 0: 2061
minwire 1: 1481
minwire 2: 1694
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000242542, 0.998889, 0.000868047, 
Output 1: 0.00645681, 0.988917, 0.00447191, 0.000154262, 
Output 2: 0.998583, 0.0013798, 3.62866e-05, 1.07772e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2382!
Begin processing the 396th record. run: 74517585 subRun: 1 event: 94996 at 23-Sep-2025 08:39:13 -03
Boundary wire vector sizes: 108, 148, 133
minwire 0: 994
minwire 1: 1667
minwire 2: 752
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000221027, 0.998082, 0.00169736, 
Output 1: 0.705151, 0.293098, 0.00169263, 5.86457e-05, 
Output 2: 0.998783, 0.00119528, 2.0915e-05, 7.8837e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1599!
Begin processing the 397th record. run: 74517585 subRun: 1 event: 94997 at 23-Sep-2025 08:39:25 -03
Boundary wire vector sizes: 325, 304, 302
minwire 0: 415
minwire 1: 2270
minwire 2: 132
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00341715, 0.00221715, 0.994366, 
Output 1: 0.170728, 0.772271, 0.0529229, 0.00407867, 
Output 2: 0.998125, 0.00176663, 0.000101762, 6.34586e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1572!
Begin processing the 398th record. run: 74517585 subRun: 1 event: 94998 at 23-Sep-2025 08:39:36 -03
Boundary wire vector sizes: 386, 354, 307
minwire 0: 686
minwire 1: 2778
minwire 2: 132
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00104701, 0.996268, 0.00268459, 
Output 1: 0.00737902, 0.0579324, 0.701359, 0.23333, 
Output 2: 0.98868, 0.0105477, 0.000721865, 5.08041e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1688!
Begin processing the 399th record. run: 74517585 subRun: 1 event: 94999 at 23-Sep-2025 08:39:48 -03
Boundary wire vector sizes: 27, 46, 32
minwire 0: 1379
minwire 1: 1414
minwire 2: 1001
Used alternate method to get min and max wires due to vertex determination failure: 379, 678
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.05658, 0.00341056, 0.940009, 
Output 1: 0.0393186, 0.473231, 0.467844, 0.0196062, 
Output 2: 0.971999, 0.0268621, 0.00110994, 2.87078e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2079!
Begin processing the 400th record. run: 74517585 subRun: 1 event: 95000 at 23-Sep-2025 08:39:59 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 307!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/65/38/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6447068_380_20231205T111536Z_gen_g4_detsim_hitreco__20240510T060746Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/64/63/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_922_20231201T230716Z_gen_g4_detsim_hitreco__20240507T204830Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/1b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_725_20231207T115556Z_gen_g4_detsim_hitreco__20240510T054448Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_949_20231204T073556Z_gen_g4_detsim_hitreco__20240509T212048Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_522_20231203T062123Z_gen_g4_detsim_hitreco__20240508T075126Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5c/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_194_20231201T121430Z_gen_g4_detsim_hitreco__20240507T194359Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1f/57/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_464_20231201T170120Z_gen_g4_detsim_hitreco__20240507T195153Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bd/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_471_20231202T160902Z_gen_g4_detsim_hitreco__20240508T043348Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/79/b0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_41_20231204T005724Z_gen_g4_detsim_hitreco__20240509T214802Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/80/a7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_126_20231122T051707Z_gen_g4_detsim_hitreco__20240503T063011Z_reco2.root
.:
total 21596
-rw-r--r-- 1 nobody nobody 8160840 Sep 23 08:40 flatcaf.root
-rw-r--r-- 1 nobody nobody 6759645 Sep 23 08:40 caf.root
-rw-r--r-- 1 nobody nobody 6759645 Sep 23 08:40 caf_fd_hd_atmo_3302_20250923T100054Z.root
-rw-r--r-- 1 nobody nobody  252177 Sep 23 08:40 caf_20250923T100054Z.log
-rw-r--r-- 1 nobody nobody  100262 Sep 23 08:40 jobscript.log
-rw-r--r-- 1 nobody nobody    1958 Sep 23 08:40 caf_20250923T100054Z.file
-rw-r--r-- 1 nobody nobody    1958 Sep 23 08:40 caf_20250923T100054Z.pfns
-rw-r--r-- 1 nobody nobody    1958 Sep 23 07:00 file.list
-rw-r--r-- 1 nobody nobody    1958 Sep 23 08:40 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:00 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 08:40 caf_20250923T100054Z.did
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:00 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:01 debugprod.log