Jobsub ID 52326.22@dunegpschedd01.fnal.gov

Jobsub ID	52326.22@dunegpschedd01.fnal.gov
Workflow ID	3298
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-22 15:12:39
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-22 17:14:59
From worker node	Hostname	wn65
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-22 15:14:27
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_398_20231122T052537Z_gen_g4_detsim_hitreco__20240507T171019Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_67_20231207T064309Z_gen_g4_detsim_hitreco__20240510T054912Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1453_2_20231024T144100Z_gen_g4_detsim_hitreco__20240503T045705Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_439_20231203T035824Z_gen_g4_detsim_hitreco__20240508T074351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_718_20231202T130254Z_gen_g4_detsim_hitreco__20240507T222358Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_332_20231203T232058Z_gen_g4_detsim_hitreco__20240509T204745Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_495_20231207T050620Z_gen_g4_detsim_hitreco__20240510T043514Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_246_20231202T170825Z_gen_g4_detsim_hitreco__20240507T203633Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_479_20231205T020437Z_gen_g4_detsim_hitreco__20240509T222738Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-22 17:14:59
Saved logs	justin-logs:52326.22-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ution                               0.593949       1.02625       2.05704       1.01448      0.215722        500    
reco:ophitspe:OpHitFinderDeco                           1.76671       1.84974       8.9507        1.84887      0.319359        500    
reco:opflash:OpFlashFinder                            0.000198038   0.00173204     0.011529     0.00159976    0.000962442      500    
reco:opslicer:OpSlicer                                5.2082e-05     0.360652       1.53853      0.278179      0.305347        500    
reco:rns:RandomNumberSaver                            2.0181e-05    3.7864e-05    0.000378195   3.61065e-05   1.72348e-05      500    
reco:anglereconue:NuAngularReco                       0.000162117   0.00197306     0.081821     0.000466499   0.00590171       500    
reco:anglereconumu:NuAngularReco                      8.8593e-05    0.00180596     0.0807584    0.000324253    0.0058258       500    
reco:anglereconuepfps:NuAngularReco                   0.000124045   0.00192615     0.0809774    0.000430958   0.00586276       500    
reco:anglereconumupfps:NuAngularReco                  0.000112114   0.00190144     0.0812042    0.000410571   0.00586009       500    
reco:anglerecohits:NuAngularReco                      0.000131315   0.00269464     0.0905807    0.000962794   0.00665369       500    
[art]:TriggerResults:TriggerResultInserter             1.049e-05    1.75664e-05   8.6634e-05    1.7006e-05    5.91638e-06      500    
end_path:cafmaker:CAFMaker                            0.00148101     0.0242271      1.0634      0.00504425     0.0754235       500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3818.67 MB
  Peak resident set size usage (VmHWM): 2132.79 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 22-Sep-2025 14:10:31 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/93/a7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_439_20231203T035824Z_gen_g4_detsim_hitreco__20240508T074351Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 2290
minwire 1: 463
minwire 2: 2010
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.485677, 0.381573, 0.13275, 
Output 1: 0.0221224, 0.250763, 0.482411, 0.244704, 
Output 2: 0.249434, 0.670816, 0.077136, 0.00261437, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2073!
Begin processing the 495th record. run: 50609765 subRun: 1 event: 23795 at 22-Sep-2025 14:07:17 -03
Boundary wire vector sizes: 49, 41, 37
minwire 0: 1485
minwire 1: 1449
minwire 2: 1436
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00984822, 0.00187085, 0.988281, 
Output 1: 0.0149248, 0.0844036, 0.879675, 0.0209966, 
Output 2: 0.968786, 0.0294474, 0.00173497, 3.20137e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1333!
Begin processing the 496th record. run: 50609765 subRun: 1 event: 23796 at 22-Sep-2025 14:07:34 -03
Boundary wire vector sizes: 384, 406, 201
minwire 0: 1926
minwire 1: 411
minwire 2: 1747
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0341605, 0.00790586, 0.957934, 
Output 1: 0.614335, 0.332441, 0.0487642, 0.00445992, 
Output 2: 0.0209166, 0.728459, 0.231691, 0.018933, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1754!
Begin processing the 497th record. run: 50609765 subRun: 1 event: 23797 at 22-Sep-2025 14:07:47 -03
Boundary wire vector sizes: 244, 484, 217
minwire 0: 754
minwire 1: 2597
minwire 2: 155
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996937, 0.000415471, 0.00264791, 
Output 1: 0.0174631, 0.859163, 0.118055, 0.00531838, 
Output 2: 0.986678, 0.012815, 0.000494266, 1.26052e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1674!
Begin processing the 498th record. run: 50609765 subRun: 1 event: 23798 at 22-Sep-2025 14:07:58 -03
Boundary wire vector sizes: 423, 511, 443
minwire 0: 1324
minwire 1: 867
minwire 2: 1245
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00120695, 0.98535, 0.0134434, 
Output 1: 0.928861, 0.0681987, 0.00271565, 0.000225076, 
Output 2: 0.96467, 0.0344988, 0.000781766, 4.96754e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2076!
Begin processing the 499th record. run: 50609765 subRun: 1 event: 23799 at 22-Sep-2025 14:08:10 -03
Boundary wire vector sizes: 323, 247, 236
minwire 0: 656
minwire 1: 1759
minwire 2: 667
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0523493, 0.628867, 0.318783, 
Output 1: 0.0197137, 0.148867, 0.667868, 0.163551, 
Output 2: 0.922293, 0.0739339, 0.00363279, 0.000140607, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1714!
Begin processing the 500th record. run: 50609765 subRun: 1 event: 23800 at 22-Sep-2025 14:08:21 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1179!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c2/6a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_398_20231122T052537Z_gen_g4_detsim_hitreco__20240507T171019Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ee/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_67_20231207T064309Z_gen_g4_detsim_hitreco__20240510T054912Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c1/82/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1453_2_20231024T144100Z_gen_g4_detsim_hitreco__20240503T045705Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/f4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/93/a7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_439_20231203T035824Z_gen_g4_detsim_hitreco__20240508T074351Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/76/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_718_20231202T130254Z_gen_g4_detsim_hitreco__20240507T222358Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/3a/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_332_20231203T232058Z_gen_g4_detsim_hitreco__20240509T204745Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_495_20231207T050620Z_gen_g4_detsim_hitreco__20240510T043514Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/91/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_246_20231202T170825Z_gen_g4_detsim_hitreco__20240507T203633Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/86/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_479_20231205T020437Z_gen_g4_detsim_hitreco__20240509T222738Z_reco2.root
.:
total 22532
-rw-r--r-- 1 nobody nobody 8511532 Sep 22 14:10 flatcaf.root
-rw-r--r-- 1 nobody nobody 7039704 Sep 22 14:10 caf.root
-rw-r--r-- 1 nobody nobody 7039704 Sep 22 14:10 caf_fd_hd_atmo_3298_20250922T151441Z.root
-rw-r--r-- 1 nobody nobody  302481 Sep 22 14:10 caf_20250922T151441Z.log
-rw-r--r-- 1 nobody nobody  113544 Sep 22 14:10 jobscript.log
-rw-r--r-- 1 nobody nobody    1944 Sep 22 14:10 caf_20250922T151441Z.file
-rw-r--r-- 1 nobody nobody    1944 Sep 22 14:10 caf_20250922T151441Z.pfns
-rw-r--r-- 1 nobody nobody    1944 Sep 22 12:14 file.list
-rw-r--r-- 1 nobody nobody    1944 Sep 22 14:10 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1370 Sep 22 12:14 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1370 Sep 22 14:10 caf_20250922T151441Z.did
-rw-r--r-- 1 nobody nobody    1370 Sep 22 12:14 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 22 12:15 debugprod.log