Jobsub ID 105970.77@dunegpschedd01.fnal.gov

Jobsub ID	105970.77@dunegpschedd01.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:51:37
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 11:14:06
From worker node	Hostname	wn75
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 09:58:27
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_980_20231201T204150Z_gen_g4_detsim_hitreco__20240507T214725Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_0_20231204T054706Z_gen_g4_detsim_hitreco__20240509T214336Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_366_20231201T075834Z_gen_g4_detsim_hitreco__20240507T192507Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_369_20231201T110336Z_gen_g4_detsim_hitreco__20240507T203317Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_566_20231202T125707Z_gen_g4_detsim_hitreco__20240507T222340Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_268_20231203T075109Z_gen_g4_detsim_hitreco__20240509T204538Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_586_20231201T191758Z_gen_g4_detsim_hitreco__20240507T214406Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 11:14:06
Saved logs	justin-logs:105970.77-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eco:energyrecnc:EnergyReco                           0.00347215     0.943579       6.20135       1.05088      0.641111        300    
reco:energyrecnumurange:EnergyReco                    0.000374925    0.054265       2.83442     0.00399877     0.242868        300    
reco:energyrecnumumcs:EnergyReco                      0.00073333     0.0553245      2.87172     0.00493607     0.241461        300    
reco:energyrecnumumcsllhd:EnergyReco                  0.000606925    0.0559707      2.97905     0.00493136     0.247436        300    
reco:opdec:Deconvolution                               0.520161       1.00108       2.61875      0.990411      0.232226        300    
reco:ophitspe:OpHitFinderDeco                           1.50277       1.73819       1.88182       1.84655      0.158807        300    
reco:opflash:OpFlashFinder                            0.000288281   0.00173714    0.00523285    0.00170789    0.000784927      300    
reco:opslicer:OpSlicer                                0.000489379    0.384227       1.36593      0.320188      0.296532        300    
reco:rns:RandomNumberSaver                            2.4591e-05    3.79589e-05   0.000490001   3.6037e-05    2.70015e-05      300    
reco:anglereconue:NuAngularReco                       0.000200408   0.00190801     0.0637279    0.000545657   0.00526015       300    
reco:anglereconumu:NuAngularReco                      8.8323e-05    0.00175927     0.0648029    0.000369395   0.00535964       300    
reco:anglereconuepfps:NuAngularReco                   0.000116135   0.00189705     0.0705009    0.000491515   0.00555231       300    
reco:anglereconumupfps:NuAngularReco                  0.000106634    0.0018541     0.0679121    0.000467802   0.00537753       300    
reco:anglerecohits:NuAngularReco                      0.000156986   0.00265562     0.0731134    0.00102411    0.00602806       300    
[art]:TriggerResults:TriggerResultInserter             1.057e-05    1.80799e-05   6.0623e-05    1.7491e-05    5.46406e-06      300    
end_path:cafmaker:CAFMaker                            0.00111798     0.0243324     0.746478     0.00580481     0.0681096       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3655.31 MB
  Peak resident set size usage (VmHWM): 1969.37 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:09:51 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/2e/11/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_369_20231201T110336Z_gen_g4_detsim_hitreco__20240507T203317Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2278!
Begin processing the 295th record. run: 65974923 subRun: 1 event: 36695 at 23-Sep-2025 08:04:25 -03
Boundary wire vector sizes: 301, 163, 188
minwire 0: 2388
minwire 1: 104
minwire 2: 2694
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00950143, 0.0371823, 0.953316, 
Output 1: 0.90829, 0.0891215, 0.00230895, 0.000279762, 
Output 2: 0.992173, 0.00762323, 0.000185189, 1.82155e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 765!
Begin processing the 296th record. run: 65974923 subRun: 1 event: 36696 at 23-Sep-2025 08:04:36 -03
Boundary wire vector sizes: 321, 671, 213
minwire 0: 391
minwire 1: 2178
minwire 2: 291
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997456, 0.000607923, 0.00193624, 
Output 1: 0.361627, 0.523534, 0.083797, 0.0310416, 
Output 2: 0.952991, 0.0439391, 0.00279423, 0.000275433, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1305!
Begin processing the 297th record. run: 65974923 subRun: 1 event: 36697 at 23-Sep-2025 08:04:47 -03
Boundary wire vector sizes: 2628, 3390, 2852
minwire 0: 1000
minwire 1: 389
minwire 2: 1041
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0928182, 0.663685, 0.243497, 
Output 1: 0.216059, 0.318064, 0.207383, 0.258494, 
Output 2: 0.0275152, 0.0770889, 0.244216, 0.65118, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2774!
Begin processing the 298th record. run: 65974923 subRun: 1 event: 36698 at 23-Sep-2025 08:05:10 -03
Boundary wire vector sizes: 499, 523, 462
minwire 0: 892
minwire 1: 729
minwire 2: 863
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00495027, 0.213147, 0.781902, 
Output 1: 0.546548, 0.443529, 0.00948672, 0.000435498, 
Output 2: 0.994686, 0.00517037, 0.000134255, 9.85717e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2029!
Begin processing the 299th record. run: 65974923 subRun: 1 event: 36699 at 23-Sep-2025 08:05:22 -03
Boundary wire vector sizes: 574, 554, 521
minwire 0: 1765
minwire 1: 643
minwire 2: 1875
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000559115, 0.997055, 0.00238567, 
Output 1: 0.197535, 0.690435, 0.101509, 0.0105204, 
Output 2: 0.507681, 0.482503, 0.00967335, 0.000143161, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2085!
Begin processing the 300th record. run: 65974923 subRun: 1 event: 36700 at 23-Sep-2025 08:05:34 -03
Boundary wire vector sizes: 327, 309, 321
minwire 0: 1451
minwire 1: 181
minwire 2: 1610
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00338649, 0.0058035, 0.99081, 
Output 1: 0.533894, 0.4612, 0.0035027, 0.00140287, 
Output 2: 0.995358, 0.00441966, 0.000185116, 3.71262e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1678!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/31/c1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_980_20231201T204150Z_gen_g4_detsim_hitreco__20240507T214725Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f7/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_0_20231204T054706Z_gen_g4_detsim_hitreco__20240509T214336Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fa/2e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_366_20231201T075834Z_gen_g4_detsim_hitreco__20240507T192507Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/2e/11/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_369_20231201T110336Z_gen_g4_detsim_hitreco__20240507T203317Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/02/3e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_566_20231202T125707Z_gen_g4_detsim_hitreco__20240507T222340Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/61/8d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_268_20231203T075109Z_gen_g4_detsim_hitreco__20240509T204538Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/13/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_586_20231201T191758Z_gen_g4_detsim_hitreco__20240507T214406Z_reco2.root
.:
total 15940
-rw-r--r-- 1 nobody nobody 6023096 Sep 23 08:09 flatcaf.root
-rw-r--r-- 1 nobody nobody 4964932 Sep 23 08:09 caf.root
-rw-r--r-- 1 nobody nobody 4964932 Sep 23 08:09 caf_fd_hd_atmo_3301_20250923T095840Z.root
-rw-r--r-- 1 nobody nobody  191542 Sep 23 08:09 caf_20250923T095840Z.log
-rw-r--r-- 1 nobody nobody   82447 Sep 23 08:09 jobscript.log
-rw-r--r-- 1 nobody nobody    1426 Sep 23 08:09 caf_20250923T095840Z.file
-rw-r--r-- 1 nobody nobody    1426 Sep 23 08:09 caf_20250923T095840Z.pfns
-rw-r--r-- 1 nobody nobody    1426 Sep 23 06:58 file.list
-rw-r--r-- 1 nobody nobody    1426 Sep 23 08:09 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     959 Sep 23 06:58 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     959 Sep 23 08:09 caf_20250923T095840Z.did
-rw-r--r-- 1 nobody nobody     959 Sep 23 06:58 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 06:59 debugprod.log