Jobsub ID 103541.53@dunegpschedd01.fnal.gov

Jobsub ID	103541.53@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:19:36
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-23 10:25:59
From worker node	Hostname	wn56
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 09:22:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_316_20231208T002247Z_gen_g4_detsim_hitreco__20240510T030524Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_696_20231204T042955Z_gen_g4_detsim_hitreco__20240509T195813Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_104_20231208T043456Z_gen_g4_detsim_hitreco__20240510T065341Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_576_20231202T075823Z_gen_g4_detsim_hitreco__20240507T200725Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_156_20231207T041538Z_gen_g4_detsim_hitreco__20240510T044900Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_171_20231203T043748Z_gen_g4_detsim_hitreco__20240508T073300Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_1186_20231202T193134Z_gen_g4_detsim_hitreco__20240508T061330Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_1053_20231203T025319Z_gen_g4_detsim_hitreco__20240508T064850Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 10:25:59
Saved logs	justin-logs:103541.53-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000433613    0.117143       5.17337     0.00359358     0.548098        200    
reco:energyrecnumumcs:EnergyReco                      0.000827378    0.118696       5.16389     0.00449255     0.550025        200    
reco:energyrecnumumcsllhd:EnergyReco                  0.00066239     0.116765       5.18177     0.00447874     0.540785        200    
reco:opdec:Deconvolution                               0.628737       1.12455       3.70314       1.0363       0.324951        200    
reco:ophitspe:OpHitFinderDeco                           1.83973        1.855        1.88057       1.85581     0.00737115       200    
reco:opflash:OpFlashFinder                            0.000394791   0.00194831    0.00524427    0.00182874    0.000886136      200    
reco:opslicer:OpSlicer                                0.000320246    0.350738       1.3025       0.257853      0.292955        200    
reco:rns:RandomNumberSaver                            2.5319e-05    6.08979e-05   0.00186334    4.5021e-05    0.000139176      200    
reco:anglereconue:NuAngularReco                       0.000222239   0.00258226     0.0410114    0.000588847   0.00623696       200    
reco:anglereconumu:NuAngularReco                      0.000137515   0.00236575     0.0415177    0.00041613    0.00618308       200    
reco:anglereconuepfps:NuAngularReco                   0.000207111   0.00254227     0.0423424    0.000577156   0.00624076       200    
reco:anglereconumupfps:NuAngularReco                  0.000198608   0.00247914     0.0422128    0.000532868   0.00622721       200    
reco:anglerecohits:NuAngularReco                      0.000193942   0.00347705     0.0469243    0.00114913    0.00708323       200    
[art]:TriggerResults:TriggerResultInserter            1.2605e-05    2.27637e-05    7.314e-05    2.1299e-05    6.20308e-06      200    
end_path:cafmaker:CAFMaker                            0.00170911     0.0331743     0.468697     0.00785716     0.0736125       200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3816.86 MB
  Peak resident set size usage (VmHWM): 2149.6 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 07:21:49 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/16/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_104_20231208T043456Z_gen_g4_detsim_hitreco__20240510T065341Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 285, 438, 328
minwire 0: 1723
minwire 1: 641
minwire 2: 1848
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996609, 0.000582566, 0.00280813, 
Output 1: 0.0132142, 0.976973, 0.00960253, 0.000210391, 
Output 2: 0.996707, 0.00320971, 8.20097e-05, 1.73549e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1361!
Begin processing the 195th record. run: 6420351 subRun: 1 event: 69695 at 23-Sep-2025 07:17:12 -03
Boundary wire vector sizes: 79, 75, 72
minwire 0: 2437
minwire 1: 770
minwire 2: 2349
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.955493, 0.00264501, 0.0418618, 
Output 1: 0.65499, 0.342207, 0.00265025, 0.000153271, 
Output 2: 0.955906, 0.0432654, 0.000803113, 2.53467e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1318!
Begin processing the 196th record. run: 6420351 subRun: 1 event: 69696 at 23-Sep-2025 07:17:24 -03
Boundary wire vector sizes: 168, 119, 125
minwire 0: 818
minwire 1: 1865
minwire 2: 690
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0111019, 0.549684, 0.439214, 
Output 1: 0.469658, 0.527537, 0.00254024, 0.000265029, 
Output 2: 0.992033, 0.00763384, 0.000301504, 3.14094e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1314!
Begin processing the 197th record. run: 6420351 subRun: 1 event: 69697 at 23-Sep-2025 07:17:36 -03
Boundary wire vector sizes: 24, 70, 54
minwire 0: 2611
minwire 1: 1134
minwire 2: 2298
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00326427, 0.00279099, 0.993945, 
Output 1: 0.00749974, 0.987535, 0.00475154, 0.000213277, 
Output 2: 0.997602, 0.00233674, 5.8904e-05, 2.1844e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1465!
Begin processing the 198th record. run: 6420351 subRun: 1 event: 69698 at 23-Sep-2025 07:17:47 -03
Boundary wire vector sizes: 166, 299, 136
minwire 0: 2766
minwire 1: 624
minwire 2: 2593
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99205, 0.000407838, 0.00754198, 
Output 1: 0.996635, 0.00310309, 0.000150876, 0.000111345, 
Output 2: 0.992466, 0.00734856, 0.000145188, 4.01523e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1301!
Begin processing the 199th record. run: 6420351 subRun: 1 event: 69699 at 23-Sep-2025 07:17:59 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 173!
Begin processing the 200th record. run: 6420351 subRun: 1 event: 69700 at 23-Sep-2025 07:18:09 -03
Boundary wire vector sizes: 91, 59, 60
minwire 0: 2256
minwire 1: 882
minwire 2: 2259
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000390884, 0.998534, 0.00107486, 
Output 1: 0.00432995, 0.989922, 0.00556662, 0.000181614, 
Output 2: 0.998492, 0.00146158, 4.52114e-05, 1.44846e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1273!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/2f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_316_20231208T002247Z_gen_g4_detsim_hitreco__20240510T030524Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/38/5e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_696_20231204T042955Z_gen_g4_detsim_hitreco__20240509T195813Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/16/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_104_20231208T043456Z_gen_g4_detsim_hitreco__20240510T065341Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/91/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_576_20231202T075823Z_gen_g4_detsim_hitreco__20240507T200725Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7b/b3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_156_20231207T041538Z_gen_g4_detsim_hitreco__20240510T044900Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/66/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_171_20231203T043748Z_gen_g4_detsim_hitreco__20240508T073300Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/18/55/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_1186_20231202T193134Z_gen_g4_detsim_hitreco__20240508T061330Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e0/8f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_1053_20231203T025319Z_gen_g4_detsim_hitreco__20240508T064850Z_reco2.root
.:
total 8764
-rw-r--r-- 1 nobody nobody 3373344 Sep 23 07:21 flatcaf.root
-rw-r--r-- 1 nobody nobody 2679925 Sep 23 07:21 caf.root
-rw-r--r-- 1 nobody nobody 2679925 Sep 23 07:21 caf_fd_hd_atmo_3302_20250923T092226Z.root
-rw-r--r-- 1 nobody nobody  128883 Sep 23 07:21 caf_20250923T092226Z.log
-rw-r--r-- 1 nobody nobody   59235 Sep 23 07:21 jobscript.log
-rw-r--r-- 1 nobody nobody    1579 Sep 23 07:21 caf_20250923T092226Z.file
-rw-r--r-- 1 nobody nobody    1579 Sep 23 07:21 caf_20250923T092226Z.pfns
-rw-r--r-- 1 nobody nobody    1579 Sep 23 06:22 file.list
-rw-r--r-- 1 nobody nobody    1579 Sep 23 07:21 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1103 Sep 23 06:22 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1103 Sep 23 07:21 caf_20250923T092226Z.did
-rw-r--r-- 1 nobody nobody    1103 Sep 23 06:22 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 06:22 debugprod.log