Jobsub ID 102322.22@dunegpschedd02.fnal.gov

Jobsub ID	102322.22@dunegpschedd02.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 10:17:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce04
Last heartbeat	2025-09-23 14:05:39
From worker node	Hostname	wn122
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:19:18
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_453_20231208T064906Z_gen_g4_detsim_hitreco__20240510T054823Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1055_20231203T065049Z_gen_g4_detsim_hitreco__20240508T042136Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_66_20231207T061711Z_gen_g4_detsim_hitreco__20240510T023605Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_1177_20231203T163340Z_gen_g4_detsim_hitreco__20240508T052316Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_33_20231207T060341Z_gen_g4_detsim_hitreco__20240510T063020Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_449_20231118T192044Z_gen_g4_detsim_hitreco__20240503T060620Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_617_20231205T045921Z_gen_g4_detsim_hitreco__20240510T021516Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_1091_20231204T053937Z_gen_g4_detsim_hitreco__20240509T212331Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_135_20231207T081705Z_gen_g4_detsim_hitreco__20240510T025948Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_1168_20231204T232403Z_gen_g4_detsim_hitreco__20240509T221123Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 14:05:39
Saved logs	justin-logs:102322.22-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

volution                               0.524649       1.05046       2.73521       1.0049       0.291012        800    
reco:ophitspe:OpHitFinderDeco                           1.42253       1.75132       1.88743       1.84479      0.141297        800    
reco:opflash:OpFlashFinder                            0.000193618   0.00169712    0.00441385    0.00164372    0.000793227      800    
reco:opslicer:OpSlicer                                6.7363e-05     0.342244       1.50765      0.273298      0.291005        800    
reco:rns:RandomNumberSaver                            2.2891e-05    3.79473e-05   0.000460369   3.6146e-05    1.8088e-05       800    
reco:anglereconue:NuAngularReco                       0.000202038   0.00236534     0.114601     0.000448978    0.0088535       800    
reco:anglereconumu:NuAngularReco                      0.000113815   0.00219358     0.115263     0.000305808   0.00873523       800    
reco:anglereconuepfps:NuAngularReco                   0.000165426    0.0023109     0.113006     0.000422642   0.00873099       800    
reco:anglereconumupfps:NuAngularReco                  0.000154786    0.0022965     0.111827     0.000395886   0.00876162       800    
reco:anglerecohits:NuAngularReco                      0.000131845   0.00310772     0.124426     0.000918448   0.00967964       800    
[art]:TriggerResults:TriggerResultInserter            1.0451e-05    1.77262e-05   6.8292e-05    1.72855e-05   5.31171e-06      800    
end_path:cafmaker:CAFMaker                            0.00144571     0.0276304      1.29597     0.00487312     0.100956        800    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3806.4 MB
  Peak resident set size usage (VmHWM): 2123.61 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 11:05:14 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Invalid address
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/b1/3c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_1091_20231204T053937Z_gen_g4_detsim_hitreco__20240509T212331Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 443
minwire 1: 1439
minwire 2: 444
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0230337, 0.472062, 0.504905, 
Output 1: 0.756909, 0.236008, 0.00671615, 0.000367607, 
Output 2: 0.803356, 0.191988, 0.00449185, 0.000163641, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1120!
Begin processing the 795th record. run: 66020489 subRun: 1 event: 61795 at 23-Sep-2025 10:58:54 -03
Boundary wire vector sizes: 67, 49, 44
minwire 0: 2705
minwire 1: 869
minwire 2: 2515
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000676079, 0.998268, 0.00105582, 
Output 1: 0.00954815, 0.986112, 0.00421278, 0.000126593, 
Output 2: 0.998718, 0.00124029, 4.03263e-05, 1.32134e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1158!
Begin processing the 796th record. run: 66020489 subRun: 1 event: 61796 at 23-Sep-2025 10:59:04 -03
Boundary wire vector sizes: 2862, 2726, 2899
minwire 0: 553
minwire 1: 1571
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.94759, 0.00865885, 0.0437508, 
Output 1: 0.26264, 0.500985, 0.142329, 0.0940457, 
Output 2: 0.00776443, 0.0244299, 0.155079, 0.812726, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3228!
Begin processing the 797th record. run: 66020489 subRun: 1 event: 61797 at 23-Sep-2025 10:59:35 -03
Boundary wire vector sizes: 12848, 10100, 5634
minwire 0: 441
minwire 1: 31
minwire 2: 243
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.111879, 0.882288, 0.00583384, 
Output 1: 0.26814, 0.427261, 0.141006, 0.163593, 
Output 2: 0.0147022, 0.0622882, 0.302309, 0.6207, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3407!
Begin processing the 798th record. run: 66020489 subRun: 1 event: 61798 at 23-Sep-2025 11:02:04 -03
Boundary wire vector sizes: 497, 395, 405
minwire 0: 838
minwire 1: 2201
minwire 2: 455
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000252261, 0.998162, 0.00158598, 
Output 1: 0.808659, 0.189727, 0.0015493, 6.40748e-05, 
Output 2: 0.997179, 0.00277777, 4.15498e-05, 1.58097e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2089!
Begin processing the 799th record. run: 66020489 subRun: 1 event: 61799 at 23-Sep-2025 11:02:16 -03
Boundary wire vector sizes: 45, 56, 30
minwire 0: 2498
minwire 1: 176
minwire 2: 2276
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.792773, 0.0281468, 0.17908, 
Output 1: 0.177319, 0.598502, 0.211926, 0.0122526, 
Output 2: 0.654084, 0.339171, 0.00664592, 9.85171e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1234!
Begin processing the 800th record. run: 66020489 subRun: 1 event: 61800 at 23-Sep-2025 11:02:27 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 902!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/af/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_453_20231208T064906Z_gen_g4_detsim_hitreco__20240510T054823Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e7/03/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1055_20231203T065049Z_gen_g4_detsim_hitreco__20240508T042136Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/61/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_66_20231207T061711Z_gen_g4_detsim_hitreco__20240510T023605Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_1177_20231203T163340Z_gen_g4_detsim_hitreco__20240508T052316Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1f/77/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_33_20231207T060341Z_gen_g4_detsim_hitreco__20240510T063020Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/a4/7f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_449_20231118T192044Z_gen_g4_detsim_hitreco__20240503T060620Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/98/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_617_20231205T045921Z_gen_g4_detsim_hitreco__20240510T021516Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/b1/3c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_1091_20231204T053937Z_gen_g4_detsim_hitreco__20240509T212331Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4a/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_135_20231207T081705Z_gen_g4_detsim_hitreco__20240510T025948Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a2/f2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_1168_20231204T232403Z_gen_g4_detsim_hitreco__20240509T221123Z_reco2.root
.:
total 39280
-rw-r--r-- 1 nobody nobody 14782791 Sep 23 11:05 flatcaf.root
-rw-r--r-- 1 nobody nobody 12359093 Sep 23 11:05 caf.root
-rw-r--r-- 1 nobody nobody 12359093 Sep 23 11:05 caf_fd_hd_atmo_3301_20250923T101931Z.root
-rw-r--r-- 1 nobody nobody   474522 Sep 23 11:05 caf_20250923T101931Z.log
-rw-r--r-- 1 nobody nobody   174555 Sep 23 11:05 jobscript.log
-rw-r--r-- 1 nobody nobody     1930 Sep 23 11:05 caf_20250923T101931Z.file
-rw-r--r-- 1 nobody nobody     1930 Sep 23 11:05 caf_20250923T101931Z.pfns
-rw-r--r-- 1 nobody nobody     1930 Sep 23 07:19 file.list
-rw-r--r-- 1 nobody nobody     1930 Sep 23 11:05 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1378 Sep 23 07:19 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1378 Sep 23 11:05 caf_20250923T101931Z.did
-rw-r--r-- 1 nobody nobody     1378 Sep 23 07:19 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 23 07:19 debugprod.log