Jobsub ID 46678.84@dunegpschedd01.fnal.gov

Jobsub ID	46678.84@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 14:37:47
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-09-15 18:06:19
From worker node	Hostname	mh-7763-11.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 14:40:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_784_20231202T070440Z_gen_g4_detsim_hitreco__20240507T204631Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_871_20231204T044620Z_gen_g4_detsim_hitreco__20240509T200939Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1052_20231204T010931Z_gen_g4_detsim_hitreco__20240509T203056Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_767_20231208T024419Z_gen_g4_detsim_hitreco__20240510T032543Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_351_20231207T024400Z_gen_g4_detsim_hitreco__20240510T033424Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_588_20231207T155332Z_gen_g4_detsim_hitreco__20240510T044254Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_8_20231207T114019Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_24_20231204T120320Z_gen_g4_detsim_hitreco__20240509T215913Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 18:06:19
Saved logs	justin-logs:46678.84-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

184478       1000    
reco:energyrecnumumcs:EnergyReco                      0.000571552    0.0433111      2.46149     0.00372406     0.178933       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000404564    0.0427666      2.4896       0.0036512     0.177431       1000    
reco:opdec:Deconvolution                                0.4151       0.798863       2.91697      0.775988      0.235295       1000    
reco:ophitspe:OpHitFinderDeco                           1.20326       1.34518       2.0854        1.34887      0.0618078      1000    
reco:opflash:OpFlashFinder                            0.000162378   0.00153993    0.00855806    0.00145074    0.000710514     1000    
reco:opslicer:OpSlicer                                5.3179e-05     0.286358       1.28505      0.222615       0.2373        1000    
reco:rns:RandomNumberSaver                            2.1559e-05    4.96945e-05   0.000595352   4.07195e-05   2.87453e-05     1000    
reco:anglereconue:NuAngularReco                       0.000161708   0.00179186     0.111547     0.000452784   0.00655102      1000    
reco:anglereconumu:NuAngularReco                      8.0699e-05    0.00158373      0.11058     0.000275296   0.00637403      1000    
reco:anglereconuepfps:NuAngularReco                   0.000109388   0.00165231     0.110646     0.00038639    0.00606821      1000    
reco:anglereconumupfps:NuAngularReco                  0.000103708   0.00161495     0.111518     0.00034379     0.0060646      1000    
reco:anglerecohits:NuAngularReco                      0.000106718   0.00235922     0.120781     0.000848558   0.00678118      1000    
[art]:TriggerResults:TriggerResultInserter             8.91e-06     1.52068e-05   6.8779e-05    1.4205e-05    6.03746e-06     1000    
end_path:cafmaker:CAFMaker                            0.00129468     0.0370426      14.7122     0.00623602      0.47063       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3751.91 MB
  Peak resident set size usage (VmHWM): 2325.09 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 11:05:57 PDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 1054, 823, 979
minwire 0: 1140
minwire 1: 989
minwire 2: 1284
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.514951, 0.0415424, 0.443506, 
Output 1: 0.351071, 0.271804, 0.161654, 0.215471, 
Output 2: 0.108228, 0.250092, 0.306208, 0.335471, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2434!
Begin processing the 995th record. run: 6422149 subRun: 1 event: 2495 at 15-Sep-2025 11:05:06 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 341!
Begin processing the 996th record. run: 6422149 subRun: 1 event: 2496 at 15-Sep-2025 11:05:12 PDT
Boundary wire vector sizes: 154, 91, 101
minwire 0: 2016
minwire 1: 935
minwire 2: 2179
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000858351, 0.991826, 0.00731603, 
Output 1: 0.995117, 0.00459472, 0.000143706, 0.000144306, 
Output 2: 0.997951, 0.00197383, 5.20623e-05, 2.29716e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1373!
Begin processing the 997th record. run: 6422149 subRun: 1 event: 2497 at 15-Sep-2025 11:05:19 PDT
Boundary wire vector sizes: 316, 220, 234
minwire 0: 113
minwire 1: 2331
minwire 2: 17
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000332701, 0.998553, 0.00111473, 
Output 1: 0.00448886, 0.991614, 0.0037828, 0.00011384, 
Output 2: 0.998307, 0.00164183, 4.91034e-05, 1.51865e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1112!
Begin processing the 998th record. run: 6422149 subRun: 1 event: 2498 at 15-Sep-2025 11:05:27 PDT
Boundary wire vector sizes: 108, 194, 101
minwire 0: 1478
minwire 1: 1756
minwire 2: 1232
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99679, 0.000723238, 0.00248673, 
Output 1: 0.00527927, 0.990238, 0.00436534, 0.000116944, 
Output 2: 0.996223, 0.00369118, 8.36522e-05, 1.82129e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2107!
Begin processing the 999th record. run: 6422149 subRun: 1 event: 2499 at 15-Sep-2025 11:05:34 PDT
Boundary wire vector sizes: 1515, 2719, 950
minwire 0: 2464
minwire 1: 0
minwire 2: 2711
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000963545, 0.992824, 0.00621253, 
Output 1: 0.162369, 0.717178, 0.110386, 0.0100675, 
Output 2: 0.610543, 0.353433, 0.034266, 0.00175816, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2266!
Begin processing the 1000th record. run: 6422149 subRun: 1 event: 2500 at 15-Sep-2025 11:05:48 PDT
Boundary wire vector sizes: 252, 219, 179
minwire 0: 126
minwire 1: 2088
minwire 2: 123
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.185419, 0.184929, 0.629652, 
Output 1: 0.0245703, 0.112428, 0.282928, 0.580073, 
Output 2: 0.708321, 0.274413, 0.016347, 0.000919381, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1014!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8c/3f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_784_20231202T070440Z_gen_g4_detsim_hitreco__20240507T204631Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f5/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_871_20231204T044620Z_gen_g4_detsim_hitreco__20240509T200939Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/28/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/8a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1052_20231204T010931Z_gen_g4_detsim_hitreco__20240509T203056Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_767_20231208T024419Z_gen_g4_detsim_hitreco__20240510T032543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/87/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_351_20231207T024400Z_gen_g4_detsim_hitreco__20240510T033424Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c9/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_588_20231207T155332Z_gen_g4_detsim_hitreco__20240510T044254Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/86/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_8_20231207T114019Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/98/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_24_20231204T120320Z_gen_g4_detsim_hitreco__20240509T215913Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root
.:
total 42640
-rw-r--r-- 1 cuser cuser 16023647 Sep 15 11:05 flatcaf.root
-rw-r--r-- 1 cuser cuser 13368544 Sep 15 11:05 caf.root
-rw-r--r-- 1 cuser cuser 13368544 Sep 15 11:05 caf_fd_hd_atmo_2501_20250915T144043Z.root
-rw-r--r-- 1 cuser cuser   592206 Sep 15 11:05 caf_20250915T144043Z.log
-rw-r--r-- 1 cuser cuser   216593 Sep 15 11:05 jobscript.log
-rw-r--r-- 1 cuser cuser     1875 Sep 15 11:05 caf_20250915T144043Z.file
-rw-r--r-- 1 cuser cuser     1875 Sep 15 11:05 caf_20250915T144043Z.pfns
-rw-r--r-- 1 cuser cuser     1875 Sep 15 07:40 file.list
-rw-r--r-- 1 cuser cuser     1875 Sep 15 11:05 justin-processed-pfns.txt
-rw-r--r-- 1 cuser cuser     1371 Sep 15 07:40 all-input-dids.txt
-rw-r--r-- 1 cuser cuser     1371 Sep 15 11:05 caf_20250915T144043Z.did
-rw-r--r-- 1 cuser cuser     1371 Sep 15 07:40 did.list
-rw-r--r-- 1 cuser cuser        0 Sep 15 07:41 debugprod.log