Jobsub ID 46595.163@dunegpschedd01.fnal.gov

Jobsub ID	46595.163@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:05:41
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-15 15:33:11
From worker node	Hostname	wn67
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:08:29
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_343_20231206T025225Z_gen_g4_detsim_hitreco__20240510T041706Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_915_20231204T100526Z_gen_g4_detsim_hitreco__20240509T202018Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_850_20231201T170208Z_gen_g4_detsim_hitreco__20240507T200604Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_65_20231204T055354Z_gen_g4_detsim_hitreco__20240509T215119Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_516_20231203T160227Z_gen_g4_detsim_hitreco__20240508T033704Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_236_20231208T092111Z_gen_g4_detsim_hitreco__20240510T063445Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_521_20231208T083421Z_gen_g4_detsim_hitreco__20240510T054955Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_864_20231203T083854Z_gen_g4_detsim_hitreco__20240508T041021Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_65_20231207T022350Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 15:33:11
Saved logs	justin-logs:46595.163-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

897     0.00533387     0.306637        800    
reco:energyrecnumumcsllhd:EnergyReco                  0.000659249    0.0717567      4.91865     0.00518025     0.310559        800    
reco:opdec:Deconvolution                               0.619737       1.0759        2.91033       1.05339       0.25288        800    
reco:ophitspe:OpHitFinderDeco                           1.83062       1.87044       3.69008       1.84561       0.10469        800    
reco:opflash:OpFlashFinder                            0.000217029   0.00221838     0.0317708    0.00178287    0.00262465       800    
reco:opslicer:OpSlicer                                6.7363e-05     0.408001       2.42414      0.319876      0.347787        800    
reco:rns:RandomNumberSaver                            2.4652e-05    3.9677e-05    0.000360876   3.73865e-05   1.8775e-05       800    
reco:anglereconue:NuAngularReco                       0.000197908   0.00235205     0.136578     0.000509372   0.00907049       800    
reco:anglereconumu:NuAngularReco                      0.000102855   0.00219257     0.135367     0.00035504    0.00927867       800    
reco:anglereconuepfps:NuAngularReco                   0.000165177   0.00229643     0.135095      0.0004784    0.00919523       800    
reco:anglereconumupfps:NuAngularReco                  0.000153537   0.00231292     0.137122     0.000455105   0.00936912       800    
reco:anglerecohits:NuAngularReco                      0.000146107   0.00308147     0.149735     0.000995998   0.00973122       800    
[art]:TriggerResults:TriggerResultInserter             1.118e-05    1.84629e-05   0.000128435   1.7851e-05    6.11879e-06      800    
end_path:cafmaker:CAFMaker                            0.00140256     0.105359       62.3196     0.00546811      2.20347        800    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3832.93 MB
  Peak resident set size usage (VmHWM): 2118.13 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 12:25:19 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991504, 0.000940838, 0.00755467, 
Output 1: 0.857116, 0.141917, 0.00089595, 7.04785e-05, 
Output 2: 0.994609, 0.00529229, 9.33376e-05, 5.35195e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 972!
Begin processing the 794th record. run: 50573344 subRun: 1 event: 86494 at 15-Sep-2025 12:24:00 -03
Boundary wire vector sizes: 90, 126, 96
minwire 0: 302
minwire 1: 2387
minwire 2: 140
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0578209, 0.00595795, 0.936221, 
Output 1: 0.00945858, 0.02596, 0.0937889, 0.870793, 
Output 2: 0.882933, 0.110353, 0.00600311, 0.00071089, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1181!
Begin processing the 795th record. run: 50573344 subRun: 1 event: 86495 at 15-Sep-2025 12:24:11 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1221!
Begin processing the 796th record. run: 50573344 subRun: 1 event: 86496 at 15-Sep-2025 12:24:21 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 284!
Begin processing the 797th record. run: 50573344 subRun: 1 event: 86497 at 15-Sep-2025 12:24:31 -03
Boundary wire vector sizes: 717, 721, 385
minwire 0: 1376
minwire 1: 1408
minwire 2: 1154
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00166122, 0.995384, 0.00295499, 
Output 1: 0.00752168, 0.0266255, 0.140499, 0.825354, 
Output 2: 0.979927, 0.0180342, 0.00162177, 0.000416977, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2904!
Begin processing the 798th record. run: 50573344 subRun: 1 event: 86498 at 15-Sep-2025 12:24:44 -03
Boundary wire vector sizes: 69, 29, 57
minwire 0: 1365
minwire 1: 2158
minwire 2: 836
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0115196, 0.00360053, 0.98488, 
Output 1: 0.0521218, 0.812631, 0.132738, 0.00250922, 
Output 2: 0.976549, 0.0226753, 0.000760533, 1.55438e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1825!
Begin processing the 799th record. run: 50573344 subRun: 1 event: 86499 at 15-Sep-2025 12:24:55 -03
Boundary wire vector sizes: 128, 169, 120
minwire 0: 768
minwire 1: 2536
minwire 2: 286
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000329039, 0.998423, 0.00124774, 
Output 1: 0.0211124, 0.939407, 0.0377323, 0.00174846, 
Output 2: 0.99719, 0.00271609, 9.12425e-05, 2.96967e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1346!
Begin processing the 800th record. run: 50573344 subRun: 1 event: 86500 at 15-Sep-2025 12:25:06 -03
Boundary wire vector sizes: 271, 282, 249
minwire 0: 1792
minwire 1: 1386
minwire 2: 1561
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00329293, 0.922602, 0.0741052, 
Output 1: 0.796986, 0.179365, 0.021452, 0.00219659, 
Output 2: 0.959801, 0.0391862, 0.000942928, 6.99506e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2255!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b6/fd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_343_20231206T025225Z_gen_g4_detsim_hitreco__20240510T041706Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a8/e7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_915_20231204T100526Z_gen_g4_detsim_hitreco__20240509T202018Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9b/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_850_20231201T170208Z_gen_g4_detsim_hitreco__20240507T200604Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7c/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_65_20231204T055354Z_gen_g4_detsim_hitreco__20240509T215119Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6f/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_516_20231203T160227Z_gen_g4_detsim_hitreco__20240508T033704Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5e/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_236_20231208T092111Z_gen_g4_detsim_hitreco__20240510T063445Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/41/f8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_521_20231208T083421Z_gen_g4_detsim_hitreco__20240510T054955Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0e/e4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_864_20231203T083854Z_gen_g4_detsim_hitreco__20240508T041021Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6d/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_65_20231207T022350Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root
.:
total 36168
-rw-r--r-- 1 nobody nobody 13629251 Sep 15 12:25 flatcaf.root
-rw-r--r-- 1 nobody nobody 11340241 Sep 15 12:25 caf.root
-rw-r--r-- 1 nobody nobody 11340241 Sep 15 12:25 caf_fd_hd_atmo_2501_20250915T120843Z.root
-rw-r--r-- 1 nobody nobody   477143 Sep 15 12:25 caf_20250915T120843Z.log
-rw-r--r-- 1 nobody nobody   186580 Sep 15 12:25 jobscript.log
-rw-r--r-- 1 nobody nobody     1864 Sep 15 12:25 caf_20250915T120843Z.file
-rw-r--r-- 1 nobody nobody     1864 Sep 15 12:25 caf_20250915T120843Z.pfns
-rw-r--r-- 1 nobody nobody     1864 Sep 15 09:08 file.list
-rw-r--r-- 1 nobody nobody     1864 Sep 15 12:25 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1374 Sep 15 09:08 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1374 Sep 15 12:25 caf_20250915T120843Z.did
-rw-r--r-- 1 nobody nobody     1374 Sep 15 09:08 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 15 09:09 debugprod.log