Jobsub ID 47590.8@dunegpschedd01.fnal.gov

Jobsub ID	47590.8@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 14:43:18
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce16-multicore
Last heartbeat	2025-09-17 15:33:54
From worker node	Hostname	hnode51.pic.es
	cpuinfo	AMD EPYC 7402P 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-17 14:43:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_642_20231208T093626Z_gen_g4_detsim_hitreco__20240510T064214Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_406_20231205T025708Z_gen_g4_detsim_hitreco__20240510T020510Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_359_20231207T000835Z_gen_g4_detsim_hitreco__20240510T060552Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_927_20231204T005409Z_gen_g4_detsim_hitreco__20240509T211309Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_502_20231208T003907Z_gen_g4_detsim_hitreco__20240510T030851Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_305_20231122T061658Z_gen_g4_detsim_hitreco__20240503T060411Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_149_20231122T081113Z_gen_g4_detsim_hitreco__20240503T064902Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_140_20231204T232440Z_gen_g4_detsim_hitreco__20240509T222405Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-17 15:33:54
Saved logs	justin-logs:47590.8-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.335129        400    
reco:energyrecnue:EnergyReco                          0.000125204   0.00260586     0.132952     0.000612938   0.00921188       400    
reco:energyrecnc:EnergyReco                           0.00182456     0.185837      0.808508      0.215555      0.120105        400    
reco:energyrecnumurange:EnergyReco                    0.000224657    0.0517232      4.38055     0.00275101     0.295572        400    
reco:energyrecnumumcs:EnergyReco                      0.00068074     0.0531462      4.63601     0.00338846     0.305459        400    
reco:energyrecnumumcsllhd:EnergyReco                  0.000564417    0.0532887      4.64512     0.00340645     0.305887        400    
reco:opdec:Deconvolution                               0.158803      0.678835       125.184       0.3684        6.23387        400    
reco:ophitspe:OpHitFinderDeco                          0.441759      0.458276      0.500313      0.459538     0.00961373       400    
reco:opflash:OpFlashFinder                            0.000209136   0.00142268    0.00547616    0.00133732    0.000669429      400    
reco:opslicer:OpSlicer                                6.0462e-05      0.27135       1.20289      0.219002      0.217871        400    
reco:rns:RandomNumberSaver                             1.723e-05    3.35206e-05   0.00032077    3.1806e-05    1.64798e-05      400    
reco:anglereconue:NuAngularReco                       0.000135524   0.00178616     0.0989775    0.000413102   0.00688248       400    
reco:anglereconumu:NuAngularReco                      6.2881e-05    0.00158692     0.0917694    0.000216452   0.00664233       400    
reco:anglereconuepfps:NuAngularReco                   9.7603e-05    0.00168702     0.0961288    0.000317285   0.00673172       400    
reco:anglereconumupfps:NuAngularReco                  8.4722e-05    0.00165425     0.0944025    0.000276224   0.00669752       400    
reco:anglerecohits:NuAngularReco                      0.000117364   0.00229107     0.100285     0.000732732   0.00725117       400    
[art]:TriggerResults:TriggerResultInserter             7.951e-06    1.35189e-05   5.1532e-05    1.2491e-05    5.07061e-06      400    
end_path:cafmaker:CAFMaker                            0.00135638     0.0224407      1.12823     0.00508864     0.0795451       400    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3677.75 MB
  Peak resident set size usage (VmHWM): 2012.59 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 17:33:31 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/32/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_502_20231208T003907Z_gen_g4_detsim_hitreco__20240510T030851Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 1074, 1051, 763
minwire 0: 2067
minwire 1: 620
minwire 2: 2136
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.703863, 0.0266898, 0.269447, 
Output 1: 0.135779, 0.483275, 0.268045, 0.112901, 
Output 2: 0.0518674, 0.210634, 0.450076, 0.287422, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2162!
Begin processing the 395th record. run: 6419587 subRun: 1 event: 92795 at 17-Sep-2025 17:33:03 CEST
Boundary wire vector sizes: 200, 124, 148
minwire 0: 568
minwire 1: 1928
minwire 2: 977
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00764093, 0.114434, 0.877925, 
Output 1: 0.572181, 0.419294, 0.00799262, 0.000533036, 
Output 2: 0.991973, 0.00786254, 0.000154993, 9.13608e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2247!
Begin processing the 396th record. run: 6419587 subRun: 1 event: 92796 at 17-Sep-2025 17:33:08 CEST
Boundary wire vector sizes: 333, 394, 294
minwire 0: 658
minwire 1: 2055
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.269108, 0.00504768, 0.725844, 
Output 1: 0.0822354, 0.411038, 0.428195, 0.0785319, 
Output 2: 0.0400639, 0.252001, 0.544383, 0.163551, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1807!
Begin processing the 397th record. run: 6419587 subRun: 1 event: 92797 at 17-Sep-2025 17:33:13 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 594!
Begin processing the 398th record. run: 6419587 subRun: 1 event: 92798 at 17-Sep-2025 17:33:17 CEST
Boundary wire vector sizes: 73, 74, 55
minwire 0: 2592
minwire 1: 1207
minwire 2: 2267
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.447604, 0.0144456, 0.537951, 
Output 1: 0.542475, 0.425807, 0.030067, 0.00165059, 
Output 2: 0.43981, 0.554884, 0.00524984, 5.58528e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1193!
Begin processing the 399th record. run: 6419587 subRun: 1 event: 92799 at 17-Sep-2025 17:33:21 CEST
Boundary wire vector sizes: 831, 656, 580
minwire 0: 802
minwire 1: 1812
minwire 2: 570
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.794984, 0.0361174, 0.168898, 
Output 1: 0.070944, 0.193701, 0.246481, 0.488874, 
Output 2: 0.218037, 0.375132, 0.260225, 0.146606, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2458!
Begin processing the 400th record. run: 6419587 subRun: 1 event: 92800 at 17-Sep-2025 17:33:26 CEST
Boundary wire vector sizes: 178, 162, 125
minwire 0: 668
minwire 1: 2081
minwire 2: 18
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.487654, 0.239706, 0.27264, 
Output 1: 0.408943, 0.492148, 0.092263, 0.00664601, 
Output 2: 0.370987, 0.560583, 0.0650244, 0.00340567, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1685!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3e/8d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_642_20231208T093626Z_gen_g4_detsim_hitreco__20240510T064214Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/86/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_406_20231205T025708Z_gen_g4_detsim_hitreco__20240510T020510Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3b/16/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_359_20231207T000835Z_gen_g4_detsim_hitreco__20240510T060552Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a6/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_927_20231204T005409Z_gen_g4_detsim_hitreco__20240509T211309Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/32/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_502_20231208T003907Z_gen_g4_detsim_hitreco__20240510T030851Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fe/93/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_305_20231122T061658Z_gen_g4_detsim_hitreco__20240503T060411Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/1f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_149_20231122T081113Z_gen_g4_detsim_hitreco__20240503T064902Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a1/fd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_140_20231204T232440Z_gen_g4_detsim_hitreco__20240509T222405Z_reco2.root
.:
total 19748
-rw-r--r-- 1 dune001 dune 7468194 Sep 17 17:33 flatcaf.root
-rw-r--r-- 1 dune001 dune 6178653 Sep 17 17:33 caf.root
-rw-r--r-- 1 dune001 dune 6178653 Sep 17 17:33 caf_fd_hd_atmo_2501_20250917T144347Z.root
-rw-r--r-- 1 dune001 dune  248670 Sep 17 17:33 caf_20250917T144347Z.log
-rw-r--r-- 1 dune001 dune  100873 Sep 17 17:33 jobscript.log
-rw-r--r-- 1 dune001 dune    1492 Sep 17 17:33 caf_20250917T144347Z.file
-rw-r--r-- 1 dune001 dune    1492 Sep 17 17:33 caf_20250917T144347Z.pfns
-rw-r--r-- 1 dune001 dune    1492 Sep 17 16:43 file.list
-rw-r--r-- 1 dune001 dune    1492 Sep 17 17:33 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune    1100 Sep 17 16:43 all-input-dids.txt
-rw-r--r-- 1 dune001 dune    1100 Sep 17 17:33 caf_20250917T144347Z.did
-rw-r--r-- 1 dune001 dune    1100 Sep 17 16:43 did.list
-rw-r--r-- 1 dune001 dune       0 Sep 17 16:43 debugprod.log