Jobsub ID 40120.99@dunegpschedd02.fnal.gov

Jobsub ID	40120.99@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 16:45:54
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce02
Last heartbeat	2025-09-15 17:04:59
From worker node	Hostname	ccwcondor0605
	cpuinfo	AMD EPYC 7453 28-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 16:46:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_54_20231207T074259Z_gen_g4_detsim_hitreco__20240510T062339Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_733_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203317Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_692_20231208T101054Z_gen_g4_detsim_hitreco__20240510T065136Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_278_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195128Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 17:04:59
Saved logs	justin-logs:40120.99-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

y                          9.6461e-05     0.0137796     0.543367     0.00074341     0.052343        300    
reco:pandorapid:Chi2ParticleID                        3.5286e-05    0.00012806    0.00130077    0.000105958   0.000104071      300    
reco:cvnmap:CVNMapper                                 1.7723e-05    0.00756045     0.0327003    0.00827411    0.00515298       300    
reco:cvneva:CVNEvaluator                              1.5068e-05     0.285912       1.92727      0.365452       0.18192        300    
reco:energyrecnumu:EnergyReco                         0.000236773    0.065576       2.47991     0.00913632     0.177797        300    
reco:energyrecnue:EnergyReco                          0.000112911   0.00164728     0.0366175    0.000447631   0.00412831       300    
reco:energyrecnc:EnergyReco                           0.00160701     0.0973737     0.393011       0.11428      0.0647668       300    
reco:energyrecnumurange:EnergyReco                    0.000202518    0.0297183      2.10875     0.00183194     0.149915        300    
reco:energyrecnumumcs:EnergyReco                      0.000440062    0.029827       2.1086      0.00214761      0.14828        300    
reco:energyrecnumumcsllhd:EnergyReco                  0.000335517    0.0291262      2.11842     0.00209868     0.146475        300    
reco:opdec:Deconvolution                               0.0548126      0.20779      0.375343      0.209623      0.0614269       300    
reco:ophitspe:OpHitFinderDeco                          0.158787       0.16713      0.176399       0.16593     0.00438479       300    
reco:opflash:OpFlashFinder                            0.000111819    0.0010515    0.00327766    0.00100879    0.000484704      300    
reco:opslicer:OpSlicer                                3.4285e-05     0.205026      0.713419      0.168207      0.166178        300    
reco:rns:RandomNumberSaver                            1.7522e-05    2.93583e-05   0.000327963   2.5367e-05    2.27015e-05      300    
reco:anglereconue:NuAngularReco                       0.000130815    0.0010086     0.0272307    0.000326155   0.00274207       300    
reco:anglereconumu:NuAngularReco                      6.9009e-05    0.000862417    0.0265757    0.000177592   0.00267919       300    
reco:anglereconuepfps:NuAngularReco                   0.000104956   0.00095179     0.0267865    0.000271417   0.00267534       300    
reco:anglereconumupfps:NuAngularReco                  9.3604e-05    0.000911501    0.0267735    0.000234388   0.00268586       300    
reco:anglerecohits:NuAngularReco                      7.7916e-05    0.00139011     0.0272989    0.00054283    0.00305528       300    
[art]:TriggerResults:TriggerResultInserter             8.315e-06    1.11151e-05   4.4883e-05    1.0289e-05    3.56965e-06      300    
end_path:cafmaker:CAFMaker                            0.00102559     0.0131218     0.309058     0.00337241     0.0370592       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3649.03 MB
  Peak resident set size usage (VmHWM): 1961.19 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 19:04:41 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/18/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_278_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195128Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0325102, 0.00332735, 0.964162, 
Output 1: 0.0213844, 0.216118, 0.727636, 0.0348615, 
Output 2: 0.841262, 0.152426, 0.0062195, 9.27358e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1462!
Begin processing the 294th record. run: 6472938 subRun: 1 event: 69294 at 15-Sep-2025 19:04:22 CEST
Boundary wire vector sizes: 110, 84, 40
minwire 0: 1577
minwire 1: 874
minwire 2: 1845
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.990085, 0.00135101, 0.008564, 
Output 1: 0.864664, 0.1339, 0.00133121, 0.000105285, 
Output 2: 0.993427, 0.00645315, 0.000113688, 6.34371e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1161!
Begin processing the 295th record. run: 6472938 subRun: 1 event: 69295 at 15-Sep-2025 19:04:25 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 741!
Begin processing the 296th record. run: 6472938 subRun: 1 event: 69296 at 15-Sep-2025 19:04:27 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1114!
Begin processing the 297th record. run: 6472938 subRun: 1 event: 69297 at 15-Sep-2025 19:04:28 CEST
Boundary wire vector sizes: 235, 155, 169
minwire 0: 745
minwire 1: 1432
minwire 2: 875
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00266023, 0.935339, 0.0620008, 
Output 1: 0.965157, 0.0345308, 0.000255646, 5.65748e-05, 
Output 2: 0.99763, 0.00228969, 6.61786e-05, 1.44796e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1525!
Begin processing the 298th record. run: 6472938 subRun: 1 event: 69298 at 15-Sep-2025 19:04:31 CEST
Boundary wire vector sizes: 893, 1402, 950
minwire 0: 1612
minwire 1: 897
minwire 2: 1404
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991472, 0.00326134, 0.00526634, 
Output 1: 0.964445, 0.0352024, 0.000322889, 2.96482e-05, 
Output 2: 0.980199, 0.0194914, 0.00029069, 1.8661e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2971!
Begin processing the 299th record. run: 6472938 subRun: 1 event: 69299 at 15-Sep-2025 19:04:36 CEST
Boundary wire vector sizes: 75, 56, 67
minwire 0: 549
minwire 1: 1993
minwire 2: 398
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00944173, 0.00212622, 0.988432, 
Output 1: 0.144499, 0.815918, 0.0388461, 0.000737813, 
Output 2: 0.967391, 0.031675, 0.000914504, 1.92229e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1157!
Begin processing the 300th record. run: 6472938 subRun: 1 event: 69300 at 15-Sep-2025 19:04:38 CEST
Boundary wire vector sizes: 198, 232, 159
minwire 0: 1699
minwire 1: 313
minwire 2: 1959
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0268699, 0.00331464, 0.969815, 
Output 1: 0.00718666, 0.0232754, 0.0723647, 0.897173, 
Output 2: 0.884905, 0.108455, 0.00587702, 0.000762891, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1718!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/ed/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_54_20231207T074259Z_gen_g4_detsim_hitreco__20240510T062339Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a2/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_733_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203317Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/14/5e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_692_20231208T101054Z_gen_g4_detsim_hitreco__20240510T065136Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/18/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_278_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195128Z_reco2.root
.:
total 12460
-rw-r--r-- 1 dune001 lbno 4723673 Sep 15 19:04 flatcaf.root
-rw-r--r-- 1 dune001 lbno 3832814 Sep 15 19:04 caf.root
-rw-r--r-- 1 dune001 lbno 3832814 Sep 15 19:04 caf_fd_hd_atmo_2501_20250915T164649Z.root
-rw-r--r-- 1 dune001 lbno  199903 Sep 15 19:04 caf_20250915T164649Z.log
-rw-r--r-- 1 dune001 lbno   79148 Sep 15 19:04 jobscript.log
-rw-r--r-- 1 dune001 lbno     745 Sep 15 19:04 caf_20250915T164649Z.file
-rw-r--r-- 1 dune001 lbno     745 Sep 15 19:04 caf_20250915T164649Z.pfns
-rw-r--r-- 1 dune001 lbno     745 Sep 15 18:46 file.list
-rw-r--r-- 1 dune001 lbno     745 Sep 15 19:04 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno     549 Sep 15 18:46 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno     549 Sep 15 19:04 caf_20250915T164649Z.did
-rw-r--r-- 1 dune001 lbno     549 Sep 15 18:46 did.list
-rw-r--r-- 1 dune001 lbno       0 Sep 15 18:47 debugprod.log