Jobsub ID 45078.1@dunegpschedd02.fnal.gov

Jobsub ID	45078.1@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 16:56:26
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce04
Last heartbeat	2025-09-21 17:43:12
From worker node	Hostname	wn119
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 16:58:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_1173_20231204T074408Z_gen_g4_detsim_hitreco__20240509T203812Z_reco2.root
Jobscript	Exit code	0
	Real time	38m (2294s)
	CPU time	10m (636s = 27%)
	Max RSS bytes	2100011008 (2002 MiB)
Outputting started	2025-09-21 17:36:59
Output files
Finished	2025-09-21 17:43:12
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

====================================================================================================
Full event                                              4.42134       12.9326       200.067       8.03767       22.6431        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.197116      0.305661      0.441129      0.238324      0.0991549       100    
reco:pandora:StandardPandora                            1.20267       6.52172       162.014       2.76928       18.4051        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000123304    0.0104189     0.169177      0.0021537     0.0245557       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000167995    0.021148      0.395749     0.00526212     0.0490524       100    
reco:pandoracalo:Calorimetry                          0.000108763    0.0395962      1.24247     0.00166259     0.152853        100    
reco:pandorapid:Chi2ParticleID                        3.9441e-05    0.00023474    0.00109556    0.000174386   0.000189163      100    
reco:cvnmap:CVNMapper                                  2.171e-05     0.0102387     0.0370846     0.0109341    0.00664478       100    
reco:cvneva:CVNEvaluator                               1.839e-05     0.573937       13.5258      0.563625       1.32207        100    
reco:energyrecnumu:EnergyReco                         0.000285649    0.394968       7.72533      0.010317      0.922428        100    
reco:energyrecnue:EnergyReco                          0.000198036   0.00379593     0.0760395    0.000941111   0.00965161       100    
reco:energyrecnc:EnergyReco                           0.00357908      1.0262        2.12218       1.19693       0.57017        100    
reco:energyrecnumurange:EnergyReco                    0.000419624    0.142992       6.91257     0.00397453     0.758047        100    
reco:energyrecnumumcs:EnergyReco                      0.000711113    0.143729       6.93468     0.00467361     0.757716        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000593189    0.143964       6.93843     0.00460141     0.759353        100    
reco:opdec:Deconvolution                               0.609403       1.02058       2.17976      0.991491      0.255257        100    
reco:ophitspe:OpHitFinderDeco                           1.76994       1.78175       1.81398       1.78189     0.00571148       100    
reco:opflash:OpFlashFinder                            0.000295529   0.00185362    0.00508139    0.00178835    0.000911717      100    
reco:opslicer:OpSlicer                                0.000384592    0.408146       1.44962        0.316       0.328944        100    
reco:rns:RandomNumberSaver                             2.762e-05    4.66481e-05   0.000611719   3.86665e-05   5.78674e-05      100    
reco:anglereconue:NuAngularReco                       0.000214047   0.00233951     0.0527785    0.000570574   0.00651647       100    
reco:anglereconumu:NuAngularReco                      0.000126014   0.00213929     0.0509228    0.000394258   0.00636507       100    
reco:anglereconuepfps:NuAngularReco                   0.000181976   0.00232379     0.0529787    0.000519257   0.00649807       100    
reco:anglereconumupfps:NuAngularReco                  0.000174036   0.00227246     0.0517533    0.00046008    0.00649349       100    
reco:anglerecohits:NuAngularReco                      0.000157355   0.00328761     0.0590617    0.00106349    0.00748827       100    
[art]:TriggerResults:TriggerResultInserter             1.111e-05    1.8765e-05    6.1012e-05    1.75455e-05   6.8235e-06       100    
end_path:cafmaker:CAFMaker                            0.00165107     0.367536       33.7465     0.00603061      3.35578        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3810.54 MB
  Peak resident set size usage (VmHWM): 2100.01 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.475277, 0.00343448, 0.521288, 
Output 1: 0.755459, 0.231438, 0.0124355, 0.000667105, 
Output 2: 0.384341, 0.59092, 0.0240605, 0.000678506, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1934!
Begin processing the 94th record. run: 74517585 subRun: 1 event: 117394 at 21-Sep-2025 14:35:36 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 736!
Begin processing the 95th record. run: 74517585 subRun: 1 event: 117395 at 21-Sep-2025 14:35:45 -03
Boundary wire vector sizes: 156, 135, 109
minwire 0: 2287
minwire 1: 503
minwire 2: 2416
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000296808, 0.998734, 0.000969511, 
Output 1: 0.00814417, 0.987694, 0.00403299, 0.000128977, 
Output 2: 0.998496, 0.00146088, 4.17907e-05, 1.25476e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1180!
Begin processing the 96th record. run: 74517585 subRun: 1 event: 117396 at 21-Sep-2025 14:35:55 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 95!
Begin processing the 97th record. run: 74517585 subRun: 1 event: 117397 at 21-Sep-2025 14:36:04 -03
Boundary wire vector sizes: 614, 579, 435
minwire 0: 629
minwire 1: 2605
minwire 2: 7
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000522243, 0.995737, 0.00374102, 
Output 1: 0.996469, 0.0032604, 0.000150238, 0.000119975, 
Output 2: 0.996331, 0.00353717, 9.68095e-05, 3.55406e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1796!
Begin processing the 98th record. run: 74517585 subRun: 1 event: 117398 at 21-Sep-2025 14:36:16 -03
Boundary wire vector sizes: 2399, 2723, 1469
minwire 0: 968
minwire 1: 1004
minwire 2: 779
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00287706, 0.992972, 0.00415076, 
Output 1: 0.274386, 0.708975, 0.0152517, 0.00138748, 
Output 2: 0.213941, 0.648618, 0.128612, 0.00882965, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2862!
Begin processing the 99th record. run: 74517585 subRun: 1 event: 117399 at 21-Sep-2025 14:36:35 -03
Boundary wire vector sizes: 494, 556, 477
minwire 0: 2453
minwire 1: 785
minwire 2: 2133
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992252, 0.000401377, 0.00734708, 
Output 1: 0.0713061, 0.781162, 0.143844, 0.0036879, 
Output 2: 0.963192, 0.0356562, 0.00112765, 2.37869e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1944!
Begin processing the 100th record. run: 74517585 subRun: 1 event: 117400 at 21-Sep-2025 14:36:46 -03
Boundary wire vector sizes: 43, 51, 30
minwire 0: 2437
minwire 1: 599
minwire 2: 2440
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.668906, 0.145802, 0.185292, 
Output 1: 0.756604, 0.235207, 0.00785297, 0.000336019, 
Output 2: 0.779079, 0.216895, 0.00393271, 9.26431e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1010!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f0/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_1173_20231204T074408Z_gen_g4_detsim_hitreco__20240509T203812Z_reco2.root
.:
total 5640
-rw-r--r-- 1 nobody nobody 2205123 Sep 21 14:36 flatcaf.root
-rw-r--r-- 1 nobody nobody 1711808 Sep 21 14:36 caf.root
-rw-r--r-- 1 nobody nobody 1711808 Sep 21 14:36 caf_fd_hd_atmo_2501_20250921T165858Z.root
-rw-r--r-- 1 nobody nobody   78942 Sep 21 14:36 caf_20250921T165858Z.log
-rw-r--r-- 1 nobody nobody   32670 Sep 21 14:36 jobscript.log
-rw-r--r-- 1 nobody nobody     188 Sep 21 14:36 caf_20250921T165858Z.file
-rw-r--r-- 1 nobody nobody     188 Sep 21 14:36 caf_20250921T165858Z.pfns
-rw-r--r-- 1 nobody nobody     188 Sep 21 13:58 file.list
-rw-r--r-- 1 nobody nobody     188 Sep 21 14:36 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     139 Sep 21 13:58 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     139 Sep 21 14:36 caf_20250921T165858Z.did
-rw-r--r-- 1 nobody nobody     139 Sep 21 13:58 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 21 13:59 debugprod.log