Jobsub ID 259479.135@dunegpschedd02.fnal.gov

Jobsub ID	259479.135@dunegpschedd02.fnal.gov
Workflow ID	11111
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-12-09 10:51:44
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-12-10 09:11:31
From worker node	Hostname	dunegli-48793-0-cmswn4069.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-12-09 16:52:52
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021820Z_gen_003893_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	16h (58314s)
	CPU time	15h (57446s = 98%)
	Max RSS bytes	2504286208 (2388 MiB)
Outputting started	2025-12-10 09:04:47
Output files
Finished	2025-12-10 09:11:31
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0.0538127, 0.160606, 0.133198, 0.652384, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 38930 at 10-Dec-2025 08:30:02 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  10-Dec-2025 09:04:13 UTC run: 5027 subRun: 0 event: 38930
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6744, 6610, 6658
minwire 0: 39
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1079, 1578
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.122906, 
Output 1: 0.0425573, 0.0796211, 0.174665, 0.703157, 
Output 2: 0.0812703, 0.217852, 0.654773, 0.0461053, 
Output 3: 0.969215, 0.0205313, 0.00299071, 0.00726282, 
Output 4: 0.188375, 0.766775, 0.0430698, 0.0017801, 
Output 5: 0.656418, 0.264499, 0.059437, 0.0196463, 
Output 6: 0.0214218, 0.0613762, 0.0708891, 0.846313, 

10-Dec-2025 09:04:15 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:04:15 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7c/fb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021820Z_gen_003893_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2049.27       5767.05       8226.67       6147.24       1663.08        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00794752     0.0222588     0.0432045     0.0236994    0.00919564       10     
reco:pandora:StandardPandora                            2048.22       5755.76       8138.11       6144.53       1649.82        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000666045    0.0890775     0.634471      0.0133637     0.186057        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000495331    0.0156883     0.0619966    0.00705815     0.0220963       10     
reco:pandoracalo:Calorimetry                          0.000342451    0.0489029     0.159099      0.0150919     0.0622788       10     
reco:pandorapid:Chi2ParticleID                        8.8718e-05    0.00143256    0.00877886    0.000251879   0.00262839       10     
reco:cvnmap:CVNMapper                                  0.0273655     0.296366       1.6832       0.131181      0.466958        10     
reco:cvneva:CVNEvaluator                               0.732629       9.96674       84.7218       1.73619       24.9257        10     
reco:energyrecnumu:EnergyReco                         0.00205186     0.143302      0.478517      0.093553      0.143456        10     
reco:energyrecnue:EnergyReco                          0.000821882    0.0127695     0.0274667    0.00906331    0.00944866       10     
reco:energyrecnc:EnergyReco                           0.00826346     0.0676694     0.209606      0.0343602     0.0709968       10     
[art]:TriggerResults:TriggerResultInserter            1.9727e-05    0.000190308   0.000972118   0.00011081    0.00026807       10     
end_path:out1:RootOutput                              1.2944e-05    8.22101e-05   0.000401873   4.7215e-05    0.000110516      10     
end_path:out1:RootOutput(write)                        0.205105      0.621034      0.949778      0.671138      0.246738        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21691 MB
  Peak resident set size usage (VmHWM): 2504.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 128.051 | hit correction : 1.21007
hit initial : 192.485 | hit correction : 1.21013
hit initial : 113.589 | hit correction : 1.21017
hit initial : 113.411 | hit correction : 1.21011
hit initial : 34.6654 | hit correction : 1.20879
hit initial : 278.239 | hit correction : 1.20884
hit initial : 115.712 | hit correction : 1.20852
hit initial : 488.467 | hit correction : 1.20885
hit initial : 240.595 | hit correction : 1.20943
hit initial : 469.064 | hit correction : 1.2096
hit initial : 557.617 | hit correction : 1.20967
 *** Total Charge : 3304.06
 Looking at view: 2 | Number of hits in view: 11
Begin processing the 7th record. run: 5027 subRun: 0 event: 38927 at 10-Dec-2025 09:04:39 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 4.07828
Completed MCParticle Loop
True Neutrino Energy: 8.9126
 Size of all hits: 0
Begin processing the 8th record. run: 5027 subRun: 0 event: 38928 at 10-Dec-2025 09:04:41 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 60.4136
Completed MCParticle Loop
True Neutrino Energy: 64.3426
 Size of all hits: 0
Begin processing the 9th record. run: 5027 subRun: 0 event: 38929 at 10-Dec-2025 09:04:43 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 60.3362
Completed MCParticle Loop
True Neutrino Energy: 67.5851
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 38930 at 10-Dec-2025 09:04:44 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  31
Added hit list length: 31 | Max was set to: -3.40282e+38
This pfp has size:  17
True Neutron Energy: 939.569
True Electron Energy: 47.2066
Completed MCParticle Loop
True Neutrino Energy: 62.7791
 Size of all hits: 31
hit initial : 45.1502 | hit correction : 1.09113
hit initial : 32.7908 | hit correction : 1.09116
hit initial : 61.4725 | hit correction : 1.08928
hit initial : 306.494 | hit correction : 1.08938
hit initial : 205.799 | hit correction : 1.0897
hit initial : 157.146 | hit correction : 1.08917
hit initial : 189.309 | hit correction : 1.09183
hit initial : 1003.34 | hit correction : 1.09186
hit initial : 888.172 | hit correction : 1.09181
hit initial : 410.696 | hit correction : 1.09169
hit initial : 320.182 | hit correction : 1.09167
hit initial : 156.162 | hit correction : 1.0917
 *** Total Charge : 4121.59
 Looking at view: 0 | Number of hits in view: 12
hit initial : 107.575 | hit correction : 1.09205
hit initial : 540.715 | hit correction : 1.09193
hit initial : 458.629 | hit correction : 1.09169
hit initial : 215.309 | hit correction : 1.0917
hit initial : 45.1492 | hit correction : 1.09175
hit initial : 69.9204 | hit correction : 1.08971
hit initial : 261.965 | hit correction : 1.08959
hit initial : 193.564 | hit correction : 1.08962
hit initial : 174.66 | hit correction : 1.08916
hit initial : 123.631 | hit correction : 1.0892
hit initial : 1046.27 | hit correction : 1.09185
hit initial : 903.386 | hit correction : 1.09176
 *** Total Charge : 4519.01
 Looking at view: 1 | Number of hits in view: 12
hit initial : 483.084 | hit correction : 1.09163
hit initial : 1999.47 | hit correction : 1.09202
hit initial : 46.5372 | hit correction : 1.09189
hit initial : 109.355 | hit correction : 1.08963
hit initial : 497.29 | hit correction : 1.08953
hit initial : 194.336 | hit correction : 1.08913
hit initial : 514.799 | hit correction : 1.09184
 *** Total Charge : 4196.33
 Looking at view: 2 | Number of hits in view: 7
10-Dec-2025 09:04:46 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0483429     0.0651285     0.0907901     0.0643735     0.0113621       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000398858   0.000616624   0.00130462    0.000506788   0.000274902      10     
end_path:ana:AnalyzeEvents         0.0477027     0.0643156     0.0900879     0.0635134     0.0112654       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19475.6 MB
  Peak resident set size usage (VmHWM): 1523.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0