justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 259479.50@dunegpschedd02.fnal.gov

Jobsub ID259479.50@dunegpschedd02.fnal.gov
Workflow ID11111
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:44
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:19:06
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:27:30
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T192524Z_gen_002254_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (60293s)
CPU time16h (58865s = 97%)
Max RSS bytes2650038272 (2527 MiB)
Outputting started2025-12-10 09:12:23
Output files
Finished2025-12-10 09:19:06
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.094144, 
Output 1: 0.0736351, 0.0538196, 0.166778, 0.705767, 
Output 2: 0.0510129, 0.106075, 0.823871, 0.0190411, 
Output 3: 0.9722, 0.018556, 0.0024464, 0.00679782, 
Output 4: 0.525081, 0.423045, 0.0440484, 0.00782545, 
Output 5: 0.421982, 0.407242, 0.129244, 0.0415321, 
Output 6: 0.0381252, 0.0870233, 0.0788975, 0.795954, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 22540 at 10-Dec-2025 08:49:37 UTC
Boundary wire vector sizes: 6805, 6698, 6516
minwire 0: 33
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.121955, 
Output 1: 0.122688, 0.0709073, 0.261429, 0.544975, 
Output 2: 0.0686502, 0.0883265, 0.822137, 0.0208864, 
Output 3: 0.976601, 0.0185774, 0.00164556, 0.00317562, 
Output 4: 0.538002, 0.41917, 0.0359309, 0.00689659, 
Output 5: 0.354502, 0.436718, 0.150096, 0.0586836, 
Output 6: 0.0916548, 0.178632, 0.109772, 0.619941, 

10-Dec-2025 09:11:54 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:11:54 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/3a/86/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T192524Z_gen_002254_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1334.62       5975.65       8728.97       6577.56       2100.76        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00627246     0.0222813     0.0368228     0.0225808    0.00880125       10     
reco:pandora:StandardPandora                            1333.57       5916.1        8640.75       6506.49       2074.79        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00280004     0.103958      0.596234      0.0530702     0.168217        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00131218     0.018956      0.0705949    0.00830958     0.0217479       10     
reco:pandoracalo:Calorimetry                          0.000504418    0.0208834     0.0997462    0.00605283     0.0315358       10     
reco:pandorapid:Chi2ParticleID                        7.3018e-05    0.00154373    0.00788002    0.000490863   0.00247285       10     
reco:cvnmap:CVNMapper                                  0.0397325      0.26771      0.535327      0.232887      0.157656        10     
reco:cvneva:CVNEvaluator                               0.695159       2.83467       11.8407       2.06676       3.06538        10     
reco:energyrecnumu:EnergyReco                          0.0101038      13.8528       73.8544      0.0345227      27.7166        10     
reco:energyrecnue:EnergyReco                          0.00349384    0.00971905     0.026721     0.00635974    0.00820157       10     
reco:energyrecnc:EnergyReco                            0.0044893      41.5595       74.4273       65.4338       34.0061        10     
[art]:TriggerResults:TriggerResultInserter            1.5088e-05    0.000211094   0.000589139   0.000199159   0.000158337      10     
end_path:out1:RootOutput                               6.402e-06    6.74061e-05   0.000249834   5.2335e-05    6.67563e-05      10     
end_path:out1:RootOutput(write)                         0.22103      0.850084       1.66974      0.849999      0.450132        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21679.9 MB
  Peak resident set size usage (VmHWM): 2650.04 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 52.3373 | hit correction : 1.37332
hit initial : 48.6377 | hit correction : 1.37329
hit initial : 233.667 | hit correction : 1.37277
hit initial : 41.5812 | hit correction : 1.37281
hit initial : 277.262 | hit correction : 1.3715
hit initial : 192.13 | hit correction : 1.37206
hit initial : 80.7521 | hit correction : 1.37237
 *** Total Charge : 1442.49
 Looking at view: 1 | Number of hits in view: 8
hit initial : 41.9627 | hit correction : 1.37331
hit initial : 235.151 | hit correction : 1.37327
hit initial : 233.345 | hit correction : 1.37287
hit initial : 123.071 | hit correction : 1.37108
hit initial : 300.016 | hit correction : 1.37148
hit initial : 309.7 | hit correction : 1.37211
hit initial : 264.044 | hit correction : 1.37242
 *** Total Charge : 2068.43
 Looking at view: 2 | Number of hits in view: 7
Begin processing the 8th record. run: 5026 subRun: 0 event: 22538 at 10-Dec-2025 09:12:17 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  33
Added hit list length: 33 | Max was set to: -3.40282e+38
This pfp has size:  11
True Electron Energy: 25.7252
Completed MCParticle Loop
True Neutrino Energy: 29.4685
 Size of all hits: 33
hit initial : 27.0122 | hit correction : 1.39118
hit initial : 40.9857 | hit correction : 1.39115
hit initial : 71.5332 | hit correction : 1.38893
hit initial : 104.578 | hit correction : 1.38892
hit initial : 42.7393 | hit correction : 1.38891
hit initial : 212.83 | hit correction : 1.39159
hit initial : 272.018 | hit correction : 1.39115
hit initial : 41.3519 | hit correction : 1.39005
hit initial : 319.716 | hit correction : 1.39038
hit initial : 265.761 | hit correction : 1.39019
hit initial : 110.416 | hit correction : 1.39015
 *** Total Charge : 2098.11
 Looking at view: 0 | Number of hits in view: 11
hit initial : 38.3433 | hit correction : 1.38895
hit initial : 108.33 | hit correction : 1.38894
hit initial : 89.7273 | hit correction : 1.38893
hit initial : 49.8921 | hit correction : 1.39012
hit initial : 136.211 | hit correction : 1.39013
hit initial : 252.226 | hit correction : 1.39273
hit initial : 153.03 | hit correction : 1.3901
hit initial : 403.62 | hit correction : 1.39221
hit initial : 349.845 | hit correction : 1.39394
hit initial : 138.291 | hit correction : 1.39401
hit initial : 22.9512 | hit correction : 1.39508
hit initial : 68.0772 | hit correction : 1.3951
hit initial : 38.1224 | hit correction : 1.39512
hit initial : 118.036 | hit correction : 1.39105
hit initial : 443.828 | hit correction : 1.39108
hit initial : 256.029 | hit correction : 1.39098
 *** Total Charge : 3711.33
 Looking at view: 1 | Number of hits in view: 16
hit initial : 227.94 | hit correction : 1.39154
hit initial : 196.486 | hit correction : 1.39116
hit initial : 144.513 | hit correction : 1.3908
hit initial : 314.455 | hit correction : 1.39023
hit initial : 269.323 | hit correction : 1.39013
hit initial : 18.4796 | hit correction : 1.39019
 *** Total Charge : 1628.77
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 9th record. run: 5026 subRun: 0 event: 22539 at 10-Dec-2025 09:12:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 3.56557
Completed MCParticle Loop
True Neutrino Energy: 8.94355
 Size of all hits: 0
Begin processing the 10th record. run: 5026 subRun: 0 event: 22540 at 10-Dec-2025 09:12:21 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.57
True Electron Energy: 30.3235
Completed MCParticle Loop
True Neutrino Energy: 34.0964
 Size of all hits: 0
10-Dec-2025 09:12:23 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0464405     0.0637756     0.0832565     0.0675546     0.0126762       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000383028   0.00212427     0.0165652    0.000461678   0.00481759       10     
end_path:ana:AnalyzeEvents         0.0458131     0.0614626     0.0825471     0.0650224     0.011216        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19473.4 MB
  Peak resident set size usage (VmHWM): 1525.01 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:18:23 UTC       justIN version: 01.05.03