justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258189.0@dunegpschedd02.fnal.gov

Jobsub ID258189.0@dunegpschedd02.fnal.gov
Workflow ID11026
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 21:41:10
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 09:30:18
From worker nodeHostnamedunegli-7832149-0-fnpc18001.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:49
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205851Z_gen_003573_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (27880s)
CPU time7h (27407s = 98%)
Max RSS bytes3067826176 (2925 MiB)
Outputting started2025-12-06 09:23:29
Output files
Finished2025-12-06 09:30:18
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

6403, 
Output 2: 0.0320328, 0.156536, 0.793444, 0.0179874, 
Output 3: 0.9719, 0.0139663, 0.00304981, 0.0110838, 
Output 4: 0.44852, 0.448179, 0.089504, 0.013797, 
Output 5: 0.477326, 0.372854, 0.120589, 0.0292321, 
Output 6: 0.033765, 0.0557262, 0.0651594, 0.845349, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 35730 at 06-Dec-2025 08:47:14 UTC
Boundary wire vector sizes: 6787, 6708, 6665
minwire 0: 25
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 12, 511
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.145903, 
Output 1: 0.0513922, 0.134482, 0.144428, 0.669697, 
Output 2: 0.0368397, 0.267616, 0.668958, 0.0265862, 
Output 3: 0.964617, 0.0237346, 0.00422328, 0.00742459, 
Output 4: 0.439647, 0.47296, 0.08029, 0.00710307, 
Output 5: 0.508693, 0.375038, 0.0995857, 0.016683, 
Output 6: 0.0107117, 0.0346274, 0.0563791, 0.898282, 

06-Dec-2025 09:22:25 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:22:25 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/28/c0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205851Z_gen_003573_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2082.61       2752.78       3884.52       2554.08       675.447        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00265257     0.0105952     0.0293377    0.00797451    0.00720725       10     
reco:pandora:StandardPandora                            2080.17       2748.99       3879.48       2550.95       674.387        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000531382    0.0388983     0.176456      0.0115893     0.0544473       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000452243   0.00867591     0.0556738    0.00249375     0.0159882       10     
reco:pandoracalo:Calorimetry                          0.000246775   0.00297805     0.0132392    0.000476138    0.0048048       10     
reco:pandorapid:Chi2ParticleID                        9.0992e-05    0.00068627     0.0053827    0.000153729   0.00156653       10     
reco:cvnmap:CVNMapper                                  0.0397383     0.0601851     0.105873      0.0510525     0.022763        10     
reco:cvneva:CVNEvaluator                                1.89525       2.98852       7.46184       2.37884       1.62005        10     
reco:energyrecnumu:EnergyReco                         0.00480921     0.0944437     0.204819      0.105828      0.0638988       10     
reco:energyrecnue:EnergyReco                          0.00165792    0.00600615     0.0153943    0.00384286    0.00467132       10     
reco:energyrecnc:EnergyReco                           0.00968104     0.0410048     0.124834      0.0164763     0.043392        10     
[art]:TriggerResults:TriggerResultInserter            2.0208e-05    6.81383e-05   0.000190169   3.2706e-05    6.20675e-05      10     
end_path:out1:RootOutput                               9.578e-06    1.88276e-05    6.87e-05     1.1757e-05    1.71184e-05      10     
end_path:out1:RootOutput(write)                         0.36696      0.530278      0.837906      0.425993      0.177709        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21778 MB
  Peak resident set size usage (VmHWM): 3067.83 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 65.0031 | hit correction : 1.34464
hit initial : 49.1862 | hit correction : 1.3434
hit initial : 229.673 | hit correction : 1.34465
hit initial : 310.165 | hit correction : 1.34454
hit initial : 283.135 | hit correction : 1.34392
hit initial : 201.391 | hit correction : 1.34353
hit initial : 142.278 | hit correction : 1.34337
 *** Total Charge : 1721.56
 Looking at view: 0 | Number of hits in view: 7
hit initial : 184.816 | hit correction : 1.34462
hit initial : 316.082 | hit correction : 1.3446
hit initial : 113.02 | hit correction : 1.34463
hit initial : 191.433 | hit correction : 1.34383
hit initial : 199.38 | hit correction : 1.34357
hit initial : 158.481 | hit correction : 1.34339
hit initial : 66.9935 | hit correction : 1.34334
 *** Total Charge : 1653.51
 Looking at view: 1 | Number of hits in view: 7
hit initial : 97.6521 | hit correction : 1.34473
hit initial : 446.157 | hit correction : 1.34459
hit initial : 144.787 | hit correction : 1.3445
hit initial : 138.831 | hit correction : 1.34364
hit initial : 442.648 | hit correction : 1.34353
hit initial : 66.1427 | hit correction : 1.34347
 *** Total Charge : 1795.99
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 10th record. run: 5026 subRun: 0 event: 35730 at 06-Dec-2025 09:23:25 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  42
Added hit list length: 42 | Max was set to: -3.40282e+38
True Neutron Energy: 939.57
True Electron Energy: 51.782
Completed MCParticle Loop
True Neutrino Energy: 55.5916
 Size of all hits: 42
hit initial : 31.7367 | hit correction : 1.39716
hit initial : 64.4837 | hit correction : 1.39326
hit initial : 37.8412 | hit correction : 1.3846
hit initial : 94.7749 | hit correction : 1.39146
hit initial : 323.175 | hit correction : 1.39141
hit initial : 394.964 | hit correction : 1.39126
hit initial : 340.199 | hit correction : 1.39096
hit initial : 253.325 | hit correction : 1.39063
hit initial : 201.053 | hit correction : 1.39047
hit initial : 123.983 | hit correction : 1.39047
hit initial : 34.4014 | hit correction : 1.39047
hit initial : 35.7138 | hit correction : 1.3891
hit initial : 80.2761 | hit correction : 1.39
hit initial : 166.728 | hit correction : 1.38919
hit initial : 227.663 | hit correction : 1.39012
hit initial : 143.744 | hit correction : 1.38922
hit initial : 231.645 | hit correction : 1.39032
hit initial : 270.874 | hit correction : 1.39056
hit initial : 324.129 | hit correction : 1.39082
hit initial : 286.768 | hit correction : 1.39129
hit initial : 300.355 | hit correction : 1.39183
hit initial : 257.792 | hit correction : 1.39212
hit initial : 244.166 | hit correction : 1.3924
hit initial : 244.341 | hit correction : 1.39267
hit initial : 192.324 | hit correction : 1.39294
hit initial : 129.107 | hit correction : 1.39317
 *** Total Charge : 7005.33
 Looking at view: 0 | Number of hits in view: 26
hit initial : 452.868 | hit correction : 1.3912
hit initial : 721.951 | hit correction : 1.39077
hit initial : 240.772 | hit correction : 1.3915
hit initial : 82.9584 | hit correction : 1.39164
hit initial : 707.129 | hit correction : 1.39058
hit initial : 259.508 | hit correction : 1.39039
hit initial : 145.424 | hit correction : 1.39013
hit initial : 25.4384 | hit correction : 1.3891
hit initial : 80.9893 | hit correction : 1.38911
hit initial : 60.4496 | hit correction : 1.3891
hit initial : 69.7831 | hit correction : 1.38909
hit initial : 42.492 | hit correction : 1.392
hit initial : 38.3634 | hit correction : 1.38914
hit initial : 313.505 | hit correction : 1.39247
hit initial : 628.103 | hit correction : 1.39239
hit initial : 136.742 | hit correction : 1.39248
 *** Total Charge : 5573.57
 Looking at view: 1 | Number of hits in view: 16
06-Dec-2025 09:23:29 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0904078     0.119049      0.190494      0.113021      0.0285021       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000511235   0.000684663   0.00129598    0.00061775    0.000215495      10     
end_path:ana:AnalyzeEvents         0.089645      0.118073      0.189653      0.112147      0.0283131       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.9 MB
  Peak resident set size usage (VmHWM): 1540.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:09:28 UTC       justIN version: 01.05.03