justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257901.58@dunegpschedd02.fnal.gov

Jobsub ID257901.58@dunegpschedd02.fnal.gov
Workflow ID11024
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 10:59:38
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:00:08
From worker nodeHostnamedunegli-7988749-0-fnpc18001.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-05 20:22:43
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T225339Z_gen_000842_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time13h (48629s)
CPU time13h (47809s = 98%)
Max RSS bytes3037704192 (2896 MiB)
Outputting started2025-12-06 09:53:13
Output files
Finished2025-12-06 10:00:08
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1423, 0.0109395, 0.0018106, 0.00582695, 
Output 4: 0.576795, 0.380999, 0.0379609, 0.0042441, 
Output 5: 0.439113, 0.312691, 0.180889, 0.0673071, 
Output 6: 0.0171439, 0.0518631, 0.0676923, 0.863301, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 8420 at 06-Dec-2025 08:44:21 UTC
Boundary wire vector sizes: 6874, 6857, 6757
minwire 0: 25
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1118, 1617
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0861968, 
Output 1: 0.0442829, 0.155922, 0.206075, 0.59372, 
Output 2: 0.0494574, 0.157983, 0.770781, 0.0217792, 
Output 3: 0.926507, 0.0591329, 0.0074141, 0.0069465, 
Output 4: 0.486818, 0.448679, 0.0569694, 0.00753353, 
Output 5: 0.509315, 0.373261, 0.0941348, 0.0232892, 
Output 6: 0.0313987, 0.0988693, 0.0926869, 0.777045, 

06-Dec-2025 09:51:58 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:51:58 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/06/d2/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T225339Z_gen_000842_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3457.15       4823.65       5687.78       5020.47       713.525        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00561331     0.0152752     0.029345      0.0131132    0.00724746       10     
reco:pandora:StandardPandora                            3454.5        4819.16       5684.02       5016.28       712.248        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00188093     0.0589578     0.328794      0.0262116     0.0947283       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000599189    0.0126329     0.0603486    0.00544566     0.0169526       10     
reco:pandoracalo:Calorimetry                          0.00113338     0.0161345     0.0626373    0.00665922     0.0211682       10     
reco:pandorapid:Chi2ParticleID                        0.00010647    0.00139031    0.00609333    0.00038481    0.00216967       10     
reco:cvnmap:CVNMapper                                  0.0669732     0.115555       0.17182      0.113026      0.0291686       10     
reco:cvneva:CVNEvaluator                                1.97112       3.51236       10.7825       2.67821       2.50943        10     
reco:energyrecnumu:EnergyReco                          0.0203697     0.118104      0.291367      0.107118      0.0714491       10     
reco:energyrecnue:EnergyReco                          0.00289209    0.00998277     0.0241837     0.0093894    0.00666313       10     
reco:energyrecnc:EnergyReco                            0.0104208     0.0457171     0.205495      0.0143151     0.0649541       10     
[art]:TriggerResults:TriggerResultInserter            2.5478e-05    6.35588e-05   0.000264458   4.16685e-05   6.78821e-05      10     
end_path:out1:RootOutput                              1.2043e-05    2.31497e-05   9.3486e-05    1.5665e-05    2.36122e-05      10     
end_path:out1:RootOutput(write)                        0.389777      0.580704      0.889121      0.568977      0.163447        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21755.6 MB
  Peak resident set size usage (VmHWM): 3037.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 50.6706 | hit correction : 1.12996
hit initial : 70.3661 | hit correction : 1.12994
hit initial : 140.243 | hit correction : 1.14559
hit initial : 178.46 | hit correction : 1.14557
hit initial : 135.211 | hit correction : 1.14552
hit initial : 187.102 | hit correction : 1.14447
hit initial : 118.959 | hit correction : 1.14447
hit initial : 121.767 | hit correction : 1.14425
hit initial : 349.478 | hit correction : 1.13182
hit initial : 257.518 | hit correction : 1.13183
hit initial : 94.3021 | hit correction : 1.13164
hit initial : 94.1807 | hit correction : 1.13147
hit initial : 45.2329 | hit correction : 1.13148
hit initial : 539.398 | hit correction : 1.13098
hit initial : 188.893 | hit correction : 1.13042
hit initial : 112.39 | hit correction : 1.13037
hit initial : 238.26 | hit correction : 1.12948
hit initial : 560.044 | hit correction : 1.12933
hit initial : 515.563 | hit correction : 1.12874
hit initial : 243.935 | hit correction : 1.12854
hit initial : 85.0415 | hit correction : 1.12857
 *** Total Charge : 4903.25
 Looking at view: 1 | Number of hits in view: 21
hit initial : 46.0935 | hit correction : 1.12995
hit initial : 55.5186 | hit correction : 1.12991
hit initial : 22.9564 | hit correction : 1.12963
hit initial : 177.993 | hit correction : 1.14562
hit initial : 345.192 | hit correction : 1.14554
hit initial : 329.24 | hit correction : 1.1449
hit initial : 190.722 | hit correction : 1.1443
hit initial : 118.927 | hit correction : 1.14403
hit initial : 86.5669 | hit correction : 1.13659
hit initial : 36.239 | hit correction : 1.13653
hit initial : 35.6707 | hit correction : 1.13169
hit initial : 296.346 | hit correction : 1.13189
hit initial : 16.8853 | hit correction : 1.13144
hit initial : 215.031 | hit correction : 1.13194
hit initial : 129.606 | hit correction : 1.13144
hit initial : 30.5549 | hit correction : 1.13144
hit initial : 149.231 | hit correction : 1.13178
hit initial : 160.509 | hit correction : 1.13156
hit initial : 163.852 | hit correction : 1.13131
hit initial : 155.473 | hit correction : 1.13108
hit initial : 196.596 | hit correction : 1.13076
hit initial : 146.915 | hit correction : 1.13059
hit initial : 226.057 | hit correction : 1.1304
hit initial : 163.714 | hit correction : 1.13017
hit initial : 163.653 | hit correction : 1.12988
hit initial : 222.133 | hit correction : 1.12959
hit initial : 199.787 | hit correction : 1.12943
hit initial : 239.054 | hit correction : 1.12922
hit initial : 138.175 | hit correction : 1.12907
hit initial : 134.21 | hit correction : 1.12893
hit initial : 138.484 | hit correction : 1.12877
hit initial : 202.754 | hit correction : 1.12863
hit initial : 158.672 | hit correction : 1.12848
hit initial : 150.603 | hit correction : 1.12829
hit initial : 27.4647 | hit correction : 1.12816
 *** Total Charge : 5975.2
 Looking at view: 2 | Number of hits in view: 35
Begin processing the 8th record. run: 5027 subRun: 0 event: 8418 at 06-Dec-2025 09:52:51 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 20.9408
Completed MCParticle Loop
True Neutrino Energy: 36.4248
 Size of all hits: 0
Begin processing the 9th record. run: 5027 subRun: 0 event: 8419 at 06-Dec-2025 09:52:58 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 9.91314
Completed MCParticle Loop
True Neutrino Energy: 18.3845
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 8420 at 06-Dec-2025 09:53:05 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.566
True Electron Energy: 15.0358
Completed MCParticle Loop
True Neutrino Energy: 20.5493
 Size of all hits: 0
06-Dec-2025 09:53:11 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0953458     0.120269      0.143709       0.1225       0.0121749       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000654675   0.000872079   0.00165241    0.000780687   0.000278316      10     
end_path:ana:AnalyzeEvents         0.0944226     0.119075      0.142763      0.121455      0.0120959       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.4 MB
  Peak resident set size usage (VmHWM): 1541.4 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:09:43 UTC       justIN version: 01.06.00