justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265279.46@dunegpschedd01.fnal.gov

Jobsub ID265279.46@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:45
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:15:58
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:38:40
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T025606Z_gen_004409_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (58867s)
CPU time16h (58061s = 98%)
Max RSS bytes2622124032 (2500 MiB)
Outputting started2025-12-10 08:59:48
Output files
Finished2025-12-10 09:15:58
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

, 
Output 4: 0.370104, 0.550051, 0.0724928, 0.00735205, 
Output 5: 0.392113, 0.426468, 0.145267, 0.0361511, 
Output 6: 0.0442994, 0.112161, 0.105639, 0.7379, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 44090 at 10-Dec-2025 08:16:14 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
Boundary wire vector sizes: 6544, 6623, 6620
minwire 0: 22
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 718, 1217
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.129523, 
Output 1: 0.0666939, 0.0327402, 0.0980092, 0.802557, 
Output 2: 0.319854, 0.288694, 0.32102, 0.0704317, 
Output 3: 0.981398, 0.0127611, 0.000910341, 0.00493044, 
Output 4: 0.238788, 0.735057, 0.0250183, 0.00113666, 
Output 5: 0.820682, 0.172088, 0.00574835, 0.00148152, 
Output 6: 0.0154191, 0.0449902, 0.0604257, 0.879165, 

10-Dec-2025 08:59:13 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 08:59:13 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/70/93/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T025606Z_gen_004409_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2574.74       5836.94       8293.81       6243.58       1591.61        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0143328     0.0240735     0.0353998     0.0269144    0.00685282       10     
reco:pandora:StandardPandora                            2573.49       5832.5        8274.66       6239.81       1588.51        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0024444     0.150831      0.737238      0.106435      0.209715        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00141121     0.0238209     0.183797     0.00614359     0.053479        10     
reco:pandoracalo:Calorimetry                          0.000258291    0.0251613     0.188861     0.00122099     0.0557234       10     
reco:pandorapid:Chi2ParticleID                        0.000116822   0.00251602     0.0189667    0.000400496   0.00555088       10     
reco:cvnmap:CVNMapper                                  0.0652717     0.175865      0.403427      0.159626      0.0933422       10     
reco:cvneva:CVNEvaluator                               0.847515       3.18586       16.0846       2.19035       4.34248        10     
reco:energyrecnumu:EnergyReco                         0.00373356     0.0978852     0.213379      0.0605878     0.0876184       10     
reco:energyrecnue:EnergyReco                          0.000933123   0.00696101     0.0187144    0.00612018    0.00481267       10     
reco:energyrecnc:EnergyReco                           0.00598604     0.0976126     0.298887      0.0671957     0.0896637       10     
[art]:TriggerResults:TriggerResultInserter            1.7123e-05    0.00013861    0.000508006   0.000112845   0.000132311      10     
end_path:out1:RootOutput                              1.0269e-05    5.37082e-05   0.000181545   3.9045e-05    4.82237e-05      10     
end_path:out1:RootOutput(write)                        0.225995      0.645538       1.12317      0.717483      0.253537        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21691.4 MB
  Peak resident set size usage (VmHWM): 2622.12 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 Size of all hits: 67
hit initial : 129.576 | hit correction : 1.15967
hit initial : 66.3441 | hit correction : 1.15968
hit initial : 48.7199 | hit correction : 1.15602
hit initial : 72.0932 | hit correction : 1.15601
hit initial : 32.4956 | hit correction : 1.15525
hit initial : 42.981 | hit correction : 1.15728
hit initial : 122.053 | hit correction : 1.15558
hit initial : 108.966 | hit correction : 1.1557
hit initial : 337.675 | hit correction : 1.15576
hit initial : 182.205 | hit correction : 1.15616
hit initial : 179.952 | hit correction : 1.15627
hit initial : 196.655 | hit correction : 1.15635
hit initial : 221.538 | hit correction : 1.15642
hit initial : 279.688 | hit correction : 1.15655
hit initial : 321.009 | hit correction : 1.15683
hit initial : 301.408 | hit correction : 1.15743
hit initial : 360.668 | hit correction : 1.15807
hit initial : 350.227 | hit correction : 1.15857
hit initial : 118.839 | hit correction : 1.15873
 *** Total Charge : 4018.55
 Looking at view: 0 | Number of hits in view: 19
hit initial : 79.705 | hit correction : 1.15967
hit initial : 132.071 | hit correction : 1.15968
hit initial : 47.0107 | hit correction : 1.15971
hit initial : 153.452 | hit correction : 1.1587
hit initial : 83.0707 | hit correction : 1.15613
hit initial : 290.958 | hit correction : 1.15851
hit initial : 331.724 | hit correction : 1.15623
hit initial : 116 | hit correction : 1.15557
hit initial : 193.92 | hit correction : 1.15785
hit initial : 50.9962 | hit correction : 1.15715
hit initial : 399.987 | hit correction : 1.15641
hit initial : 32.0002 | hit correction : 1.15788
hit initial : 67.1876 | hit correction : 1.15714
hit initial : 230.373 | hit correction : 1.15646
hit initial : 25.863 | hit correction : 1.15527
hit initial : 20.1954 | hit correction : 1.1553
hit initial : 40.8302 | hit correction : 1.15337
hit initial : 25.5847 | hit correction : 1.15336
hit initial : 61.2263 | hit correction : 1.15556
hit initial : 33.0648 | hit correction : 1.15557
hit initial : 26.0333 | hit correction : 1.1557
hit initial : 74.2654 | hit correction : 1.15602
 *** Total Charge : 2910.64
 Looking at view: 1 | Number of hits in view: 22
hit initial : 21.0369 | hit correction : 1.15784
hit initial : 26.1503 | hit correction : 1.15774
hit initial : 43.5169 | hit correction : 1.15971
hit initial : 208.467 | hit correction : 1.15966
hit initial : 28.5818 | hit correction : 1.15807
hit initial : 21.7005 | hit correction : 1.15558
hit initial : 53.7504 | hit correction : 1.15507
hit initial : 190.528 | hit correction : 1.15553
hit initial : 280.172 | hit correction : 1.15574
hit initial : 88.3727 | hit correction : 1.15599
hit initial : 156.705 | hit correction : 1.15601
hit initial : 151.364 | hit correction : 1.15616
hit initial : 217.213 | hit correction : 1.15625
hit initial : 208.971 | hit correction : 1.15634
hit initial : 165.557 | hit correction : 1.15639
hit initial : 144.658 | hit correction : 1.15644
hit initial : 240.44 | hit correction : 1.15657
hit initial : 195.648 | hit correction : 1.15674
hit initial : 168.101 | hit correction : 1.15711
hit initial : 158.551 | hit correction : 1.15739
hit initial : 155.042 | hit correction : 1.15768
hit initial : 162.627 | hit correction : 1.15795
hit initial : 170.32 | hit correction : 1.15824
hit initial : 171.382 | hit correction : 1.15849
hit initial : 169.299 | hit correction : 1.1588
hit initial : 106.187 | hit correction : 1.15885
 *** Total Charge : 4286.2
 Looking at view: 2 | Number of hits in view: 26
Begin processing the 10th record. run: 5027 subRun: 0 event: 44090 at 10-Dec-2025 08:59:45 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.574
True Electron Energy: 2.56624
Completed MCParticle Loop
True Neutrino Energy: 7.94401
 Size of all hits: 0
10-Dec-2025 08:59:47 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.048612      0.0678931     0.0996207     0.0663268     0.0141915       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000480593   0.000838348   0.00217108    0.000613851   0.000486745      10     
end_path:ana:AnalyzeEvents         0.0474057     0.0668248     0.0967646     0.0653485     0.0137923       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.5 MB
  Peak resident set size usage (VmHWM): 1533.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:27:07 UTC       justIN version: 01.05.03