justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258132.0@dunegpschedd02.fnal.gov

Jobsub ID258132.0@dunegpschedd02.fnal.gov
Workflow ID11012
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 19:28:03
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:42:14
From worker nodeHostnamedunegli-7988605-0-fnpc18008.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:32:21
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232315Z_gen_001667_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (32540s)
CPU time8h (32195s = 98%)
Max RSS bytes3085864960 (2942 MiB)
Outputting started2025-12-06 10:34:41
Output files
Finished2025-12-06 10:42:14
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

, 0.187375, 0.12276, 0.600521, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 16670 at 06-Dec-2025 09:44:04 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  06-Dec-2025 10:32:43 UTC run: 5027 subRun: 0 event: 16670
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6520, 6572, 6583
minwire 0: 55
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2363, 2862
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.069515, 
Output 1: 0.173263, 0.0524517, 0.103867, 0.670418, 
Output 2: 0.0694898, 0.273257, 0.62511, 0.0321432, 
Output 3: 0.953465, 0.024685, 0.00501143, 0.0168389, 
Output 4: 0.705667, 0.25594, 0.0329511, 0.0054417, 
Output 5: 0.860053, 0.126081, 0.0110139, 0.00285219, 
Output 6: 0.0157699, 0.035772, 0.0478662, 0.900592, 

06-Dec-2025 10:32:48 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:32:48 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/19/44/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232315Z_gen_001667_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2475.29       3213.78       4696.12       2844.64       780.201        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00485215    0.00931748     0.025888     0.00700015    0.00623845       10     
reco:pandora:StandardPandora                            2471.72       3209.07       4681.88       2840.89       778.099        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000551126    0.072059       0.45076      0.0401565     0.127396        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00189119    0.00857991     0.0351137    0.00530448    0.00963469       10     
reco:pandoracalo:Calorimetry                          0.000424641   0.00215266    0.00830061    0.000790544   0.00261969       10     
reco:pandorapid:Chi2ParticleID                         0.0001107    0.000910058   0.00478782    0.00019919    0.00151345       10     
reco:cvnmap:CVNMapper                                  0.0444931     0.0901562     0.205126      0.0580365     0.0600792       10     
reco:cvneva:CVNEvaluator                                1.92938       3.76814       12.4436       2.94965       2.9293         10     
reco:energyrecnumu:EnergyReco                          0.0136371     0.0837519     0.262835      0.0624689     0.0771732       10     
reco:energyrecnue:EnergyReco                          0.00171218    0.00555852     0.0209026    0.00249075    0.00629856       10     
reco:energyrecnc:EnergyReco                           0.00994993     0.0712537     0.132306      0.0701975     0.0545996       10     
[art]:TriggerResults:TriggerResultInserter             2.221e-05    5.26122e-05   0.000229921   3.3245e-05    5.96013e-05      10     
end_path:out1:RootOutput                              1.2153e-05    2.58557e-05    8.828e-05    1.70415e-05   2.17489e-05      10     
end_path:out1:RootOutput(write)                        0.400106      0.601548      0.844086      0.588158      0.146107        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21784.5 MB
  Peak resident set size usage (VmHWM): 3085.86 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 306.942 | hit correction : 1.18593
hit initial : 546.189 | hit correction : 1.18596
hit initial : 234.767 | hit correction : 1.18606
hit initial : 164.84 | hit correction : 1.20678
hit initial : 307.562 | hit correction : 1.20684
hit initial : 280.33 | hit correction : 1.20704
hit initial : 255.244 | hit correction : 1.2076
hit initial : 350.214 | hit correction : 1.20836
hit initial : 202.492 | hit correction : 1.2085
hit initial : 52.173 | hit correction : 1.20855
hit initial : 398.036 | hit correction : 1.20959
hit initial : 424.896 | hit correction : 1.20971
hit initial : 315.051 | hit correction : 1.21033
hit initial : 72.9886 | hit correction : 1.21041
hit initial : 115.948 | hit correction : 1.21047
 *** Total Charge : 4993.44
 Looking at view: 0 | Number of hits in view: 19
hit initial : 27.7481 | hit correction : 1.18465
hit initial : 24.7377 | hit correction : 1.18467
hit initial : 357.532 | hit correction : 1.18576
hit initial : 68.1466 | hit correction : 1.18576
hit initial : 350.171 | hit correction : 1.18596
hit initial : 299.384 | hit correction : 1.1862
hit initial : 35.3557 | hit correction : 1.18634
hit initial : 24.9942 | hit correction : 1.2063
hit initial : 70.5931 | hit correction : 1.20689
hit initial : 142.338 | hit correction : 1.20689
hit initial : 773.632 | hit correction : 1.20694
hit initial : 294.624 | hit correction : 1.20827
hit initial : 454.19 | hit correction : 1.20853
hit initial : 614.141 | hit correction : 1.20962
hit initial : 348.265 | hit correction : 1.21004
hit initial : 222.12 | hit correction : 1.21048
hit initial : 88.3476 | hit correction : 1.2105
 *** Total Charge : 5045.15
 Looking at view: 2 | Number of hits in view: 17
Begin processing the 10th record. run: 5027 subRun: 0 event: 16670 at 06-Dec-2025 10:34:34 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  29
Added hit list length: 29 | Max was set to: -3.40282e+38
This pfp has size:  10
True Neutron Energy: 939.574
True Electron Energy: 13.1818
Completed MCParticle Loop
True Neutrino Energy: 33.2089
 Size of all hits: 29
hit initial : 94.0067 | hit correction : 1.1232
hit initial : 168.887 | hit correction : 1.12325
hit initial : 211.321 | hit correction : 1.12326
hit initial : 205.541 | hit correction : 1.13418
hit initial : 187.174 | hit correction : 1.13417
hit initial : 90.9703 | hit correction : 1.13425
hit initial : 104.271 | hit correction : 1.13423
hit initial : 87.5205 | hit correction : 1.1352
hit initial : 211.595 | hit correction : 1.13519
hit initial : 125.774 | hit correction : 1.13517
 *** Total Charge : 1681.84
 Looking at view: 0 | Number of hits in view: 10
hit initial : 102.386 | hit correction : 1.12318
hit initial : 227.761 | hit correction : 1.12321
hit initial : 131.783 | hit correction : 1.12328
hit initial : 64.3768 | hit correction : 1.13415
hit initial : 185.403 | hit correction : 1.13415
hit initial : 52.8818 | hit correction : 1.13397
hit initial : 52.5202 | hit correction : 1.13415
hit initial : 269.854 | hit correction : 1.13442
hit initial : 41.6286 | hit correction : 1.13524
hit initial : 246.415 | hit correction : 1.1352
hit initial : 238.594 | hit correction : 1.13514
hit initial : 39.2742 | hit correction : 1.13516
 *** Total Charge : 1870.21
 Looking at view: 1 | Number of hits in view: 12
hit initial : 126.169 | hit correction : 1.12318
hit initial : 171.468 | hit correction : 1.12318
hit initial : 111.919 | hit correction : 1.12336
hit initial : 57.3883 | hit correction : 1.12325
hit initial : 187.035 | hit correction : 1.13411
hit initial : 273.636 | hit correction : 1.13422
hit initial : 193.986 | hit correction : 1.13468
 *** Total Charge : 1267.08
 Looking at view: 2 | Number of hits in view: 7
06-Dec-2025 10:34:40 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.159824      0.341516      0.743601      0.212009      0.211059        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000718467    0.0485702     0.259982     0.000942417    0.0953864       10     
end_path:ana:AnalyzeEvents         0.156442       0.29262      0.526084      0.210799      0.142157        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.6 MB
  Peak resident set size usage (VmHWM): 1533.97 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 14:51:37 UTC       justIN version: 01.05.03