justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267469.4@dunegpschedd01.fnal.gov

Jobsub ID267469.4@dunegpschedd01.fnal.gov
Workflow ID11169
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-12 21:19:51
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 12:52:55
From worker nodeHostnamedunegli-7890369-0-fnpc19004.fnal.gov
cpuinfoIntel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 00:52:37
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031819Z_gen_004910_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time12h (43202s)
CPU time11h (42414s = 98%)
Max RSS bytes2918637568 (2783 MiB)
Outputting started2025-12-13 12:52:40
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11169/1/001/analysisLowE_20251213T005239Z.root
Finished2025-12-13 12:52:55
Saved logsjustin-logs:267469.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 12:50:49 UTC run: 5027 subRun: 0 event: 49100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6884, 6883, 6627
minwire 0: 59
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1834, 2333
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.102507, 
Output 1: 0.0620824, 0.0655668, 0.120444, 0.751907, 
Output 2: 0.0536357, 0.194077, 0.736722, 0.0155644, 
Output 3: 0.959951, 0.031414, 0.00292334, 0.00571133, 
Output 4: 0.284107, 0.604151, 0.10045, 0.0112925, 
Output 5: 0.434749, 0.487061, 0.0678979, 0.0102924, 
Output 6: 0.0281244, 0.0872311, 0.097248, 0.787396, 

13-Dec-2025 12:50:52 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 12:50:52 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5f/76/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031819Z_gen_004910_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3296.7        4279.36       5093.91       4298.13       546.306        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00158091    0.00701393     0.0325513    0.00213963     0.0103842       10     
reco:pandora:StandardPandora                            3294.63       4276.07       5081.6        4296.19       544.74         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00123788     0.0978648     0.211818      0.109942      0.0723143       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00220686     0.0377414     0.249173     0.00953594     0.0722667       10     
reco:pandoracalo:Calorimetry                          0.000537304   0.00261619     0.0099941    0.00114903    0.00300881       10     
reco:pandorapid:Chi2ParticleID                        0.000141374   0.000994497   0.00377742    0.000222736   0.00130129       10     
reco:cvnmap:CVNMapper                                  0.115833      0.169784      0.203014      0.169106      0.0288372       10     
reco:cvneva:CVNEvaluator                               0.932218       2.13743       10.8464       1.18327       2.90843        10     
reco:energyrecnumu:EnergyReco                          0.125792       0.16009      0.203246      0.162071      0.0217776       10     
reco:energyrecnue:EnergyReco                          0.00273216    0.00603371     0.0200906    0.00421743    0.00495849       10     
reco:energyrecnc:EnergyReco                             0.01256      0.0165083     0.0190482     0.0167304    0.00193518       10     
[art]:TriggerResults:TriggerResultInserter            3.1324e-05    7.35984e-05   0.000279934   5.2351e-05    7.01031e-05      10     
end_path:out1:RootOutput                               9.571e-06    2.27773e-05   8.4359e-05    1.59745e-05   2.08616e-05      10     
end_path:out1:RootOutput(write)                        0.435715      0.653313      0.945539       0.62979      0.127748        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19542.9 MB
  Peak resident set size usage (VmHWM): 2918.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 Looking at view: 2 | Number of hits in view: 3
Begin processing the 9th record. run: 5027 subRun: 0 event: 49099 at 13-Dec-2025 12:52:22 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  36
Added hit list length: 36 | Max was set to: -3.40282e+38
This pfp has size:  5
This pfp has size:  3
True Electron Energy: 24.956
Completed MCParticle Loop
True Neutrino Energy: 28.702
 Size of all hits: 36
hit initial : 30.9602 | hit correction : 1.26045
hit initial : 18.2961 | hit correction : 1.26043
hit initial : 159.811 | hit correction : 1.28256
hit initial : 193.408 | hit correction : 1.28502
hit initial : 47.6073 | hit correction : 1.2841
hit initial : 312.168 | hit correction : 1.28467
hit initial : 295.123 | hit correction : 1.28273
hit initial : 529.637 | hit correction : 1.28396
hit initial : 136.101 | hit correction : 1.28309
hit initial : 50.2187 | hit correction : 1.2852
hit initial : 465.659 | hit correction : 1.28435
 *** Total Charge : 2873.58
 Looking at view: 0 | Number of hits in view: 11
hit initial : 30.3639 | hit correction : 1.2605
hit initial : 19.5754 | hit correction : 1.26046
hit initial : 89.1979 | hit correction : 1.28268
hit initial : 54.3747 | hit correction : 1.28249
hit initial : 202.862 | hit correction : 1.28451
hit initial : 43.7245 | hit correction : 1.28327
hit initial : 687.806 | hit correction : 1.28446
hit initial : 365.18 | hit correction : 1.2829
hit initial : 365.162 | hit correction : 1.28439
hit initial : 81.9909 | hit correction : 1.28292
hit initial : 114.646 | hit correction : 1.28421
 *** Total Charge : 2637.16
 Looking at view: 1 | Number of hits in view: 11
hit initial : 45.972 | hit correction : 1.28525
hit initial : 45.8072 | hit correction : 1.26045
hit initial : 129.943 | hit correction : 1.28246
hit initial : 297.987 | hit correction : 1.28262
hit initial : 148.401 | hit correction : 1.28486
hit initial : 139.932 | hit correction : 1.28345
hit initial : 119.234 | hit correction : 1.28324
hit initial : 273.802 | hit correction : 1.28449
hit initial : 190.367 | hit correction : 1.28461
hit initial : 358.323 | hit correction : 1.28395
hit initial : 109.022 | hit correction : 1.28309
hit initial : 155.565 | hit correction : 1.28513
hit initial : 130.167 | hit correction : 1.28298
hit initial : 342.512 | hit correction : 1.2842
 *** Total Charge : 3191.92
 Looking at view: 2 | Number of hits in view: 14
Begin processing the 10th record. run: 5027 subRun: 0 event: 49100 at 13-Dec-2025 12:52:30 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  11
Added hit list length: 11 | Max was set to: -3.40282e+38
This pfp has size:  9
True Neutron Energy: 939.577
True Electron Energy: 4.90744
Completed MCParticle Loop
True Neutrino Energy: 8.63347
 Size of all hits: 11
hit initial : 48.9868 | hit correction : 1.3603
hit initial : 172.24 | hit correction : 1.36035
hit initial : 179.224 | hit correction : 1.36037
hit initial : 42.5639 | hit correction : 1.36038
 *** Total Charge : 602.657
 Looking at view: 0 | Number of hits in view: 4
hit initial : 29.0532 | hit correction : 1.3603
hit initial : 124.87 | hit correction : 1.36034
hit initial : 182.534 | hit correction : 1.36036
hit initial : 95.952 | hit correction : 1.36039
 *** Total Charge : 588.23
 Looking at view: 1 | Number of hits in view: 4
hit initial : 90.3131 | hit correction : 1.36032
hit initial : 249.833 | hit correction : 1.36035
hit initial : 83.8533 | hit correction : 1.36039
 *** Total Charge : 576.789
 Looking at view: 2 | Number of hits in view: 3
13-Dec-2025 12:52:39 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.201467      0.314151      0.626659      0.266026      0.132679        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000696263   0.00231195     0.0154311    0.000879762    0.0043739       10     
end_path:ana:AnalyzeEvents         0.194549      0.311539      0.625641      0.264957      0.133928        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17330.1 MB
  Peak resident set size usage (VmHWM): 1533.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:09:12 UTC       justIN version: 01.06.00