justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267836.1@dunegpschedd01.fnal.gov

Jobsub ID267836.1@dunegpschedd01.fnal.gov
Workflow ID11174
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 06:38:19
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 19:18:35
From worker nodeHostnamedunegli-7890499-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 06:39:41
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T130859Z_gen_000195_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time12h (45508s)
CPU time12h (44556s = 97%)
Max RSS bytes2792742912 (2663 MiB)
Outputting started2025-12-13 19:18:09
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11174/1/001/analysisLowE_20251213T063945Z.root
Finished2025-12-13 19:18:35
Saved logsjustin-logs:267836.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 19:16:43 UTC run: 4696 subRun: 0 event: 1950
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6902, 6908, 6750
minwire 0: 51
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0534719, 
Output 1: 0.0452459, 0.0801123, 0.131072, 0.74357, 
Output 2: 0.0492015, 0.144794, 0.787078, 0.0189266, 
Output 3: 0.930799, 0.0505801, 0.00870668, 0.00991441, 
Output 4: 0.68581, 0.241152, 0.0561958, 0.0168424, 
Output 5: 0.689517, 0.256167, 0.0433378, 0.0109777, 
Output 6: 0.0342577, 0.103034, 0.0883994, 0.774309, 

13-Dec-2025 19:16:50 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 19:16:50 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/82/41/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T130859Z_gen_000195_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3174.07       4481.56       7355.93       3808.12       1428.44        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00112199    0.00592639     0.0252319     0.0016983    0.00795585       10     
reco:pandora:StandardPandora                            3169.69       4475.41       7337.14       3803.55       1425.17        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0005414     0.0893158     0.269654      0.0973436     0.0727243       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000518737    0.0180085     0.0814332    0.00641929     0.0259414       10     
reco:pandoracalo:Calorimetry                          0.000277463   0.00391901     0.0179312    0.00107529    0.00613449       10     
reco:pandorapid:Chi2ParticleID                        0.000101451    0.0014676    0.00858093    0.000212245    0.0026948       10     
reco:cvnmap:CVNMapper                                  0.0947323     0.138537      0.294003      0.110007      0.0658309       10     
reco:cvneva:CVNEvaluator                                2.21297       4.92993       16.4179       3.53178       3.94159        10     
reco:energyrecnumu:EnergyReco                          0.102131      0.165207       0.4276       0.128191      0.0903544       10     
reco:energyrecnue:EnergyReco                          0.00258426    0.00751743     0.0226479    0.00337149    0.00646592       10     
reco:energyrecnc:EnergyReco                            0.012066      0.0178704     0.0248709     0.016407     0.00422482       10     
[art]:TriggerResults:TriggerResultInserter            3.2652e-05    0.000101961   0.000573179   4.38485e-05   0.000158267      10     
end_path:out1:RootOutput                               9.618e-06    3.25131e-05   0.000176381    1.602e-05    4.82709e-05      10     
end_path:out1:RootOutput(write)                        0.405608       0.7681        1.49386      0.611174      0.305536        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21577.8 MB
  Peak resident set size usage (VmHWM): 2792.74 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 169.775 | hit correction : 1.14665
hit initial : 161.684 | hit correction : 1.14688
hit initial : 142.895 | hit correction : 1.14699
hit initial : 167.985 | hit correction : 1.14707
hit initial : 44.0469 | hit correction : 1.14785
hit initial : 237.354 | hit correction : 1.14724
hit initial : 325.105 | hit correction : 1.14786
hit initial : 27.1553 | hit correction : 1.14799
hit initial : 241.901 | hit correction : 1.14736
hit initial : 157.152 | hit correction : 1.14753
hit initial : 176.834 | hit correction : 1.14775
hit initial : 184.023 | hit correction : 1.14808
hit initial : 195.885 | hit correction : 1.14834
hit initial : 215.045 | hit correction : 1.1486
hit initial : 195.824 | hit correction : 1.14882
hit initial : 172.037 | hit correction : 1.14909
hit initial : 150.757 | hit correction : 1.14933
hit initial : 147.678 | hit correction : 1.14955
hit initial : 161.983 | hit correction : 1.1497
hit initial : 77.7521 | hit correction : 1.14983
 *** Total Charge : 5046.72
 Looking at view: 2 | Number of hits in view: 23
Begin processing the 10th record. run: 4696 subRun: 0 event: 1950 at 13-Dec-2025 19:18:04 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  42
Added hit list length: 42 | Max was set to: -3.40282e+38
This pfp has size:  12
True Neutron Energy: 939.569
True Neutron Energy: 940.495
True Electron Energy: 37.6878
Completed MCParticle Loop
True Neutrino Energy: 54.9601
 Size of all hits: 42
hit initial : 240.327 | hit correction : 1.46389
hit initial : 105.362 | hit correction : 1.46392
hit initial : 88.6581 | hit correction : 1.46401
hit initial : 90.2951 | hit correction : 1.46299
hit initial : 348.542 | hit correction : 1.46297
hit initial : 344.309 | hit correction : 1.46244
hit initial : 242.834 | hit correction : 1.46159
hit initial : 252.582 | hit correction : 1.46022
hit initial : 751.938 | hit correction : 1.46017
hit initial : 573.447 | hit correction : 1.46006
hit initial : 172.551 | hit correction : 1.45994
 *** Total Charge : 4692.27
 Looking at view: 0 | Number of hits in view: 11
hit initial : 27.8516 | hit correction : 1.45427
hit initial : 85.3768 | hit correction : 1.46398
hit initial : 131.235 | hit correction : 1.46404
hit initial : 85.7628 | hit correction : 1.464
hit initial : 180.153 | hit correction : 1.4631
hit initial : 490.376 | hit correction : 1.46297
hit initial : 211.847 | hit correction : 1.46277
hit initial : 107.247 | hit correction : 1.46173
hit initial : 142.269 | hit correction : 1.46085
hit initial : 378.073 | hit correction : 1.46055
hit initial : 538.193 | hit correction : 1.46023
hit initial : 441.857 | hit correction : 1.45997
hit initial : 63.1459 | hit correction : 1.45977
hit initial : 241.225 | hit correction : 1.45983
 *** Total Charge : 4566.16
 Looking at view: 1 | Number of hits in view: 14
hit initial : 155.592 | hit correction : 1.46383
hit initial : 276.616 | hit correction : 1.46388
hit initial : 311.371 | hit correction : 1.46331
hit initial : 242.883 | hit correction : 1.46293
hit initial : 135.907 | hit correction : 1.46259
hit initial : 133.012 | hit correction : 1.46223
hit initial : 160.692 | hit correction : 1.46179
hit initial : 20.7673 | hit correction : 1.46087
hit initial : 176.334 | hit correction : 1.46124
hit initial : 155.379 | hit correction : 1.46067
hit initial : 200.653 | hit correction : 1.46038
hit initial : 347.839 | hit correction : 1.46033
hit initial : 325.28 | hit correction : 1.46018
hit initial : 268.799 | hit correction : 1.46001
hit initial : 208.447 | hit correction : 1.45983
hit initial : 141.863 | hit correction : 1.45973
hit initial : 52.6541 | hit correction : 1.45964
 *** Total Charge : 4843.41
 Looking at view: 2 | Number of hits in view: 17
13-Dec-2025 19:18:08 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0973073     0.154491      0.293088      0.135388      0.0583877       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000697824   0.00163518     0.007154      0.0010002    0.00185364       10     
end_path:ana:AnalyzeEvents         0.0960232     0.152427      0.292078      0.130624      0.0587272       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19473.9 MB
  Peak resident set size usage (VmHWM): 1533.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:55 UTC       justIN version: 01.06.00