justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268060.0@dunegpschedd01.fnal.gov

Jobsub ID268060.0@dunegpschedd01.fnal.gov
Workflow ID11175
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 15:30:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 21:25:51
From worker nodeHostnamedunegli-7890610-0-fnpc9038.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 15:32:02
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T203138Z_gen_002926_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (21213s)
CPU time5h (21042s = 99%)
Max RSS bytes2842836992 (2711 MiB)
Outputting started2025-12-13 21:25:36
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11175/1/001/analysisLowE_20251213T153204Z.root
Finished2025-12-13 21:25:51
Saved logsjustin-logs:268060.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6538, 6440, 6682
minwire 0: 51
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1869, 2368
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.133749, 
Output 1: 0.0571554, 0.0433696, 0.0829488, 0.816526, 
Output 2: 0.0677373, 0.26115, 0.634234, 0.0368791, 
Output 3: 0.979253, 0.0142048, 0.00148157, 0.00506057, 
Output 4: 0.331684, 0.611432, 0.0535532, 0.0033307, 
Output 5: 0.522386, 0.34848, 0.102637, 0.0264979, 
Output 6: 0.0175705, 0.0486203, 0.0689746, 0.864835, 

13-Dec-2025 21:24:48 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 21:24:48 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/bd/37/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T203138Z_gen_002926_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1555.43       2086.7        4542.07       1627.47       926.005        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000682906   0.00455322     0.0199147    0.00082449    0.00721067       10     
reco:pandora:StandardPandora                            1553.76       2084.02       4531.76       1625.66       923.643        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0368279     0.0677252     0.204166      0.0487497     0.0479418       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00190975    0.00973983     0.0463886    0.00544098     0.0123918       10     
reco:pandoracalo:Calorimetry                          0.000433682   0.00193412     0.0134977    0.000636728   0.00385854       10     
reco:pandorapid:Chi2ParticleID                        9.6324e-05    0.000668421    0.0055765    0.00010472    0.00163638       10     
reco:cvnmap:CVNMapper                                  0.070251       0.09351      0.189161      0.0779203     0.034583        10     
reco:cvneva:CVNEvaluator                                1.07274       1.94777       8.85431       1.09211       2.31269        10     
reco:energyrecnumu:EnergyReco                          0.0853379     0.116557       0.20352      0.103216      0.0330233       10     
reco:energyrecnue:EnergyReco                          0.00134659    0.00203272    0.00622047    0.00143591    0.00141574       10     
reco:energyrecnc:EnergyReco                           0.00801766    0.00982874     0.0185966    0.00822887    0.00312376       10     
[art]:TriggerResults:TriggerResultInserter             1.555e-05    3.13956e-05   0.000150046   1.80665e-05   3.96244e-05      10     
end_path:out1:RootOutput                               6.367e-06    1.25862e-05   5.6167e-05    7.5125e-06    1.45758e-05      10     
end_path:out1:RootOutput(write)                         0.33679      0.427308      0.747641      0.352815      0.132452        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11909.3 MB
  Peak resident set size usage (VmHWM): 2842.84 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 250.652 | hit correction : 1.2957
hit initial : 341.863 | hit correction : 1.29473
hit initial : 282.681 | hit correction : 1.29454
hit initial : 55.9846 | hit correction : 1.29431
 *** Total Charge : 6062.12
 Looking at view: 0 | Number of hits in view: 23
hit initial : 38.6229 | hit correction : 1.31648
hit initial : 40.3656 | hit correction : 1.31647
hit initial : 106.078 | hit correction : 1.29145
hit initial : 200.515 | hit correction : 1.2915
hit initial : 165.467 | hit correction : 1.29154
hit initial : 57.9707 | hit correction : 1.29157
hit initial : 20.6949 | hit correction : 1.29966
hit initial : 31.6441 | hit correction : 1.29969
hit initial : 38.5901 | hit correction : 1.29395
hit initial : 88.5555 | hit correction : 1.29395
hit initial : 36.3699 | hit correction : 1.29395
hit initial : 130.179 | hit correction : 1.29576
hit initial : 428.516 | hit correction : 1.2958
hit initial : 400.822 | hit correction : 1.29595
hit initial : 64.4435 | hit correction : 1.29729
hit initial : 74.2548 | hit correction : 1.2974
hit initial : 25.9867 | hit correction : 1.2974
hit initial : 273.466 | hit correction : 1.29609
hit initial : 281.466 | hit correction : 1.29608
hit initial : 259.026 | hit correction : 1.2961
hit initial : 312.358 | hit correction : 1.29607
hit initial : 319.457 | hit correction : 1.29594
hit initial : 270.2 | hit correction : 1.29568
hit initial : 245.978 | hit correction : 1.29536
hit initial : 268.029 | hit correction : 1.29491
hit initial : 235.545 | hit correction : 1.29452
hit initial : 130.46 | hit correction : 1.29427
hit initial : 29.0216 | hit correction : 1.29417
hit initial : 29.729 | hit correction : 1.29362
 *** Total Charge : 5964.56
 Looking at view: 1 | Number of hits in view: 29
hit initial : 46.893 | hit correction : 1.29667
hit initial : 60.5716 | hit correction : 1.29665
hit initial : 26.8335 | hit correction : 1.31122
hit initial : 24.3247 | hit correction : 1.31132
hit initial : 41.6181 | hit correction : 1.31644
hit initial : 60.9814 | hit correction : 1.31644
hit initial : 119.391 | hit correction : 1.29142
hit initial : 182.496 | hit correction : 1.29149
hit initial : 140.436 | hit correction : 1.29154
hit initial : 87.3303 | hit correction : 1.29156
hit initial : 34.6898 | hit correction : 1.29969
hit initial : 33.5212 | hit correction : 1.29969
hit initial : 94.8727 | hit correction : 1.29396
hit initial : 54.1051 | hit correction : 1.29398
hit initial : 61.4555 | hit correction : 1.29568
hit initial : 330.957 | hit correction : 1.29568
hit initial : 233.351 | hit correction : 1.29581
hit initial : 165.109 | hit correction : 1.29596
hit initial : 110.572 | hit correction : 1.29738
hit initial : 126.776 | hit correction : 1.29609
hit initial : 144.255 | hit correction : 1.29744
hit initial : 108.431 | hit correction : 1.29616
hit initial : 110.918 | hit correction : 1.29615
hit initial : 107.662 | hit correction : 1.29607
hit initial : 146.044 | hit correction : 1.29603
hit initial : 181.395 | hit correction : 1.29608
hit initial : 154.469 | hit correction : 1.29613
hit initial : 108.636 | hit correction : 1.29615
hit initial : 105.049 | hit correction : 1.29615
hit initial : 135.083 | hit correction : 1.29609
hit initial : 182.977 | hit correction : 1.29602
hit initial : 149.572 | hit correction : 1.29593
hit initial : 128.328 | hit correction : 1.29578
hit initial : 124.565 | hit correction : 1.29567
hit initial : 122.187 | hit correction : 1.29553
hit initial : 123.149 | hit correction : 1.29537
hit initial : 123.164 | hit correction : 1.29515
hit initial : 116.503 | hit correction : 1.29494
hit initial : 129.349 | hit correction : 1.29472
hit initial : 122.847 | hit correction : 1.29453
hit initial : 119.545 | hit correction : 1.29431
hit initial : 115.531 | hit correction : 1.29414
hit initial : 51.6766 | hit correction : 1.29361
 *** Total Charge : 6411.63
 Looking at view: 2 | Number of hits in view: 43
13-Dec-2025 21:25:36 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0874548     0.0927759     0.104724      0.0909178    0.00502288       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000425429   0.000512374   0.000901525   0.000463423   0.000133229      10     
end_path:ana:AnalyzeEvents         0.0869006     0.0921129     0.103377      0.0902907    0.00484672       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9807.26 MB
  Peak resident set size usage (VmHWM): 1534.57 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:22:55 UTC       justIN version: 01.06.00