justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258295.1@dunegpschedd02.fnal.gov

Jobsub ID258295.1@dunegpschedd02.fnal.gov
Workflow ID11029
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 23:47:16
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 09:59:52
From worker nodeHostnamedunegli-7988752-0-fnpc18001.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:40:32
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T020805Z_gen_003616_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (29542s)
CPU time8h (29090s = 98%)
Max RSS bytes3024130048 (2884 MiB)
Outputting started2025-12-06 09:52:55
Output files
Finished2025-12-06 09:59:52
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 7007, 6944, 6929
minwire 0: 31
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 325, 824
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.105478, 
Output 1: 0.350304, 0.0645056, 0.224039, 0.361151, 
Output 2: 0.0206886, 0.0574169, 0.915115, 0.00677969, 
Output 3: 0.962991, 0.0315327, 0.00208252, 0.00339412, 
Output 4: 0.577101, 0.372396, 0.0408507, 0.00965207, 
Output 5: 0.168628, 0.387694, 0.312606, 0.131072, 
Output 6: 0.124782, 0.229599, 0.130921, 0.514698, 

06-Dec-2025 09:51:58 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:51:58 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/88/81/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T020805Z_gen_003616_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1968.8        2918.74       4030.63       2871.62       541.42         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00619776     0.0109296     0.0267524    0.00787956    0.00703814       10     
reco:pandora:StandardPandora                            1966.45       2914.96       4021.1        2868.76       539.802        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000484764    0.0532718     0.235425      0.0466217     0.0646509       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000414021   0.00796363     0.0330343    0.00347075     0.0105169       10     
reco:pandoracalo:Calorimetry                          0.000211379   0.00669832     0.0408654    0.000897277    0.0118167       10     
reco:pandorapid:Chi2ParticleID                        8.1274e-05    0.000771991   0.00340497    0.000179193   0.00117023       10     
reco:cvnmap:CVNMapper                                  0.0468529     0.0654953     0.114277      0.0639375     0.0184431       10     
reco:cvneva:CVNEvaluator                                1.81948       2.95634       8.28903       2.3589        1.84492        10     
reco:energyrecnumu:EnergyReco                          0.0164115     0.103811       0.20694      0.119105      0.0628234       10     
reco:energyrecnue:EnergyReco                          0.00237503    0.00680365     0.0160673    0.00338743    0.00521607       10     
reco:energyrecnc:EnergyReco                           0.00971031     0.0467553     0.130262      0.0149983     0.0508163       10     
[art]:TriggerResults:TriggerResultInserter            2.3214e-05     4.667e-05    0.000189697   2.93955e-05   4.79179e-05      10     
end_path:out1:RootOutput                               9.488e-06    1.69059e-05   5.9863e-05    1.2313e-05    1.43795e-05      10     
end_path:out1:RootOutput(write)                        0.357153      0.519926      0.656826      0.562457      0.102219        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21756.3 MB
  Peak resident set size usage (VmHWM): 3024.13 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 198.07 | hit correction : 1.26468
hit initial : 122.142 | hit correction : 1.26467
hit initial : 118.068 | hit correction : 1.26464
hit initial : 17.8414 | hit correction : 1.26468
hit initial : 97.6988 | hit correction : 1.26307
hit initial : 553.303 | hit correction : 1.26431
hit initial : 214.619 | hit correction : 1.2645
hit initial : 143.645 | hit correction : 1.26451
hit initial : 155.894 | hit correction : 1.26454
 *** Total Charge : 4799.23
 Looking at view: 2 | Number of hits in view: 21
Begin processing the 8th record. run: 5027 subRun: 0 event: 36158 at 06-Dec-2025 09:52:44 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  41
Added hit list length: 41 | Max was set to: -3.40282e+38
True Electron Energy: 30.4124
Completed MCParticle Loop
True Neutrino Energy: 34.1881
 Size of all hits: 41
hit initial : 159.752 | hit correction : 1.17951
hit initial : 29.7732 | hit correction : 1.17698
hit initial : 22.869 | hit correction : 1.17701
hit initial : 16.6093 | hit correction : 1.17538
hit initial : 36.3012 | hit correction : 1.17543
hit initial : 36.5768 | hit correction : 1.17809
hit initial : 59.9056 | hit correction : 1.17811
hit initial : 259.032 | hit correction : 1.17895
hit initial : 261.32 | hit correction : 1.17918
hit initial : 45.4571 | hit correction : 1.18092
hit initial : 188.886 | hit correction : 1.18083
hit initial : 120.494 | hit correction : 1.18092
hit initial : 433.192 | hit correction : 1.18175
hit initial : 36.4989 | hit correction : 1.18164
hit initial : 281.032 | hit correction : 1.18261
hit initial : 317.944 | hit correction : 1.18319
hit initial : 337.083 | hit correction : 1.18382
 *** Total Charge : 3121.43
 Looking at view: 0 | Number of hits in view: 17
hit initial : 27.6049 | hit correction : 1.17818
hit initial : 97.1225 | hit correction : 1.17846
hit initial : 136.632 | hit correction : 1.17905
hit initial : 85.3417 | hit correction : 1.18091
hit initial : 749.048 | hit correction : 1.18104
hit initial : 65.3081 | hit correction : 1.18091
hit initial : 18.9755 | hit correction : 1.18166
hit initial : 64.9258 | hit correction : 1.18262
hit initial : 97.1853 | hit correction : 1.18268
hit initial : 141.959 | hit correction : 1.18347
hit initial : 47.7188 | hit correction : 1.18352
 *** Total Charge : 1809.26
 Looking at view: 1 | Number of hits in view: 11
hit initial : 244.172 | hit correction : 1.18415
hit initial : 131.534 | hit correction : 1.17812
hit initial : 400.509 | hit correction : 1.17864
hit initial : 333.015 | hit correction : 1.17935
hit initial : 181.867 | hit correction : 1.18091
hit initial : 38.3604 | hit correction : 1.18074
hit initial : 715.497 | hit correction : 1.18086
hit initial : 27.0113 | hit correction : 1.18167
hit initial : 184.95 | hit correction : 1.18179
hit initial : 239.821 | hit correction : 1.18252
hit initial : 272.744 | hit correction : 1.18296
hit initial : 219.923 | hit correction : 1.18345
hit initial : 246.503 | hit correction : 1.18393
 *** Total Charge : 3822.7
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 9th record. run: 5027 subRun: 0 event: 36159 at 06-Dec-2025 09:52:47 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 2.59897
Completed MCParticle Loop
True Neutrino Energy: 6.32305
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 36160 at 06-Dec-2025 09:52:51 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.577
True Electron Energy: 36.2276
Completed MCParticle Loop
True Neutrino Energy: 65.0243
 Size of all hits: 0
06-Dec-2025 09:52:54 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0945604     0.111458      0.128504      0.110643      0.0111477       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000531662   0.000753034   0.00145058     0.0006736    0.000240471      10     
end_path:ana:AnalyzeEvents         0.0936479     0.110407       0.12764      0.109804      0.0110697       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19479.7 MB
  Peak resident set size usage (VmHWM): 1540.98 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:08:19 UTC       justIN version: 01.06.00