justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266994.0@dunegpschedd01.fnal.gov

Jobsub ID266994.0@dunegpschedd01.fnal.gov
Workflow ID11094
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 20:36:34
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-12 00:36:28
From worker nodeHostnamedunegli-7878621-0-fnpc19118.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-11 20:37:52
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T193633Z_gen_002461_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time3h (14302s)
CPU time3h (14188s = 99%)
Max RSS bytes3039346688 (2898 MiB)
Outputting started2025-12-12 00:36:14
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11094/1/001/analysisLowE_20251211T203753Z.root
Finished2025-12-12 00:36:28
Saved logsjustin-logs:266994.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ine#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6745, 6822, 6674
minwire 0: 64
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1407, 1906
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.129375, 
Output 1: 0.0514967, 0.16095, 0.179451, 0.608102, 
Output 2: 0.0335559, 0.164228, 0.785087, 0.0171289, 
Output 3: 0.954845, 0.0356522, 0.00431586, 0.00518703, 
Output 4: 0.454744, 0.447665, 0.0856225, 0.0119683, 
Output 5: 0.320023, 0.461432, 0.182029, 0.0365159, 
Output 6: 0.0309836, 0.0899656, 0.0967621, 0.782289, 

12-Dec-2025 00:35:34 UTC  Closed output file "matt_reco2.root"
12-Dec-2025 00:35:34 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f0/4e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T193633Z_gen_002461_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1347.28       1417.23       1459.54       1428.99       36.2751        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00148197    0.00620718     0.0188558    0.00400644    0.00542361       10     
reco:pandora:StandardPandora                            1346.04       1415.85       1458.31       1427.7        36.3182        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000558685    0.016529      0.0621973    0.00848901     0.0182194       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000954217    0.0058142     0.0216433    0.00379994    0.00593275       10     
reco:pandoracalo:Calorimetry                          0.000168391   0.00787992     0.0528966    0.000306702    0.0163005       10     
reco:pandorapid:Chi2ParticleID                         5.435e-05    0.000334382   0.00176095    8.6841e-05    0.000505219      10     
reco:cvnmap:CVNMapper                                  0.0240271     0.0287814     0.0432454     0.0279361    0.00522562       10     
reco:cvneva:CVNEvaluator                               0.807152      0.950061       2.10781      0.821851      0.386049        10     
reco:energyrecnumu:EnergyReco                         0.00473138     0.0354966     0.0896217    0.00907635     0.0349134       10     
reco:energyrecnue:EnergyReco                          0.000870097   0.00210401    0.00679383    0.000935556   0.00231029       10     
reco:energyrecnc:EnergyReco                           0.00640345     0.046994      0.0870645     0.0679944     0.0333319       10     
[art]:TriggerResults:TriggerResultInserter             1.088e-05    1.8843e-05     5.676e-05    1.4075e-05    1.29752e-05      10     
end_path:out1:RootOutput                               6.75e-06     8.8901e-06     1.788e-05     7.395e-06    3.4165e-06       10     
end_path:out1:RootOutput(write)                        0.267697       0.27924      0.293275      0.277255     0.00707624       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21751.2 MB
  Peak resident set size usage (VmHWM): 3039.35 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 277.285 | hit correction : 1.1694
hit initial : 293.769 | hit correction : 1.1696
hit initial : 249.567 | hit correction : 1.16978
 *** Total Charge : 2261.5
 Looking at view: 1 | Number of hits in view: 10
hit initial : 15.1036 | hit correction : 1.17006
hit initial : 97.7987 | hit correction : 1.16906
hit initial : 393.361 | hit correction : 1.16914
hit initial : 341.625 | hit correction : 1.16929
hit initial : 386.863 | hit correction : 1.16953
hit initial : 316.758 | hit correction : 1.16976
hit initial : 346.935 | hit correction : 1.16997
 *** Total Charge : 2220.24
 Looking at view: 2 | Number of hits in view: 7
Begin processing the 9th record. run: 5026 subRun: 0 event: 24609 at 12-Dec-2025 00:36:08 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  44
Added hit list length: 44 | Max was set to: -3.40282e+38
True Electron Energy: 16.68
Completed MCParticle Loop
True Neutrino Energy: 24.4683
 Size of all hits: 44
hit initial : 39.7166 | hit correction : 1.3257
hit initial : 25.9258 | hit correction : 1.32794
hit initial : 46.9541 | hit correction : 1.32674
hit initial : 28.9322 | hit correction : 1.32385
hit initial : 99.5056 | hit correction : 1.32387
hit initial : 123.287 | hit correction : 1.32387
hit initial : 41.5857 | hit correction : 1.32384
hit initial : 32.0346 | hit correction : 1.32465
hit initial : 121.638 | hit correction : 1.32472
hit initial : 192.139 | hit correction : 1.32483
hit initial : 198.33 | hit correction : 1.32511
hit initial : 192.869 | hit correction : 1.32551
hit initial : 224.209 | hit correction : 1.3259
hit initial : 224.191 | hit correction : 1.32636
hit initial : 150.816 | hit correction : 1.32664
 *** Total Charge : 2308.98
 Looking at view: 0 | Number of hits in view: 15
hit initial : 102.549 | hit correction : 1.32661
hit initial : 272.278 | hit correction : 1.32564
hit initial : 32.6762 | hit correction : 1.32658
hit initial : 38.3576 | hit correction : 1.32886
hit initial : 45.5298 | hit correction : 1.32736
hit initial : 56.8462 | hit correction : 1.32738
hit initial : 13.897 | hit correction : 1.3274
hit initial : 31.9564 | hit correction : 1.32683
hit initial : 88.2699 | hit correction : 1.32683
hit initial : 39.3759 | hit correction : 1.32681
hit initial : 42.1497 | hit correction : 1.3238
hit initial : 134.521 | hit correction : 1.3239
hit initial : 29.4162 | hit correction : 1.32469
hit initial : 185.762 | hit correction : 1.32483
hit initial : 242.704 | hit correction : 1.32507
hit initial : 294.829 | hit correction : 1.32614
 *** Total Charge : 2188.95
 Looking at view: 1 | Number of hits in view: 16
hit initial : 21.0783 | hit correction : 1.32784
hit initial : 35.1677 | hit correction : 1.32784
hit initial : 46.7106 | hit correction : 1.32685
hit initial : 123.707 | hit correction : 1.3267
hit initial : 11.6927 | hit correction : 1.32481
hit initial : 130.681 | hit correction : 1.32382
hit initial : 135.759 | hit correction : 1.32392
hit initial : 274.177 | hit correction : 1.32477
hit initial : 250.39 | hit correction : 1.32498
hit initial : 161.358 | hit correction : 1.32547
hit initial : 155.256 | hit correction : 1.32573
hit initial : 164.465 | hit correction : 1.32608
hit initial : 160.527 | hit correction : 1.32643
 *** Total Charge : 2214.72
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 10th record. run: 5026 subRun: 0 event: 24610 at 12-Dec-2025 00:36:11 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.569
True Neutron Energy: 939.705
True Electron Energy: 18.2999
Completed MCParticle Loop
True Neutrino Energy: 27.257
 Size of all hits: 0
12-Dec-2025 00:36:14 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0697493     0.0825093     0.0934035     0.0849512    0.00845477       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000347853   0.000594047   0.00144367    0.000405143   0.000334745      10     
end_path:ana:AnalyzeEvents         0.0691249     0.0817533     0.0925426     0.0839143    0.00854422       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19475.7 MB
  Peak resident set size usage (VmHWM): 1536.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 16:30:07 UTC       justIN version: 01.06.00