justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263633.1@dunegpschedd01.fnal.gov

Jobsub ID263633.1@dunegpschedd01.fnal.gov
Workflow ID11006
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 05:37:22
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-05 09:42:47
From worker nodeHostnamedunegli-7985799-0-fnpc19123.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-05 05:39:11
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232326Z_gen_001671_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time4h (14591s)
CPU time3h (14129s = 96%)
Max RSS bytes3066949632 (2924 MiB)
Outputting started2025-12-05 09:42:23
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11006/1/001/analysisLowE_20251205T054506Z.root
Finished2025-12-05 09:42:47
Saved logsjustin-logs:263633.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 7000, 6941, 6750
minwire 0: 20
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 798, 1297
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.124249, 
Output 1: 0.0780316, 0.0711305, 0.109896, 0.740942, 
Output 2: 0.0389839, 0.184534, 0.747471, 0.0290104, 
Output 3: 0.978129, 0.0106634, 0.00232832, 0.0088796, 
Output 4: 0.674338, 0.287194, 0.0335923, 0.00487603, 
Output 5: 0.349047, 0.398125, 0.200842, 0.0519863, 
Output 6: 0.0261522, 0.0572237, 0.0677705, 0.848854, 

05-Dec-2025 09:41:38 UTC  Closed output file "matt_reco2.root"
05-Dec-2025 09:41:38 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/db/4b/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232326Z_gen_001671_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1387.51       1411.33       1451.71       1400.52       22.9576        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00117606    0.00623561     0.0197679    0.00376773    0.00611492       10     
reco:pandora:StandardPandora                            1386.28       1409.9        1450.14       1399.29       23.0066        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00220865     0.0189168     0.070504      0.0140637     0.0202369       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00243433    0.00482576     0.0138018    0.00422326    0.00312804       10     
reco:pandoracalo:Calorimetry                          0.000235911   0.00655571     0.0227178    0.000494498   0.00902451       10     
reco:pandorapid:Chi2ParticleID                         5.463e-05    0.000225538   0.00105288     8.08e-05     0.000288507      10     
reco:cvnmap:CVNMapper                                  0.0217708     0.0258419     0.0362413     0.0239334    0.00454878       10     
reco:cvneva:CVNEvaluator                               0.811607      0.993436       2.20735      0.829137      0.413074        10     
reco:energyrecnumu:EnergyReco                         0.00158673     0.036807      0.083605      0.024586      0.0327628       10     
reco:energyrecnue:EnergyReco                          0.000833683   0.00338241    0.00758633    0.00111538    0.00298152       10     
reco:energyrecnc:EnergyReco                            0.0066302     0.0499327     0.104731      0.0708216     0.036306        10     
[art]:TriggerResults:TriggerResultInserter             1.211e-05    1.94341e-05    6.621e-05     1.357e-05    1.57392e-05      10     
end_path:out1:RootOutput                               4.99e-06      7.985e-06     2.128e-05     5.775e-06    4.92431e-06      10     
end_path:out1:RootOutput(write)                        0.267839      0.282904      0.324317      0.273201      0.0182126       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21793.1 MB
  Peak resident set size usage (VmHWM): 3066.95 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 192.592 | hit correction : 1.47549
 *** Total Charge : 2624.18
 Looking at view: 0 | Number of hits in view: 8
hit initial : 99.0009 | hit correction : 1.47568
hit initial : 182.818 | hit correction : 1.47545
hit initial : 198.855 | hit correction : 1.47523
hit initial : 178.738 | hit correction : 1.47429
hit initial : 94.1384 | hit correction : 1.47423
hit initial : 199.048 | hit correction : 1.47455
hit initial : 294.06 | hit correction : 1.47475
hit initial : 263.177 | hit correction : 1.47484
hit initial : 192.415 | hit correction : 1.47503
 *** Total Charge : 2510.62
 Looking at view: 1 | Number of hits in view: 9
Begin processing the 8th record. run: 5027 subRun: 0 event: 16708 at 05-Dec-2025 09:42:14 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  37
Added hit list length: 37 | Max was set to: -3.40282e+38
True Electron Energy: 19.1745
Completed MCParticle Loop
True Neutrino Energy: 24.5703
 Size of all hits: 37
hit initial : 21.1281 | hit correction : 1.33078
hit initial : 40.2644 | hit correction : 1.33409
hit initial : 137.128 | hit correction : 1.33398
hit initial : 129.003 | hit correction : 1.33381
hit initial : 37.6864 | hit correction : 1.33153
hit initial : 48.6412 | hit correction : 1.33159
hit initial : 21.5292 | hit correction : 1.33153
hit initial : 262.824 | hit correction : 1.33298
hit initial : 90.3306 | hit correction : 1.33234
hit initial : 170.778 | hit correction : 1.33216
hit initial : 221.79 | hit correction : 1.33181
hit initial : 224.389 | hit correction : 1.33128
hit initial : 121.595 | hit correction : 1.33093
 *** Total Charge : 2034.58
 Looking at view: 0 | Number of hits in view: 13
hit initial : 32.9278 | hit correction : 1.33161
hit initial : 39.3685 | hit correction : 1.33159
hit initial : 225.158 | hit correction : 1.33376
hit initial : 80.381 | hit correction : 1.331
hit initial : 163.187 | hit correction : 1.33395
hit initial : 99.114 | hit correction : 1.33404
hit initial : 133.479 | hit correction : 1.33411
hit initial : 45.8069 | hit correction : 1.33236
hit initial : 312.503 | hit correction : 1.33223
hit initial : 154.422 | hit correction : 1.33205
hit initial : 688.304 | hit correction : 1.33206
hit initial : 22.7436 | hit correction : 1.33152
 *** Total Charge : 2661.74
 Looking at view: 1 | Number of hits in view: 12
hit initial : 59.3436 | hit correction : 1.3316
hit initial : 24.0718 | hit correction : 1.33159
hit initial : 65.0825 | hit correction : 1.33068
hit initial : 173.388 | hit correction : 1.33092
hit initial : 137.408 | hit correction : 1.33402
hit initial : 230.91 | hit correction : 1.33308
hit initial : 285.38 | hit correction : 1.33376
hit initial : 136.216 | hit correction : 1.33253
hit initial : 150.912 | hit correction : 1.3323
hit initial : 167.077 | hit correction : 1.33213
hit initial : 207.82 | hit correction : 1.33179
hit initial : 212.828 | hit correction : 1.33142
 *** Total Charge : 2465.48
 Looking at view: 2 | Number of hits in view: 12
Begin processing the 9th record. run: 5027 subRun: 0 event: 16709 at 05-Dec-2025 09:42:17 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 3.01968
Completed MCParticle Loop
True Neutrino Energy: 6.74385
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 16710 at 05-Dec-2025 09:42:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.575
True Neutron Energy: 940.184
True Electron Energy: 31.2285
Completed MCParticle Loop
True Neutrino Energy: 46.7553
 Size of all hits: 0
05-Dec-2025 09:42:22 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0728253     0.0854465     0.120579      0.084666      0.0127871       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000329301   0.000414386   0.000503472   0.000409137   6.28208e-05      10     
end_path:ana:AnalyzeEvents         0.0723175      0.0849       0.119739      0.0841449     0.0127009       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19479.5 MB
  Peak resident set size usage (VmHWM): 1543.27 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 20:39:38 UTC       justIN version: 01.06.00