justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257913.141@dunegpschedd02.fnal.gov

Jobsub ID257913.141@dunegpschedd02.fnal.gov
Workflow ID11027
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 11:27:40
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 11:47:40
From worker nodeHostnamedunegli-7832233-0-fnpc18004.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-06 00:49:59
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T164023Z_gen_000509_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time10h (39440s)
CPU time10h (37743s = 95%)
Max RSS bytes3007283200 (2867 MiB)
Outputting started2025-12-06 11:47:19
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11027/1/001/analysisLowE_20251206T005000Z.root
Finished2025-12-06 11:47:40
Saved logsjustin-logs:257913.141-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6676, 6617, 6705
minwire 0: 39
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 304, 803
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0723375, 
Output 1: 0.0420078, 0.095122, 0.0835415, 0.779329, 
Output 2: 0.082064, 0.230937, 0.64486, 0.0421386, 
Output 3: 0.961267, 0.0232969, 0.00452391, 0.010912, 
Output 4: 0.899304, 0.0840148, 0.0124853, 0.00419613, 
Output 5: 0.644422, 0.284179, 0.0564993, 0.0148999, 
Output 6: 0.0183137, 0.0508783, 0.0603324, 0.870476, 

06-Dec-2025 11:45:34 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 11:45:34 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5a/85/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T164023Z_gen_000509_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2666.09       3909.44       5628.17       3676.08       996.506        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00273602     0.0116436     0.0354947    0.00957054    0.00843927       10     
reco:pandora:StandardPandora                            2661.71       3904.54        5623         3672.18       996.56         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00369149     0.0636553      0.32075      0.0209304     0.0941144       10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.002434      0.0153807     0.086304     0.00530238     0.0242982       10     
reco:pandoracalo:Calorimetry                          0.000727044    0.0113397     0.0536053    0.00202174     0.0173039       10     
reco:pandorapid:Chi2ParticleID                        0.000154552   0.000981281   0.00715115    0.000250579   0.00205994       10     
reco:cvnmap:CVNMapper                                  0.0554825     0.102266      0.287516      0.0766094     0.0655201       10     
reco:cvneva:CVNEvaluator                                2.42545       3.82558       8.72537       3.36182       1.6882         10     
reco:energyrecnumu:EnergyReco                         0.00810299     0.105634      0.179933      0.131963      0.0619456       10     
reco:energyrecnue:EnergyReco                          0.00218505    0.00571979     0.0149758    0.00340294    0.00460879       10     
reco:energyrecnc:EnergyReco                            0.012225      0.0512265     0.150587      0.016511      0.0550976       10     
[art]:TriggerResults:TriggerResultInserter            3.2582e-05    5.67926e-05   0.000175071   3.8839e-05    4.06338e-05      10     
end_path:out1:RootOutput                              1.1161e-05    1.95148e-05   5.4142e-05    1.4718e-05    1.24989e-05      10     
end_path:out1:RootOutput(write)                        0.593733      0.706379      0.808033       0.73856      0.075849        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21779.9 MB
  Peak resident set size usage (VmHWM): 3007.28 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 216.985 | hit correction : 1.45472
 *** Total Charge : 3055.68
 Looking at view: 0 | Number of hits in view: 15
hit initial : 69.8277 | hit correction : 1.44032
hit initial : 30.2731 | hit correction : 1.4485
hit initial : 55.4981 | hit correction : 1.44855
hit initial : 22.7812 | hit correction : 1.44855
hit initial : 35.8376 | hit correction : 1.45171
hit initial : 468.004 | hit correction : 1.4524
hit initial : 419.398 | hit correction : 1.4531
hit initial : 62.4635 | hit correction : 1.4529
hit initial : 249.24 | hit correction : 1.454
hit initial : 289.432 | hit correction : 1.45409
hit initial : 149.18 | hit correction : 1.45522
 *** Total Charge : 2690.1
 Looking at view: 1 | Number of hits in view: 11
hit initial : 28.8471 | hit correction : 1.44605
hit initial : 37.4675 | hit correction : 1.44604
hit initial : 16.7916 | hit correction : 1.45089
hit initial : 83.1675 | hit correction : 1.45545
hit initial : 29.5417 | hit correction : 1.45767
hit initial : 22.4555 | hit correction : 1.45759
hit initial : 24.8779 | hit correction : 1.44035
hit initial : 95.5483 | hit correction : 1.4403
hit initial : 29.3978 | hit correction : 1.44021
hit initial : 19.7227 | hit correction : 1.44854
hit initial : 67.4446 | hit correction : 1.44854
hit initial : 181.944 | hit correction : 1.4512
hit initial : 207.091 | hit correction : 1.4517
hit initial : 131.801 | hit correction : 1.45226
hit initial : 130.404 | hit correction : 1.45265
hit initial : 140.201 | hit correction : 1.45308
hit initial : 54.3788 | hit correction : 1.45303
hit initial : 39.8756 | hit correction : 1.45302
hit initial : 117.077 | hit correction : 1.45342
hit initial : 107.648 | hit correction : 1.45367
hit initial : 124.485 | hit correction : 1.4539
hit initial : 158.307 | hit correction : 1.4542
hit initial : 207.034 | hit correction : 1.45458
hit initial : 181.673 | hit correction : 1.45512
 *** Total Charge : 3248.69
 Looking at view: 2 | Number of hits in view: 24
Begin processing the 9th record. run: 5026 subRun: 0 event: 5089 at 06-Dec-2025 11:47:08 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  19
Added hit list length: 19 | Max was set to: -3.40282e+38
True Electron Energy: 13.9701
Completed MCParticle Loop
True Neutrino Energy: 19.3521
 Size of all hits: 19
hit initial : 56.6813 | hit correction : 1.08282
hit initial : 103.429 | hit correction : 1.08281
hit initial : 185.2 | hit correction : 1.0811
hit initial : 138.554 | hit correction : 1.08108
hit initial : 91.6992 | hit correction : 1.08055
hit initial : 283.547 | hit correction : 1.08052
hit initial : 301.329 | hit correction : 1.08063
hit initial : 135.655 | hit correction : 1.08067
hit initial : 305.234 | hit correction : 1.08047
hit initial : 295.023 | hit correction : 1.08051
 *** Total Charge : 2049.64
 Looking at view: 0 | Number of hits in view: 10
hit initial : 192.611 | hit correction : 1.0828
hit initial : 210.457 | hit correction : 1.08111
hit initial : 512.021 | hit correction : 1.08075
hit initial : 187.188 | hit correction : 1.08058
hit initial : 146.046 | hit correction : 1.08049
hit initial : 137.007 | hit correction : 1.0804
hit initial : 169.85 | hit correction : 1.08043
hit initial : 324.613 | hit correction : 1.08053
hit initial : 69.677 | hit correction : 1.08057
 *** Total Charge : 2107.1
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 5026 subRun: 0 event: 5090 at 06-Dec-2025 11:47:13 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.682
True Electron Energy: 21.3567
Completed MCParticle Loop
True Neutrino Energy: 24.6549
 Size of all hits: 0
06-Dec-2025 11:47:18 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0918619     0.221897      0.644578      0.149809       0.16035        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000568855   0.000814069   0.00203008    0.000691186   0.000414729      10     
end_path:ana:AnalyzeEvents         0.0907646     0.220814      0.641827      0.148913      0.159887        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19471.1 MB
  Peak resident set size usage (VmHWM): 1530.26 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:11:00 UTC       justIN version: 01.06.00