justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267936.1@dunegpschedd01.fnal.gov

Jobsub ID267936.1@dunegpschedd01.fnal.gov
Workflow ID11174
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 09:20:27
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 21:03:27
From worker nodeHostnamedunegli-7890501-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 09:22:32
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T225534Z_gen_000904_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time11h (42039s)
CPU time11h (41280s = 98%)
Max RSS bytes2795593728 (2666 MiB)
Outputting started2025-12-13 21:03:12
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11174/1/001/analysisLowE_20251213T092236Z.root
Finished2025-12-13 21:03:27
Saved logsjustin-logs:267936.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

i/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 21:01:59 UTC run: 5027 subRun: 0 event: 9040
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6913, 6868, 6712
minwire 0: 53
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1673, 2172
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0870312, 
Output 1: 0.0384528, 0.0423519, 0.0701482, 0.849047, 
Output 2: 0.0939588, 0.281797, 0.588157, 0.0360866, 
Output 3: 0.976891, 0.0167244, 0.00155704, 0.00482765, 
Output 4: 0.329826, 0.609069, 0.0572298, 0.00387531, 
Output 5: 0.780112, 0.192226, 0.0219805, 0.00568215, 
Output 6: 0.0156845, 0.0512463, 0.06689, 0.866179, 

13-Dec-2025 21:02:04 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 21:02:04 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/07/0d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T225534Z_gen_000904_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3122.09       4127.73       7640.18       3784.24       1232.25        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00111191    0.00859016     0.0552377    0.00211895     0.0160877       10     
reco:pandora:StandardPandora                            3119.23       4123.44       7626.73       3781.14       1229.21        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000592656    0.104294      0.215516      0.117959      0.0617693       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000522904    0.0149067     0.0961624    0.00612355     0.0274088       10     
reco:pandoracalo:Calorimetry                          0.000318781    0.0032626     0.0231109    0.000892557   0.00665286       10     
reco:pandorapid:Chi2ParticleID                        0.000106721   0.00107361     0.0089177     0.0002101    0.00261545       10     
reco:cvnmap:CVNMapper                                  0.0776944     0.119998      0.281545      0.102807      0.0573244       10     
reco:cvneva:CVNEvaluator                                2.05592       3.35253       11.3952       2.35281       2.72473        10     
reco:energyrecnumu:EnergyReco                          0.113683      0.165505      0.268145      0.154376      0.0420076       10     
reco:energyrecnue:EnergyReco                          0.00225687    0.00545772     0.0117128    0.00321508     0.0039446       10     
reco:energyrecnc:EnergyReco                            0.0104274     0.0127579     0.0186691     0.011254     0.00293766       10     
[art]:TriggerResults:TriggerResultInserter             2.608e-05    6.29546e-05   0.000231365   4.4554e-05    5.77029e-05      10     
end_path:out1:RootOutput                              1.3625e-05    2.64387e-05   9.3676e-05    1.6732e-05    2.32131e-05      10     
end_path:out1:RootOutput(write)                        0.386161      0.502508       1.07165      0.411244      0.211986        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21577.4 MB
  Peak resident set size usage (VmHWM): 2795.59 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 386.823 | hit correction : 1.18612
hit initial : 47.566 | hit correction : 1.18694
hit initial : 29.6518 | hit correction : 1.18748
hit initial : 420.681 | hit correction : 1.18745
hit initial : 342.826 | hit correction : 1.18756
hit initial : 68.1562 | hit correction : 1.18769
 *** Total Charge : 3765.18
 Looking at view: 1 | Number of hits in view: 13
hit initial : 36.5175 | hit correction : 1.18459
hit initial : 112.691 | hit correction : 1.18587
hit initial : 89.1219 | hit correction : 1.18586
hit initial : 292.718 | hit correction : 1.18622
hit initial : 394.501 | hit correction : 1.18615
hit initial : 194.254 | hit correction : 1.18548
hit initial : 1106.03 | hit correction : 1.18581
hit initial : 373.145 | hit correction : 1.18613
hit initial : 158.94 | hit correction : 1.18637
hit initial : 144.462 | hit correction : 1.18669
hit initial : 243.572 | hit correction : 1.1869
hit initial : 186.736 | hit correction : 1.18714
hit initial : 406.438 | hit correction : 1.18745
hit initial : 284.802 | hit correction : 1.18752
hit initial : 162.555 | hit correction : 1.18773
hit initial : 65.2802 | hit correction : 1.18789
 *** Total Charge : 5044.39
 Looking at view: 2 | Number of hits in view: 16
Begin processing the 10th record. run: 5027 subRun: 0 event: 9040 at 13-Dec-2025 21:03:05 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  39
Added hit list length: 39 | Max was set to: -3.40282e+38
This pfp has size:  9
True Neutron Energy: 939.568
True Electron Energy: 21.9555
Completed MCParticle Loop
True Neutrino Energy: 37.4554
 Size of all hits: 39
hit initial : 74.0877 | hit correction : 1.22691
hit initial : 127.765 | hit correction : 1.22692
hit initial : 43.4286 | hit correction : 1.22692
hit initial : 34.3169 | hit correction : 1.23202
hit initial : 23.838 | hit correction : 1.23204
hit initial : 46.8002 | hit correction : 1.23394
hit initial : 128.481 | hit correction : 1.23414
hit initial : 814.447 | hit correction : 1.2342
hit initial : 594.952 | hit correction : 1.23414
hit initial : 96.4698 | hit correction : 1.23465
 *** Total Charge : 2447.46
 Looking at view: 0 | Number of hits in view: 10
hit initial : 66.5724 | hit correction : 1.22691
hit initial : 139.279 | hit correction : 1.22692
hit initial : 63.4576 | hit correction : 1.22695
hit initial : 39.086 | hit correction : 1.23204
hit initial : 18.13 | hit correction : 1.23205
hit initial : 188.297 | hit correction : 1.23394
hit initial : 340.495 | hit correction : 1.23397
hit initial : 308.808 | hit correction : 1.23407
hit initial : 264.579 | hit correction : 1.23419
hit initial : 213.085 | hit correction : 1.23429
hit initial : 213.373 | hit correction : 1.23448
hit initial : 236.8 | hit correction : 1.23462
hit initial : 134.665 | hit correction : 1.23468
 *** Total Charge : 2746.09
 Looking at view: 1 | Number of hits in view: 13
hit initial : 40.0422 | hit correction : 1.22582
hit initial : 165.863 | hit correction : 1.22689
hit initial : 97.8715 | hit correction : 1.22695
hit initial : 26.8322 | hit correction : 1.23205
hit initial : 40.2068 | hit correction : 1.23204
hit initial : 30.0204 | hit correction : 1.2339
hit initial : 238.413 | hit correction : 1.23391
hit initial : 125.74 | hit correction : 1.23403
hit initial : 172.819 | hit correction : 1.23394
hit initial : 138.65 | hit correction : 1.23397
hit initial : 132.395 | hit correction : 1.23412
hit initial : 223.804 | hit correction : 1.23419
hit initial : 222.563 | hit correction : 1.23427
hit initial : 206.751 | hit correction : 1.23444
hit initial : 328.857 | hit correction : 1.23463
hit initial : 104.798 | hit correction : 1.2347
 *** Total Charge : 2830.93
 Looking at view: 2 | Number of hits in view: 16
13-Dec-2025 21:03:11 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0907775     0.157102      0.326905       0.11491      0.0839145       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000494682    0.0203128     0.197708     0.000619642    0.0591317       10     
end_path:ana:AnalyzeEvents         0.0899565     0.136506      0.314905      0.114042      0.0619912       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19478.2 MB
  Peak resident set size usage (VmHWM): 1538.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:05:49 UTC       justIN version: 01.06.00