justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261461.0@dunegpschedd02.fnal.gov

Jobsub ID261461.0@dunegpschedd02.fnal.gov
Workflow ID11158
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-12 04:12:59
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-13 03:02:13
From worker nodeHostnamedunegli-8046965-0-fnpc23109.fnal.gov
cpuinfoAMD EPYC 9354 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 00:48:46
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T185953Z_gen_001810_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time2h (7962s)
CPU time1h (7162s = 89%)
Max RSS bytes2845032448 (2713 MiB)
Outputting started2025-12-13 03:01:29
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11158/1/001/analysisLowE_20251213T004847Z.root
Finished2025-12-13 03:02:13
Saved logsjustin-logs:261461.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

doraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 03:00:37 UTC run: 5026 subRun: 0 event: 18100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6504, 6566, 6528
minwire 0: 45
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2318, 2817
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.069036, 
Output 1: 0.146859, 0.0900966, 0.128712, 0.634332, 
Output 2: 0.0346544, 0.151619, 0.798332, 0.0153943, 
Output 3: 0.961054, 0.0253606, 0.00330827, 0.0102775, 
Output 4: 0.744858, 0.223519, 0.0263149, 0.00530841, 
Output 5: 0.587182, 0.303264, 0.0839606, 0.0255931, 
Output 6: 0.0265929, 0.0513134, 0.05825, 0.863844, 

13-Dec-2025 03:00:39 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 03:00:39 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/3b/a3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T185953Z_gen_001810_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              679.364       782.326       1062.78       750.068       107.983        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000523144   0.00779256     0.0428821    0.000889775    0.0129372       10     
reco:pandora:StandardPandora                            678.363       780.985       1059.43       748.874       107.378        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0021426     0.0365847     0.0616629     0.0365443     0.0159428       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000795727   0.00311169     0.0127631    0.00140684    0.00353868       10     
reco:pandoracalo:Calorimetry                          0.000228077   0.00213616     0.0154182    0.000384718   0.00450674       10     
reco:pandorapid:Chi2ParticleID                        7.0287e-05    0.000212408   0.00136278    7.80885e-05   0.000383811      10     
reco:cvnmap:CVNMapper                                  0.0413758     0.0633891     0.109713      0.0541255     0.0210806       10     
reco:cvneva:CVNEvaluator                                0.54562      0.881221       2.8092       0.665675      0.649271        10     
reco:energyrecnumu:EnergyReco                          0.0536581     0.0681489     0.103337      0.0628057     0.0156664       10     
reco:energyrecnue:EnergyReco                          0.000755967   0.00102853    0.00216585    0.000838522   0.000415146      10     
reco:energyrecnc:EnergyReco                           0.00404955    0.00446595    0.00652233    0.00427113    0.000692106      10     
[art]:TriggerResults:TriggerResultInserter             9.965e-06    1.91039e-05   7.1919e-05    1.2619e-05    1.78028e-05      10     
end_path:out1:RootOutput                               8.142e-06     1.121e-05    2.3396e-05    9.8605e-06    4.2575e-06       10     
end_path:out1:RootOutput(write)                        0.197911      0.272433      0.387992       0.26716      0.0606089       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21607.2 MB
  Peak resident set size usage (VmHWM): 2845.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 151.821 | hit correction : 1.33705
hit initial : 173.548 | hit correction : 1.33721
hit initial : 255.782 | hit correction : 1.33726
hit initial : 266.872 | hit correction : 1.33724
hit initial : 206.869 | hit correction : 1.33722
hit initial : 181.561 | hit correction : 1.33722
hit initial : 145.401 | hit correction : 1.33722
hit initial : 60.8764 | hit correction : 1.33724
 *** Total Charge : 4492.33
 Looking at view: 0 | Number of hits in view: 26
hit initial : 19.8923 | hit correction : 1.30107
hit initial : 24.299 | hit correction : 1.30106
hit initial : 28.1062 | hit correction : 1.30151
hit initial : 33.3614 | hit correction : 1.3015
hit initial : 78.6772 | hit correction : 1.30212
hit initial : 77.6402 | hit correction : 1.30213
hit initial : 74.4014 | hit correction : 1.33404
hit initial : 105.278 | hit correction : 1.33493
hit initial : 91.5354 | hit correction : 1.33401
hit initial : 45.1254 | hit correction : 1.33399
hit initial : 113.525 | hit correction : 1.33576
hit initial : 222.165 | hit correction : 1.33583
hit initial : 246.279 | hit correction : 1.33619
hit initial : 270.117 | hit correction : 1.33665
hit initial : 313.62 | hit correction : 1.33698
hit initial : 333.202 | hit correction : 1.3372
hit initial : 357.268 | hit correction : 1.33724
hit initial : 314.804 | hit correction : 1.33722
hit initial : 204.212 | hit correction : 1.33722
hit initial : 55.3938 | hit correction : 1.33723
 *** Total Charge : 4012.42
 Looking at view: 1 | Number of hits in view: 20
hit initial : 22.0645 | hit correction : 1.30327
hit initial : 38.4464 | hit correction : 1.30106
hit initial : 18.0388 | hit correction : 1.30151
hit initial : 49.1894 | hit correction : 1.3015
hit initial : 51.8133 | hit correction : 1.30212
hit initial : 132.864 | hit correction : 1.30213
hit initial : 161.371 | hit correction : 1.33398
hit initial : 69.4531 | hit correction : 1.33482
hit initial : 304.531 | hit correction : 1.33508
hit initial : 107.147 | hit correction : 1.33407
hit initial : 288.563 | hit correction : 1.33583
hit initial : 503.717 | hit correction : 1.33631
hit initial : 252.031 | hit correction : 1.33685
hit initial : 257.693 | hit correction : 1.33713
hit initial : 471.501 | hit correction : 1.33724
hit initial : 382.889 | hit correction : 1.33724
hit initial : 51.629 | hit correction : 1.33724
hit initial : 230.868 | hit correction : 1.33723
 *** Total Charge : 4524.62
 Looking at view: 2 | Number of hits in view: 18
Begin processing the 10th record. run: 5026 subRun: 0 event: 18100 at 13-Dec-2025 03:01:26 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  13
Added hit list length: 13 | Max was set to: -3.40282e+38
This pfp has size:  8
True Neutron Energy: 939.573
True Electron Energy: 8.00668
Completed MCParticle Loop
True Neutrino Energy: 11.7343
 Size of all hits: 13
hit initial : 78.7106 | hit correction : 1.26317
hit initial : 201.381 | hit correction : 1.26314
hit initial : 263.885 | hit correction : 1.26301
hit initial : 215.046 | hit correction : 1.26272
hit initial : 64.0397 | hit correction : 1.2626
 *** Total Charge : 1039.49
 Looking at view: 0 | Number of hits in view: 5
hit initial : 253.837 | hit correction : 1.263
hit initial : 391.717 | hit correction : 1.26297
hit initial : 127.286 | hit correction : 1.26296
 *** Total Charge : 976.083
 Looking at view: 1 | Number of hits in view: 3
hit initial : 66.0869 | hit correction : 1.26321
hit initial : 190.125 | hit correction : 1.26318
hit initial : 163.75 | hit correction : 1.26314
hit initial : 210.818 | hit correction : 1.26283
hit initial : 178.418 | hit correction : 1.26262
 *** Total Charge : 1021.98
 Looking at view: 2 | Number of hits in view: 5
13-Dec-2025 03:01:28 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0893776     0.0964157     0.129849      0.0916069     0.0117443       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000406176   0.00161094     0.0107522    0.000609525   0.00304967       10     
end_path:ana:AnalyzeEvents         0.0886698     0.0945961     0.118529      0.0908584    0.00880159       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19480.4 MB
  Peak resident set size usage (VmHWM): 1546.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 13:29:33 UTC       justIN version: 01.06.00