justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262052.1@dunegpschedd02.fnal.gov

Jobsub ID262052.1@dunegpschedd02.fnal.gov
Workflow ID11169
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 07:50:22
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 15:11:36
From worker nodeHostnamedunegli-7890418-0-fnpc18002.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 07:51:42
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T183235Z_gen_001561_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (26373s)
CPU time7h (26249s = 99%)
Max RSS bytes2856296448 (2723 MiB)
Outputting started2025-12-13 15:11:16
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11169/1/001/analysisLowE_20251213T075143Z.root
Finished2025-12-13 15:11:36
Saved logsjustin-logs:262052.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

/scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 15:09:39 UTC run: 5026 subRun: 0 event: 15610
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 7043, 7041, 6807
minwire 0: 18
minwire 1: 1
minwire 2: 7
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.131836, 
Output 1: 0.0478486, 0.0568132, 0.0846127, 0.810726, 
Output 2: 0.0829122, 0.288448, 0.597318, 0.0313223, 
Output 3: 0.975999, 0.0170954, 0.00155692, 0.00534865, 
Output 4: 0.304017, 0.637205, 0.0552339, 0.00354469, 
Output 5: 0.451342, 0.465953, 0.0709953, 0.01171, 
Output 6: 0.0149416, 0.0471944, 0.0704141, 0.86745, 

13-Dec-2025 15:09:45 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 15:09:45 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f4/10/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T183235Z_gen_001561_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1984.61       2608.88       3100.15       2549.86       347.443        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000851518    0.0113293     0.0570417    0.00157631     0.0175405       10     
reco:pandora:StandardPandora                             1979         2604.82       3095.52       2545.5        347.681        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000533247    0.038824      0.089376      0.0341005     0.0370963       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000507265   0.00518455     0.0279669    0.00306995    0.00769615       10     
reco:pandoracalo:Calorimetry                          0.000267497   0.00120633    0.00664012    0.000598096    0.001834        10     
reco:pandorapid:Chi2ParticleID                        8.2484e-05    0.000563775   0.00293415    0.000209701   0.000823188      10     
reco:cvnmap:CVNMapper                                  0.0824639     0.124682      0.263306      0.110503      0.0489972       10     
reco:cvneva:CVNEvaluator                                2.04504       3.14903       5.0206        3.06576      0.981711        10     
reco:energyrecnumu:EnergyReco                          0.0963571     0.144136      0.221932      0.137697      0.0371423       10     
reco:energyrecnue:EnergyReco                          0.00176359    0.00618763     0.0172021    0.00320587    0.00570487       10     
reco:energyrecnc:EnergyReco                            0.0101757     0.013608      0.0195415     0.0133299    0.00334056       10     
[art]:TriggerResults:TriggerResultInserter            2.4166e-05    4.89385e-05   0.000112725   3.60385e-05   2.69231e-05      10     
end_path:out1:RootOutput                              1.2082e-05    1.92973e-05   3.6807e-05    1.59935e-05   8.49106e-06      10     
end_path:out1:RootOutput(write)                        0.372189      0.565223      0.770739      0.624754      0.149209        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21576.8 MB
  Peak resident set size usage (VmHWM): 2856.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 148.04 | hit correction : 1.2418
hit initial : 107.509 | hit correction : 1.24183
hit initial : 144.933 | hit correction : 1.25495
hit initial : 104.903 | hit correction : 1.25475
hit initial : 177.329 | hit correction : 1.25534
hit initial : 130.809 | hit correction : 1.2554
hit initial : 43.5379 | hit correction : 1.25614
hit initial : 104.993 | hit correction : 1.25554
hit initial : 369.897 | hit correction : 1.25614
hit initial : 71.996 | hit correction : 1.25614
hit initial : 64.744 | hit correction : 1.25613
hit initial : 293.295 | hit correction : 1.25707
hit initial : 42.851 | hit correction : 1.25798
hit initial : 186.616 | hit correction : 1.25823
hit initial : 343.941 | hit correction : 1.25833
hit initial : 209.788 | hit correction : 1.25887
hit initial : 86.1148 | hit correction : 1.2591
 *** Total Charge : 7041.25
 Looking at view: 1 | Number of hits in view: 31
hit initial : 15.0766 | hit correction : 1.23988
hit initial : 58.6732 | hit correction : 1.24032
hit initial : 244.261 | hit correction : 1.24039
hit initial : 161.089 | hit correction : 1.24058
hit initial : 176.377 | hit correction : 1.24061
hit initial : 163.396 | hit correction : 1.24067
hit initial : 191.952 | hit correction : 1.2405
hit initial : 190.644 | hit correction : 1.24061
hit initial : 249.564 | hit correction : 1.2408
hit initial : 243.809 | hit correction : 1.24098
hit initial : 47.9997 | hit correction : 1.24274
hit initial : 361.307 | hit correction : 1.2413
hit initial : 261.045 | hit correction : 1.24144
hit initial : 101.165 | hit correction : 1.24143
hit initial : 531.48 | hit correction : 1.24166
hit initial : 143.764 | hit correction : 1.24177
hit initial : 133.366 | hit correction : 1.24182
hit initial : 106.179 | hit correction : 1.25471
hit initial : 273.07 | hit correction : 1.25536
hit initial : 454.561 | hit correction : 1.25534
hit initial : 209.769 | hit correction : 1.25618
hit initial : 307.32 | hit correction : 1.25687
hit initial : 484.63 | hit correction : 1.25793
hit initial : 395.923 | hit correction : 1.2585
hit initial : 113.536 | hit correction : 1.25907
 *** Total Charge : 7011.8
 Looking at view: 2 | Number of hits in view: 25
Begin processing the 8th record. run: 5026 subRun: 0 event: 15608 at 13-Dec-2025 15:11:03 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  1
True Neutron Energy: 945.977
True Neutron Energy: 944.038
True Neutron Energy: 940.569
True Electron Energy: 28.0714
Completed MCParticle Loop
True Neutrino Energy: 43.5582
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 15609 at 13-Dec-2025 15:11:07 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Electron Energy: 4.85351
Completed MCParticle Loop
True Neutrino Energy: 8.13886
 Size of all hits: 0
Begin processing the 10th record. run: 5026 subRun: 0 event: 15610 at 13-Dec-2025 15:11:11 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  3
Added hit list length: 3 | Max was set to: -3.40282e+38
True Neutron Energy: 939.574
True Electron Energy: 48.7222
Completed MCParticle Loop
True Neutrino Energy: 54.1068
 Size of all hits: 3
hit initial : 23.6391 | hit correction : 1.02674
 *** Total Charge : 24.2712
 Looking at view: 0 | Number of hits in view: 1
hit initial : 33.3655 | hit correction : 1.02675
 *** Total Charge : 34.2579
 Looking at view: 1 | Number of hits in view: 1
hit initial : 49.2776 | hit correction : 1.02673
 *** Total Charge : 50.5946
 Looking at view: 2 | Number of hits in view: 1
13-Dec-2025 15:11:15 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0934559     0.302747       1.54361      0.147504      0.421564        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000554269   0.00119166    0.00438512    0.000807587   0.00107307       10     
end_path:ana:AnalyzeEvents         0.0926898     0.301233       1.53808      0.146357      0.420234        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.9 MB
  Peak resident set size usage (VmHWM): 1539.9 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:08:58 UTC       justIN version: 01.06.00