justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261237.1@dunegpschedd02.fnal.gov

Jobsub ID261237.1@dunegpschedd02.fnal.gov
Workflow ID11114
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 21:22:36
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-12 01:04:22
From worker nodeHostnamedunegli-7878610-0-fnpc19136.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-11 21:23:39
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001205Z_gen_001918_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time3h (13231s)
CPU time3h (13163s = 99%)
Max RSS bytes2929836032 (2794 MiB)
Outputting started2025-12-12 01:04:11
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11114/1/001/analysisLowE_20251211T212340Z.root
Finished2025-12-12 01:04:22
Saved logsjustin-logs:261237.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lement holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  12-Dec-2025 01:03:30 UTC run: 5027 subRun: 0 event: 19180
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  12-Dec-2025 01:03:30 UTC run: 5027 subRun: 0 event: 19180
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6802, 6790, 6614
minwire 0: 18
minwire 1: 1
minwire 2: 3
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0716295, 
Output 1: 0.0690921, 0.151402, 0.184186, 0.59532, 
Output 2: 0.0523102, 0.11237, 0.82383, 0.0114903, 
Output 3: 0.958637, 0.0376592, 0.0017792, 0.00192425, 
Output 4: 0.501022, 0.440698, 0.050218, 0.00806183, 
Output 5: 0.417612, 0.459632, 0.0973465, 0.0254091, 
Output 6: 0.100689, 0.23572, 0.131437, 0.532155, 

12-Dec-2025 01:03:32 UTC  Closed output file "matt_reco2.root"
12-Dec-2025 01:03:32 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/56/2c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001205Z_gen_001918_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1295.36       1311.61       1341.93       1302.37       17.305         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0010549    0.00628095     0.0241187    0.00297179    0.00740099       10     
reco:pandora:StandardPandora                            1294.22       1310.32       1340.8        1301.24       17.4084        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00131471     0.0124411     0.0600573    0.00338188     0.0178343       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00164445    0.00387911     0.0116227    0.00253287    0.00291435       10     
reco:pandoracalo:Calorimetry                          0.000177761    0.0053067     0.0212096    0.00112616    0.00784754       10     
reco:pandorapid:Chi2ParticleID                         6.752e-05    0.000272947   0.000585795   0.000234948   0.000158674      10     
reco:cvnmap:CVNMapper                                  0.0207387     0.0248795     0.0350613     0.0228843    0.00420704       10     
reco:cvneva:CVNEvaluator                               0.739511      0.883346       2.06497      0.742944      0.394189        10     
reco:energyrecnumu:EnergyReco                         0.00435458     0.059084      0.104592      0.0738157     0.036022        10     
reco:energyrecnue:EnergyReco                          0.000794458   0.00474514    0.00650829    0.00638692    0.00255003       10     
reco:energyrecnc:EnergyReco                           0.00612932     0.0272602     0.0909263    0.00635235     0.0326401       10     
[art]:TriggerResults:TriggerResultInserter             8.86e-06     1.39083e-05   4.2231e-05    1.09655e-05   9.56647e-06      10     
end_path:out1:RootOutput                                6.1e-06      8.113e-06     1.775e-05     6.665e-06    3.46708e-06      10     
end_path:out1:RootOutput(write)                        0.253993      0.262483       0.32088      0.256169      0.0194849       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21686.6 MB
  Peak resident set size usage (VmHWM): 2929.84 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 *** Total Charge : 2825.74
 Looking at view: 0 | Number of hits in view: 10
hit initial : 214.22 | hit correction : 1.19781
hit initial : 404.028 | hit correction : 1.19784
hit initial : 95.1815 | hit correction : 1.19868
hit initial : 44.1882 | hit correction : 1.1978
hit initial : 81.3666 | hit correction : 1.20059
hit initial : 221.689 | hit correction : 1.19959
hit initial : 87.4786 | hit correction : 1.19883
hit initial : 67.5536 | hit correction : 1.2046
hit initial : 114.116 | hit correction : 1.20057
hit initial : 186.94 | hit correction : 1.19966
hit initial : 167.078 | hit correction : 1.20042
hit initial : 163.165 | hit correction : 1.19972
hit initial : 56.1033 | hit correction : 1.20017
hit initial : 300.366 | hit correction : 1.19938
hit initial : 260.987 | hit correction : 1.1994
hit initial : 96.7356 | hit correction : 1.1994
 *** Total Charge : 3071.67
 Looking at view: 1 | Number of hits in view: 16
hit initial : 420.212 | hit correction : 1.1994
hit initial : 151.515 | hit correction : 1.19933
hit initial : 156.139 | hit correction : 1.19916
hit initial : 196.822 | hit correction : 1.19883
hit initial : 214.52 | hit correction : 1.19856
hit initial : 472.433 | hit correction : 1.19795
hit initial : 55.7789 | hit correction : 1.19725
hit initial : 358.991 | hit correction : 1.19784
hit initial : 181.557 | hit correction : 1.19771
hit initial : 192.385 | hit correction : 1.19944
 *** Total Charge : 2876.98
 Looking at view: 2 | Number of hits in view: 10
Begin processing the 9th record. run: 5027 subRun: 0 event: 19179 at 12-Dec-2025 01:04:05 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 12.7635
Completed MCParticle Loop
True Neutrino Energy: 16.056
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 19180 at 12-Dec-2025 01:04:08 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  27
Added hit list length: 27 | Max was set to: -3.40282e+38
This pfp has size:  18
True Neutron Energy: 939.567
True Neutron Energy: 939.69
True Electron Energy: 50.129
Completed MCParticle Loop
True Neutrino Energy: 59.2141
 Size of all hits: 27
hit initial : 39.3661 | hit correction : 1.22579
hit initial : 282.687 | hit correction : 1.2262
hit initial : 278.042 | hit correction : 1.22639
hit initial : 236.892 | hit correction : 1.22857
hit initial : 128.044 | hit correction : 1.22678
hit initial : 257.948 | hit correction : 1.22775
hit initial : 207.969 | hit correction : 1.22709
hit initial : 222.941 | hit correction : 1.22681
hit initial : 91.4397 | hit correction : 1.22674
hit initial : 54.3897 | hit correction : 1.22704
hit initial : 142.375 | hit correction : 1.22763
hit initial : 198.112 | hit correction : 1.22776
 *** Total Charge : 2626.32
 Looking at view: 1 | Number of hits in view: 12
hit initial : 37.3925 | hit correction : 1.22608
hit initial : 23.8451 | hit correction : 1.2257
hit initial : 295.283 | hit correction : 1.22741
hit initial : 225.093 | hit correction : 1.2278
hit initial : 237.496 | hit correction : 1.22721
hit initial : 370.62 | hit correction : 1.22681
hit initial : 51.0105 | hit correction : 1.22675
hit initial : 277.384 | hit correction : 1.22611
hit initial : 361.419 | hit correction : 1.22656
hit initial : 323.699 | hit correction : 1.228
hit initial : 57.7439 | hit correction : 1.22833
hit initial : 372.877 | hit correction : 1.22863
hit initial : 327.004 | hit correction : 1.22878
hit initial : 132.272 | hit correction : 1.2293
hit initial : 132.723 | hit correction : 1.22947
 *** Total Charge : 3960.15
 Looking at view: 2 | Number of hits in view: 15
12-Dec-2025 01:04:10 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0637204     0.071958      0.103819      0.0661543     0.011891        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000295593   0.000488743   0.000742387   0.000389373   0.000183105      10     
end_path:ana:AnalyzeEvents         0.063244      0.0713449     0.102973      0.0655333     0.0118419       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477 MB
  Peak resident set size usage (VmHWM): 1535.67 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 14:57:49 UTC       justIN version: 01.06.00