justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266813.0@dunegpschedd01.fnal.gov

Jobsub ID266813.0@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 14:24:12
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-11 22:12:20
From worker nodeHostnamedunegli-8028969-0-fnpc19102.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-11 14:25:30
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T210011Z_gen_003616_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (27995s)
CPU time7h (27706s = 98%)
Max RSS bytes2912026624 (2777 MiB)
Outputting started2025-12-11 22:12:06
Output files
Finished2025-12-11 22:12:20
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

45, 
Output 2: 0.0561136, 0.251496, 0.671601, 0.0207893, 
Output 3: 0.974495, 0.018177, 0.00180944, 0.00551828, 
Output 4: 0.371068, 0.560073, 0.0635552, 0.00530325, 
Output 5: 0.531331, 0.403025, 0.0560892, 0.00955497, 
Output 6: 0.0241179, 0.0611286, 0.0758527, 0.838901, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 36160 at 11-Dec-2025 21:26:17 UTC
Boundary wire vector sizes: 6895, 6864, 6683
minwire 0: 8
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2331, 2830
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0904423, 
Output 1: 0.0386397, 0.071875, 0.145664, 0.743821, 
Output 2: 0.0662503, 0.228595, 0.673966, 0.0311882, 
Output 3: 0.918409, 0.0638979, 0.00898665, 0.00870646, 
Output 4: 0.352905, 0.561309, 0.0785907, 0.00719494, 
Output 5: 0.61038, 0.322723, 0.0561489, 0.0107485, 
Output 6: 0.0201072, 0.0650131, 0.0771346, 0.837745, 

11-Dec-2025 22:11:20 UTC  Closed output file "matt_reco2.root"
11-Dec-2025 22:11:20 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/06/8f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T210011Z_gen_003616_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2506.76       2782.93       3180.69       2754.41       167.02         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00261305     0.0104221     0.0186626     0.0102649     0.0046999       10     
reco:pandora:StandardPandora                            2505.29       2781.1        3179.23       2752.74       166.842        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000637376    0.0255891     0.197344     0.00568029     0.0575332       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000502295    0.0036842     0.0116437    0.00274542    0.00314102       10     
reco:pandoracalo:Calorimetry                          0.000308833    0.0205007     0.0711298    0.00865911     0.0241274       10     
reco:pandorapid:Chi2ParticleID                        7.2071e-05    0.000255483   0.000940199   8.9941e-05    0.000321346      10     
reco:cvnmap:CVNMapper                                  0.0455434     0.0675853     0.0878487     0.0665345     0.0128103       10     
reco:cvneva:CVNEvaluator                               0.880564       1.29582       4.14196      0.980799      0.951605        10     
reco:energyrecnumu:EnergyReco                          0.006526      0.0581269     0.130901       0.06446      0.043572        10     
reco:energyrecnue:EnergyReco                          0.000966048   0.00541955    0.00887792    0.00760597    0.00336909       10     
reco:energyrecnc:EnergyReco                           0.00730728     0.0389451     0.0971818    0.00906992     0.0383367       10     
[art]:TriggerResults:TriggerResultInserter             1.621e-05    3.69823e-05   0.000148682   1.99705e-05    3.847e-05       10     
end_path:out1:RootOutput                               7.63e-06     1.44631e-05   4.6101e-05     8.655e-06    1.17071e-05      10     
end_path:out1:RootOutput(write)                        0.281302      0.306791      0.351798      0.302877      0.0199055       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21682.6 MB
  Peak resident set size usage (VmHWM): 2912.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 *** Total Charge : 1643.57
 Looking at view: 1 | Number of hits in view: 11
hit initial : 41.0166 | hit correction : 1.20189
hit initial : 13.8259 | hit correction : 1.20358
hit initial : 16.0819 | hit correction : 1.203
hit initial : 193.42 | hit correction : 1.20191
hit initial : 155.393 | hit correction : 1.20301
hit initial : 49.7935 | hit correction : 1.20199
hit initial : 65.0973 | hit correction : 1.2028
hit initial : 56.8183 | hit correction : 1.20127
hit initial : 353.22 | hit correction : 1.20113
hit initial : 425.695 | hit correction : 1.20006
hit initial : 370.847 | hit correction : 1.19927
hit initial : 179.769 | hit correction : 1.19854
hit initial : 334.781 | hit correction : 1.19812
 *** Total Charge : 2707.54
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 8th record. run: 5026 subRun: 0 event: 36158 at 11-Dec-2025 22:11:57 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 7.71637
Completed MCParticle Loop
True Neutrino Energy: 13.22
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 36159 at 11-Dec-2025 22:12:00 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  35
Added hit list length: 35 | Max was set to: -3.40282e+38
True Electron Energy: 41.3745
Completed MCParticle Loop
True Neutrino Energy: 58.7012
 Size of all hits: 35
hit initial : 166.628 | hit correction : 1.08827
hit initial : 336.097 | hit correction : 1.08809
hit initial : 52.4696 | hit correction : 1.08824
hit initial : 55.9207 | hit correction : 1.08633
hit initial : 68.715 | hit correction : 1.08637
hit initial : 76.823 | hit correction : 1.08422
hit initial : 245.665 | hit correction : 1.08424
hit initial : 337.703 | hit correction : 1.08438
hit initial : 377.377 | hit correction : 1.08504
hit initial : 329.62 | hit correction : 1.08581
hit initial : 427.665 | hit correction : 1.08639
hit initial : 87.1239 | hit correction : 1.08711
hit initial : 332.714 | hit correction : 1.08719
 *** Total Charge : 3143.81
 Looking at view: 0 | Number of hits in view: 13
hit initial : 131.711 | hit correction : 1.08425
hit initial : 324.443 | hit correction : 1.08427
hit initial : 70.2401 | hit correction : 1.08634
hit initial : 91.1545 | hit correction : 1.08426
hit initial : 32.2255 | hit correction : 1.08834
hit initial : 311.318 | hit correction : 1.08758
hit initial : 43.1607 | hit correction : 1.08652
hit initial : 281.397 | hit correction : 1.08822
hit initial : 109.395 | hit correction : 1.08829
hit initial : 389.684 | hit correction : 1.08501
hit initial : 561.687 | hit correction : 1.08614
hit initial : 489.907 | hit correction : 1.08661
 *** Total Charge : 3080.77
 Looking at view: 1 | Number of hits in view: 12
hit initial : 752.057 | hit correction : 1.0874
hit initial : 273.702 | hit correction : 1.08831
hit initial : 94.8973 | hit correction : 1.0842
hit initial : 350.074 | hit correction : 1.08422
hit initial : 150.776 | hit correction : 1.08439
hit initial : 192.016 | hit correction : 1.08479
hit initial : 524.303 | hit correction : 1.08549
hit initial : 386.767 | hit correction : 1.08634
hit initial : 73.2052 | hit correction : 1.08637
hit initial : 30.8403 | hit correction : 1.08698
 *** Total Charge : 3072.24
 Looking at view: 2 | Number of hits in view: 10
Begin processing the 10th record. run: 5026 subRun: 0 event: 36160 at 11-Dec-2025 22:12:03 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.574
True Electron Energy: 16.6667
Completed MCParticle Loop
True Neutrino Energy: 33.9258
 Size of all hits: 0
11-Dec-2025 22:12:05 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0676401     0.0880397     0.165031      0.0835748     0.0273085       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000384553   0.000555212   0.00152265    0.000449929   0.000326647      10     
end_path:ana:AnalyzeEvents         0.0670613     0.0872949     0.162982      0.0829296     0.0268962       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19473.8 MB
  Peak resident set size usage (VmHWM): 1534.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 14:59:41 UTC       justIN version: 01.06.00