justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 259479.160@dunegpschedd02.fnal.gov

Jobsub ID259479.160@dunegpschedd02.fnal.gov
Workflow ID11111
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:44
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:11:07
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 17:01:31
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T224713Z_gen_000636_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (57770s)
CPU time15h (56939s = 98%)
Max RSS bytes2645872640 (2523 MiB)
Outputting started2025-12-10 09:04:22
Output files
Finished2025-12-10 09:11:07
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1: 0.211961, 0.0576935, 0.164787, 0.565558, 
Output 2: 0.0575959, 0.155774, 0.762396, 0.0242342, 
Output 3: 0.962197, 0.0239099, 0.00385509, 0.0100382, 
Output 4: 0.615476, 0.346198, 0.0329411, 0.00538449, 
Output 5: 0.503052, 0.384278, 0.0880681, 0.0246019, 
Output 6: 0.0654087, 0.115895, 0.0925337, 0.726162, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 6360 at 10-Dec-2025 08:29:48 UTC
Boundary wire vector sizes: 6847, 6767, 6734
minwire 0: 59
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1706, 2205
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0658271, 
Output 1: 0.0443296, 0.103221, 0.101087, 0.751362, 
Output 2: 0.0408308, 0.170075, 0.770467, 0.0186274, 
Output 3: 0.971082, 0.0211193, 0.00245714, 0.0053414, 
Output 4: 0.717191, 0.228313, 0.0446392, 0.00985671, 
Output 5: 0.759105, 0.183772, 0.0427953, 0.0143271, 
Output 6: 0.0179329, 0.0494065, 0.0581709, 0.87449, 

10-Dec-2025 09:03:50 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:03:50 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/da/83/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T224713Z_gen_000636_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2035.28       5703.34       8330.11       6158.05       1692.56        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00825318     0.0211022     0.0304809     0.0211254    0.00626903       10     
reco:pandora:StandardPandora                            2034.12       5698.77       8311.43       6154.86       1689.53        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00158558     0.0644478      0.27948      0.0157766     0.0941859       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00101347    0.00983118     0.0303695    0.00767242    0.00885187       10     
reco:pandoracalo:Calorimetry                          0.000285192    0.0326056      0.15675     0.00211017     0.0525105       10     
reco:pandorapid:Chi2ParticleID                        0.000101452   0.000576795   0.00175711    0.000504498   0.000450648      10     
reco:cvnmap:CVNMapper                                  0.0419511     0.241573      0.443445      0.225785      0.108483        10     
reco:cvneva:CVNEvaluator                               0.739554       3.26753       16.2149       2.08198       4.34992        10     
reco:energyrecnumu:EnergyReco                         0.00650512     0.117496      0.289514      0.0846918     0.107967        10     
reco:energyrecnue:EnergyReco                          0.000914026    0.0111335     0.0271783    0.00734855    0.00879559       10     
reco:energyrecnc:EnergyReco                            0.0170142     0.134746      0.297943      0.126921      0.110151        10     
[art]:TriggerResults:TriggerResultInserter            2.0519e-05    0.000137221   0.000459884   9.7154e-05    0.000116932      10     
end_path:out1:RootOutput                               7.394e-06    4.73682e-05   0.000138634   3.58735e-05   3.59372e-05      10     
end_path:out1:RootOutput(write)                        0.230049      0.671865       1.39601      0.593677       0.30409        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21685.7 MB
  Peak resident set size usage (VmHWM): 2645.87 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 239.664 | hit correction : 1.04879
hit initial : 282.936 | hit correction : 1.04865
hit initial : 282.001 | hit correction : 1.0485
 *** Total Charge : 1816.59
 Looking at view: 1 | Number of hits in view: 11
hit initial : 26.6472 | hit correction : 1.04805
hit initial : 49.0231 | hit correction : 1.04895
hit initial : 147.271 | hit correction : 1.04887
hit initial : 352.763 | hit correction : 1.04875
hit initial : 648.144 | hit correction : 1.04846
hit initial : 324.645 | hit correction : 1.04828
 *** Total Charge : 1623.65
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 7th record. run: 5027 subRun: 0 event: 6357 at 10-Dec-2025 09:04:13 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 943.265
True Neutron Energy: 941.065
True Electron Energy: 4.70946
Completed MCParticle Loop
True Neutrino Energy: 17.298
 Size of all hits: 0
Begin processing the 8th record. run: 5027 subRun: 0 event: 6358 at 10-Dec-2025 09:04:15 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 26.8651
Completed MCParticle Loop
True Neutrino Energy: 48.8415
 Size of all hits: 0
Begin processing the 9th record. run: 5027 subRun: 0 event: 6359 at 10-Dec-2025 09:04:17 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 6.2408
Completed MCParticle Loop
True Neutrino Energy: 11.7446
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 6360 at 10-Dec-2025 09:04:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  32
Added hit list length: 32 | Max was set to: -3.40282e+38
True Neutron Energy: 939.572
True Electron Energy: 46.8416
Completed MCParticle Loop
True Neutrino Energy: 64.1535
 Size of all hits: 32
hit initial : 320.655 | hit correction : 1.05587
hit initial : 64.0885 | hit correction : 1.05577
hit initial : 236.562 | hit correction : 1.05581
hit initial : 371.422 | hit correction : 1.05588
hit initial : 283.804 | hit correction : 1.05584
hit initial : 214.493 | hit correction : 1.05573
hit initial : 270.448 | hit correction : 1.05554
hit initial : 322.935 | hit correction : 1.05526
hit initial : 286.778 | hit correction : 1.05497
hit initial : 241.504 | hit correction : 1.05466
hit initial : 252.084 | hit correction : 1.05444
hit initial : 229.24 | hit correction : 1.05418
hit initial : 238.136 | hit correction : 1.05395
hit initial : 212.468 | hit correction : 1.05368
hit initial : 113.838 | hit correction : 1.05352
 *** Total Charge : 3860.03
 Looking at view: 0 | Number of hits in view: 15
hit initial : 204.708 | hit correction : 1.0559
hit initial : 238.67 | hit correction : 1.0559
hit initial : 331.498 | hit correction : 1.05588
hit initial : 226.356 | hit correction : 1.0558
hit initial : 149.811 | hit correction : 1.05583
hit initial : 98.4884 | hit correction : 1.05577
hit initial : 159.937 | hit correction : 1.05577
hit initial : 208.601 | hit correction : 1.05564
hit initial : 243.929 | hit correction : 1.05543
hit initial : 277.787 | hit correction : 1.05514
hit initial : 206.079 | hit correction : 1.0549
hit initial : 257.559 | hit correction : 1.05465
hit initial : 199.879 | hit correction : 1.05444
hit initial : 204.808 | hit correction : 1.05422
hit initial : 244.009 | hit correction : 1.054
hit initial : 218.879 | hit correction : 1.05373
hit initial : 177.82 | hit correction : 1.05348
 *** Total Charge : 3849.8
 Looking at view: 2 | Number of hits in view: 17
10-Dec-2025 09:04:21 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0467017     0.0604236     0.0781409     0.0589366    0.00993761       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000403106   0.000619487   0.00153863    0.000492335   0.00033015       10     
end_path:ana:AnalyzeEvents         0.0460844     0.0596228     0.0760714     0.0583166    0.00962348       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.8 MB
  Peak resident set size usage (VmHWM): 1529.12 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:18:01 UTC       justIN version: 01.05.03