justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267038.2@dunegpschedd01.fnal.gov

Jobsub ID267038.2@dunegpschedd01.fnal.gov
Workflow ID11158
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 21:52:40
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-11 23:44:14
From worker nodeHostnamedunegli-7878628-0-fnpc23034.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-11 21:54:31
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T010325Z_gen_002955_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time1h (6570s)
CPU time1h (6514s = 99%)
Max RSS bytes2857746432 (2725 MiB)
Outputting started2025-12-11 23:44:01
Output files
Finished2025-12-11 23:44:14
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ularShowerCreation:pandoraShower@BeginModule  11-Dec-2025 23:43:29 UTC run: 5027 subRun: 0 event: 29550
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  11-Dec-2025 23:43:29 UTC run: 5027 subRun: 0 event: 29550
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6970, 6915, 6741
minwire 0: 39
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2300, 2799
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.108973, 
Output 1: 0.0638639, 0.112888, 0.155142, 0.668106, 
Output 2: 0.0489679, 0.106066, 0.832758, 0.0122081, 
Output 3: 0.953807, 0.0424327, 0.00158841, 0.00217143, 
Output 4: 0.406292, 0.516122, 0.0668217, 0.0107645, 
Output 5: 0.189188, 0.468095, 0.260721, 0.0819956, 
Output 6: 0.0912374, 0.1846, 0.136278, 0.587884, 

11-Dec-2025 23:43:31 UTC  Closed output file "matt_reco2.root"
11-Dec-2025 23:43:31 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/75/8d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T010325Z_gen_002955_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              641.095       647.852       658.094       646.734       5.48488        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000510946    0.0036309     0.0169671    0.000695398   0.00600991       10     
reco:pandora:StandardPandora                            639.894       646.663       657.058       645.135       5.53723        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00103387     0.0162945     0.0308021     0.0195783     0.0121176       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00116556    0.00377181     0.0103069    0.00357262    0.00241465       10     
reco:pandoracalo:Calorimetry                          0.000211193    0.0028139     0.0215703    0.000332395   0.00628626       10     
reco:pandorapid:Chi2ParticleID                        5.0397e-05    0.000189693   0.000801099   7.6486e-05    0.000231999      10     
reco:cvnmap:CVNMapper                                  0.036227      0.0422061     0.0458697     0.0429951    0.00273733       10     
reco:cvneva:CVNEvaluator                                0.70429      0.823573       1.73379      0.722206      0.303691        10     
reco:energyrecnumu:EnergyReco                          0.0536861     0.0656945     0.0960382     0.0610916     0.0130999       10     
reco:energyrecnue:EnergyReco                          0.000915237   0.00156694    0.00536537    0.00103242    0.00129069       10     
reco:energyrecnc:EnergyReco                           0.00514217     0.0056654    0.00898133     0.0053299    0.00110722       10     
[art]:TriggerResults:TriggerResultInserter             7.494e-06    1.41511e-05   3.5638e-05    1.0255e-05    8.00621e-06      10     
end_path:out1:RootOutput                               4.94e-06     9.5453e-06    1.6391e-05    6.4925e-06    4.71762e-06      10     
end_path:out1:RootOutput(write)                        0.201685      0.222338      0.338815      0.209632      0.039403        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21584.2 MB
  Peak resident set size usage (VmHWM): 2857.75 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  6
This pfp has size:  63
Added hit list length: 63 | Max was set to: -3.40282e+38
This pfp has size:  10
This pfp has size:  8
True Neutron Energy: 939.572
True Neutron Energy: 941.61
True Neutron Energy: 940.129
True Electron Energy: 46.5584
Completed MCParticle Loop
True Neutrino Energy: 62.1416
 Size of all hits: 63
hit initial : 190.045 | hit correction : 1.20866
hit initial : 447.55 | hit correction : 1.20865
hit initial : 344.417 | hit correction : 1.20849
hit initial : 291.609 | hit correction : 1.20799
hit initial : 360.811 | hit correction : 1.20759
hit initial : 357.749 | hit correction : 1.20722
hit initial : 314.913 | hit correction : 1.20708
hit initial : 156.637 | hit correction : 1.20703
hit initial : 172.757 | hit correction : 1.20329
hit initial : 98.1773 | hit correction : 1.20514
hit initial : 40.7119 | hit correction : 1.20324
hit initial : 225.13 | hit correction : 1.20551
hit initial : 251.094 | hit correction : 1.20435
hit initial : 243.266 | hit correction : 1.20335
hit initial : 303.448 | hit correction : 1.20519
hit initial : 284.993 | hit correction : 1.20383
hit initial : 208.313 | hit correction : 1.20582
hit initial : 229.256 | hit correction : 1.20602
hit initial : 235.294 | hit correction : 1.2061
 *** Total Charge : 5737.88
 Looking at view: 0 | Number of hits in view: 19
hit initial : 62.1221 | hit correction : 1.20864
hit initial : 364.905 | hit correction : 1.2086
hit initial : 44.026 | hit correction : 1.20867
hit initial : 375.455 | hit correction : 1.20866
hit initial : 288.145 | hit correction : 1.20805
hit initial : 127.131 | hit correction : 1.20724
hit initial : 441.916 | hit correction : 1.20723
hit initial : 394.348 | hit correction : 1.20711
hit initial : 76.9267 | hit correction : 1.20706
hit initial : 241.077 | hit correction : 1.20332
hit initial : 53.5367 | hit correction : 1.20328
hit initial : 37.5185 | hit correction : 1.20511
hit initial : 485.184 | hit correction : 1.20509
hit initial : 185.028 | hit correction : 1.20342
hit initial : 103.694 | hit correction : 1.20428
hit initial : 60.667 | hit correction : 1.20422
hit initial : 243.566 | hit correction : 1.20589
hit initial : 421.489 | hit correction : 1.20597
 *** Total Charge : 4834.09
 Looking at view: 1 | Number of hits in view: 18
hit initial : 118.545 | hit correction : 1.20606
hit initial : 127.096 | hit correction : 1.20857
hit initial : 324.169 | hit correction : 1.20867
hit initial : 242.332 | hit correction : 1.2087
hit initial : 154.665 | hit correction : 1.20855
hit initial : 158.808 | hit correction : 1.20826
hit initial : 129.007 | hit correction : 1.20807
hit initial : 122.801 | hit correction : 1.20795
hit initial : 133.513 | hit correction : 1.20779
hit initial : 131.699 | hit correction : 1.20759
hit initial : 214.479 | hit correction : 1.20731
hit initial : 200.392 | hit correction : 1.20722
hit initial : 124.754 | hit correction : 1.2071
hit initial : 219.522 | hit correction : 1.20704
hit initial : 190.873 | hit correction : 1.20699
hit initial : 207.504 | hit correction : 1.20327
hit initial : 470.827 | hit correction : 1.20492
hit initial : 188.093 | hit correction : 1.20539
hit initial : 328.629 | hit correction : 1.2034
hit initial : 317.969 | hit correction : 1.20403
hit initial : 153.49 | hit correction : 1.20563
hit initial : 155.761 | hit correction : 1.20584
hit initial : 132.222 | hit correction : 1.20597
hit initial : 128.56 | hit correction : 1.20608
hit initial : 113.156 | hit correction : 1.20613
hit initial : 121.346 | hit correction : 1.20613
 *** Total Charge : 5923.64
 Looking at view: 2 | Number of hits in view: 26
11-Dec-2025 23:44:01 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0480647     0.0590286     0.0803355     0.0575767     0.0077827       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000300514   0.000380155   0.000611107   0.000351897   9.03052e-05      10     
end_path:ana:AnalyzeEvents         0.0476428     0.0584936     0.0792871     0.0570904    0.00761591       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19480.2 MB
  Peak resident set size usage (VmHWM): 1540.78 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 04:23:35 UTC       justIN version: 01.05.03