justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258278.0@dunegpschedd02.fnal.gov

Jobsub ID258278.0@dunegpschedd02.fnal.gov
Workflow ID11026
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 23:25:15
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 10:10:09
From worker nodeHostnamedunegli-7832544-0-fnpc7304.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:39:41
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015400Z_gen_003332_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (30221s)
CPU time8h (29765s = 98%)
Max RSS bytes3059716096 (2917 MiB)
Outputting started2025-12-06 10:03:23
Output files
Finished2025-12-06 10:10:09
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

9
Classifier summary: 
Output 0: 0.035888, 
Output 1: 0.0271287, 0.0386701, 0.0910137, 0.843188, 
Output 2: 0.0659286, 0.166271, 0.738276, 0.0295239, 
Output 3: 0.901495, 0.0646299, 0.0170624, 0.0168126, 
Output 4: 0.789019, 0.154369, 0.0413473, 0.0152652, 
Output 5: 0.851474, 0.128323, 0.0159736, 0.00422973, 
Output 6: 0.0233206, 0.0652072, 0.0614539, 0.850018, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 33320 at 06-Dec-2025 09:21:40 UTC
Boundary wire vector sizes: 6714, 6690, 6764
minwire 0: 36
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.170743, 
Output 1: 0.0820083, 0.144645, 0.222177, 0.55117, 
Output 2: 0.0411479, 0.16436, 0.766572, 0.0279194, 
Output 3: 0.966018, 0.0254416, 0.0034521, 0.00508853, 
Output 4: 0.486294, 0.456866, 0.0511161, 0.00572384, 
Output 5: 0.277364, 0.400589, 0.248929, 0.0731182, 
Output 6: 0.0302144, 0.0910959, 0.0973005, 0.781389, 

06-Dec-2025 10:01:46 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:01:46 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/17/88/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015400Z_gen_003332_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2399.68       2991.22       3348.65       2936.6        254.317        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0020527     0.0110906     0.0190586     0.0124472    0.00537459       10     
reco:pandora:StandardPandora                            2396.58       2987.54       3345.26       2933.37       254.339        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00234449     0.0481297     0.174558      0.0338735     0.0482276       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00194708    0.00941137     0.0258237    0.00706967     0.0078051       10     
reco:pandoracalo:Calorimetry                          0.000342244   0.00299096     0.0110863    0.00147382    0.00325283       10     
reco:pandorapid:Chi2ParticleID                        9.4893e-05    0.000561397   0.00265219    0.000253556   0.000745229      10     
reco:cvnmap:CVNMapper                                  0.0560842     0.0878181     0.142609      0.0837687     0.0237829       10     
reco:cvneva:CVNEvaluator                                1.35484       2.66036       5.8077        2.27277       1.1715         10     
reco:energyrecnumu:EnergyReco                          0.0130456     0.0777146     0.177649      0.0609262     0.0624356       10     
reco:energyrecnue:EnergyReco                          0.00220496    0.00566401     0.020299     0.00321465    0.00547123       10     
reco:energyrecnc:EnergyReco                            0.010959      0.0744604     0.177629      0.0569655     0.061973        10     
[art]:TriggerResults:TriggerResultInserter            2.3837e-05    5.15089e-05   0.000152401   3.53675e-05   3.63851e-05      10     
end_path:out1:RootOutput                               8.039e-06    1.63574e-05   4.5298e-05    1.01275e-05   1.1816e-05       10     
end_path:out1:RootOutput(write)                         0.59154      0.702705      0.915917      0.690675       0.08168        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9949.35 MB
  Peak resident set size usage (VmHWM): 3059.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
True Neutron Energy: 940.541
True Electron Energy: 45.1696
Completed MCParticle Loop
True Neutrino Energy: 62.4634
 Size of all hits: 71
hit initial : 323.014 | hit correction : 1.24412
hit initial : 201.154 | hit correction : 1.24404
hit initial : 28.1976 | hit correction : 1.24394
hit initial : 54.2854 | hit correction : 1.24242
hit initial : 42.001 | hit correction : 1.23143
hit initial : 174.774 | hit correction : 1.23152
hit initial : 105.004 | hit correction : 1.23141
hit initial : 75.1499 | hit correction : 1.23138
hit initial : 51.3496 | hit correction : 1.23145
hit initial : 523.598 | hit correction : 1.23179
hit initial : 82.1746 | hit correction : 1.23143
hit initial : 37.9533 | hit correction : 1.23135
hit initial : 278.563 | hit correction : 1.2327
hit initial : 444.838 | hit correction : 1.23327
hit initial : 189.903 | hit correction : 1.23348
hit initial : 29.7051 | hit correction : 1.24244
hit initial : 150.802 | hit correction : 1.2442
hit initial : 69.9955 | hit correction : 1.24544
hit initial : 244.967 | hit correction : 1.24542
hit initial : 218.916 | hit correction : 1.24536
hit initial : 25.7195 | hit correction : 1.24614
hit initial : 159.169 | hit correction : 1.24673
hit initial : 384.297 | hit correction : 1.24692
hit initial : 232.213 | hit correction : 1.24786
hit initial : 296.167 | hit correction : 1.2484
hit initial : 126.978 | hit correction : 1.24871
 *** Total Charge : 5642.96
 Looking at view: 0 | Number of hits in view: 26
hit initial : 58.2805 | hit correction : 1.24414
hit initial : 269.591 | hit correction : 1.24411
hit initial : 264.049 | hit correction : 1.2441
hit initial : 200.239 | hit correction : 1.23149
hit initial : 252.309 | hit correction : 1.23143
hit initial : 637.867 | hit correction : 1.2316
hit initial : 83.8528 | hit correction : 1.23159
hit initial : 175.186 | hit correction : 1.23197
hit initial : 25.1476 | hit correction : 1.23253
hit initial : 311.647 | hit correction : 1.23257
hit initial : 71.493 | hit correction : 1.23253
hit initial : 30.2581 | hit correction : 1.23254
hit initial : 471.709 | hit correction : 1.23285
hit initial : 438.779 | hit correction : 1.23328
hit initial : 220.565 | hit correction : 1.23353
hit initial : 42.0615 | hit correction : 1.23365
hit initial : 121.304 | hit correction : 1.24532
hit initial : 236.458 | hit correction : 1.24537
hit initial : 94.8686 | hit correction : 1.24546
hit initial : 46.0299 | hit correction : 1.24604
hit initial : 31.7052 | hit correction : 1.24608
hit initial : 252.379 | hit correction : 1.24674
hit initial : 191.703 | hit correction : 1.24754
hit initial : 273.884 | hit correction : 1.24806
hit initial : 198.611 | hit correction : 1.24854
hit initial : 94.2223 | hit correction : 1.24877
 *** Total Charge : 6307.41
 Looking at view: 1 | Number of hits in view: 26
hit initial : 227.38 | hit correction : 1.24426
hit initial : 268.988 | hit correction : 1.23137
hit initial : 160.291 | hit correction : 1.2314
hit initial : 228.203 | hit correction : 1.23147
hit initial : 332.148 | hit correction : 1.23206
hit initial : 788.764 | hit correction : 1.23212
hit initial : 518.808 | hit correction : 1.23292
hit initial : 538.184 | hit correction : 1.23343
hit initial : 44.4609 | hit correction : 1.23363
hit initial : 142.14 | hit correction : 1.24529
hit initial : 15.036 | hit correction : 1.24537
hit initial : 190.589 | hit correction : 1.24532
hit initial : 200.82 | hit correction : 1.24556
hit initial : 156.925 | hit correction : 1.24596
hit initial : 45.0343 | hit correction : 1.2466
hit initial : 585.867 | hit correction : 1.24727
hit initial : 183.249 | hit correction : 1.2477
hit initial : 246.551 | hit correction : 1.24855
hit initial : 75.6812 | hit correction : 1.24874
 *** Total Charge : 6128.52
 Looking at view: 2 | Number of hits in view: 19
06-Dec-2025 10:03:22 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.148293       0.20246      0.244164       0.20724      0.0329588       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000584458   0.00322899     0.0144946    0.000726414   0.00505653       10     
end_path:ana:AnalyzeEvents         0.147306      0.198992      0.231891       0.20638      0.0299867       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7665.08 MB
  Peak resident set size usage (VmHWM): 1539.38 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:09:27 UTC       justIN version: 01.05.03