justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258267.1@dunegpschedd02.fnal.gov

Jobsub ID258267.1@dunegpschedd02.fnal.gov
Workflow ID11022
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 23:09:14
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:24:58
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:09
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T155513Z_gen_000077_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (31153s)
CPU time8h (30441s = 97%)
Max RSS bytes3063918592 (2921 MiB)
Outputting started2025-12-06 10:17:22
Output files
Finished2025-12-06 10:24:58
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

put 0: 0.0928447, 
Output 1: 0.0645661, 0.058926, 0.0936477, 0.78286, 
Output 2: 0.0424206, 0.290282, 0.642353, 0.0249447, 
Output 3: 0.946819, 0.0381817, 0.00639666, 0.00860225, 
Output 4: 0.573969, 0.348353, 0.0693413, 0.00833701, 
Output 5: 0.593684, 0.340476, 0.0580474, 0.00779236, 
Output 6: 0.0115873, 0.0406608, 0.0609226, 0.886829, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 770 at 06-Dec-2025 09:24:40 UTC
Boundary wire vector sizes: 6988, 6898, 6738
minwire 0: 23
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.096397, 
Output 1: 0.199418, 0.0551148, 0.224157, 0.52131, 
Output 2: 0.031231, 0.11517, 0.838774, 0.0148258, 
Output 3: 0.910441, 0.0672229, 0.0108847, 0.0114516, 
Output 4: 0.425922, 0.490182, 0.0726202, 0.0112763, 
Output 5: 0.335621, 0.440344, 0.17974, 0.0442944, 
Output 6: 0.072275, 0.150585, 0.11547, 0.66167, 

06-Dec-2025 10:15:24 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:15:24 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/72/71/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T155513Z_gen_000077_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2912.11       3083.99       3339.84       3045.78       137.952        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00333872    0.00942639     0.0294379    0.00720754    0.00700288       10     
reco:pandora:StandardPandora                            2907.95       3078.92       3335.45       3038.45       137.815        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00487754     0.0527159     0.202437      0.0456832     0.0537886       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00151947    0.00807769     0.0444254    0.00353284     0.0123878       10     
reco:pandoracalo:Calorimetry                          0.000658242   0.00822345     0.0542143    0.000958732    0.0158559       10     
reco:pandorapid:Chi2ParticleID                        0.000176803   0.000707135   0.00460572    0.000256023   0.00130148       10     
reco:cvnmap:CVNMapper                                  0.0491138     0.0931168     0.242833      0.0608881     0.0633494       10     
reco:cvneva:CVNEvaluator                                3.08417       3.99463       8.90509       3.31763       1.6964         10     
reco:energyrecnumu:EnergyReco                          0.0197995     0.112342      0.186285      0.141137      0.0606991       10     
reco:energyrecnue:EnergyReco                          0.00220605    0.00713215     0.0175676    0.00307646    0.00611699       10     
reco:energyrecnc:EnergyReco                            0.0148891     0.0555302     0.155998      0.0167014     0.0603206       10     
[art]:TriggerResults:TriggerResultInserter            3.2581e-05    6.03768e-05   0.000186732   4.03115e-05   4.52401e-05      10     
end_path:out1:RootOutput                              1.2073e-05    1.86222e-05   5.5164e-05    1.4192e-05    1.23296e-05      10     
end_path:out1:RootOutput(write)                        0.637307      0.728012       1.06219      0.664509      0.137507        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21795.9 MB
  Peak resident set size usage (VmHWM): 3063.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 59.9639 | hit correction : 1.20923
hit initial : 189.547 | hit correction : 1.20865
hit initial : 75.3035 | hit correction : 1.20866
hit initial : 297.821 | hit correction : 1.23218
hit initial : 94.8509 | hit correction : 1.23222
hit initial : 119.749 | hit correction : 1.23224
hit initial : 151.261 | hit correction : 1.23414
hit initial : 152.458 | hit correction : 1.23416
hit initial : 125.825 | hit correction : 1.23426
hit initial : 153.9 | hit correction : 1.23441
hit initial : 132.622 | hit correction : 1.23441
hit initial : 334.729 | hit correction : 1.23446
hit initial : 23.7399 | hit correction : 1.2346
hit initial : 173.715 | hit correction : 1.23471
hit initial : 31.7497 | hit correction : 1.2348
hit initial : 116.065 | hit correction : 1.23491
 *** Total Charge : 2956.57
 Looking at view: 2 | Number of hits in view: 18
Begin processing the 10th record. run: 5026 subRun: 0 event: 770 at 06-Dec-2025 10:17:15 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  48
Added hit list length: 48 | Max was set to: -3.40282e+38
True Neutron Energy: 939.574
True Electron Energy: 24.8061
Completed MCParticle Loop
True Neutrino Energy: 30.2147
 Size of all hits: 48
hit initial : 83.7234 | hit correction : 1.37481
hit initial : 193.942 | hit correction : 1.37481
hit initial : 207.066 | hit correction : 1.37486
hit initial : 136.149 | hit correction : 1.37516
hit initial : 88.9847 | hit correction : 1.37618
hit initial : 51.3497 | hit correction : 1.37617
hit initial : 110.296 | hit correction : 1.37624
hit initial : 55.9778 | hit correction : 1.37623
hit initial : 337.411 | hit correction : 1.37704
hit initial : 37.7355 | hit correction : 1.37731
hit initial : 37.8784 | hit correction : 1.37725
hit initial : 95.5721 | hit correction : 1.37806
hit initial : 77.9204 | hit correction : 1.37824
hit initial : 116.899 | hit correction : 1.37949
hit initial : 111.778 | hit correction : 1.37929
hit initial : 184.905 | hit correction : 1.3795
hit initial : 35.7505 | hit correction : 1.37949
 *** Total Charge : 2703.31
 Looking at view: 0 | Number of hits in view: 17
hit initial : 146.353 | hit correction : 1.37808
hit initial : 7.13213 | hit correction : 1.37862
hit initial : 86.7358 | hit correction : 1.3748
hit initial : 214.809 | hit correction : 1.3748
hit initial : 215.398 | hit correction : 1.3749
hit initial : 126.147 | hit correction : 1.37622
hit initial : 109.438 | hit correction : 1.37629
hit initial : 369.083 | hit correction : 1.37729
hit initial : 76.0836 | hit correction : 1.37719
hit initial : 39.3602 | hit correction : 1.3773
hit initial : 73.7322 | hit correction : 1.37808
hit initial : 36.2119 | hit correction : 1.37799
hit initial : 120.839 | hit correction : 1.37941
hit initial : 249.265 | hit correction : 1.37947
hit initial : 119.98 | hit correction : 1.37949
 *** Total Charge : 2741.34
 Looking at view: 1 | Number of hits in view: 15
hit initial : 29.6304 | hit correction : 1.37945
hit initial : 92.3939 | hit correction : 1.3795
hit initial : 41.5223 | hit correction : 1.37777
hit initial : 28.3539 | hit correction : 1.37375
hit initial : 28.2689 | hit correction : 1.37489
hit initial : 443.94 | hit correction : 1.3749
hit initial : 233.228 | hit correction : 1.37487
hit initial : 27.6516 | hit correction : 1.37509
hit initial : 370.29 | hit correction : 1.37621
hit initial : 139.314 | hit correction : 1.37632
hit initial : 184.977 | hit correction : 1.37731
hit initial : 90.6573 | hit correction : 1.37723
hit initial : 148.793 | hit correction : 1.37802
hit initial : 300.175 | hit correction : 1.37846
hit initial : 285.036 | hit correction : 1.37944
hit initial : 58.0247 | hit correction : 1.3799
 *** Total Charge : 3445.45
 Looking at view: 2 | Number of hits in view: 16
06-Dec-2025 10:17:21 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.141149      0.330109       1.22789      0.189381      0.312478        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000623255    0.0282706     0.171567     0.000831638    0.0569685       10     
end_path:ana:AnalyzeEvents         0.136438      0.301506        1.122        0.16561      0.287778        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19482.7 MB
  Peak resident set size usage (VmHWM): 1546.93 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:09:02 UTC       justIN version: 01.06.00