justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266826.0@dunegpschedd01.fnal.gov

Jobsub ID266826.0@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 14:46:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-11 22:57:30
From worker nodeHostnamedunegli-8029038-0-fnpc8027.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-11 14:47:24
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T230735Z_gen_001191_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (29372s)
CPU time8h (29051s = 98%)
Max RSS bytes2909347840 (2774 MiB)
Outputting started2025-12-11 22:56:57
Output files
Finished2025-12-11 22:57:24
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0835042, 0.0513842, 0.824432, 
Output 2: 0.118731, 0.401161, 0.455109, 0.0249997, 
Output 3: 0.978066, 0.0153632, 0.000928544, 0.00564273, 
Output 4: 0.563648, 0.404505, 0.0292175, 0.00262974, 
Output 5: 0.29887, 0.663792, 0.0347523, 0.00258621, 
Output 6: 0.00629003, 0.0211863, 0.045583, 0.926941, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 11910 at 11-Dec-2025 22:17:21 UTC
Boundary wire vector sizes: 6657, 6661, 6610
minwire 0: 66
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 549, 1048
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.136363, 
Output 1: 0.0610421, 0.0524409, 0.163222, 0.723295, 
Output 2: 0.0622546, 0.184461, 0.718377, 0.0349074, 
Output 3: 0.968927, 0.0191092, 0.00335365, 0.00861035, 
Output 4: 0.279115, 0.656768, 0.0596552, 0.00446159, 
Output 5: 0.493902, 0.371696, 0.106014, 0.0283883, 
Output 6: 0.0218102, 0.0603559, 0.072256, 0.845578, 

11-Dec-2025 22:55:05 UTC  Closed output file "matt_reco2.root"
11-Dec-2025 22:55:05 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/94/cb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T230735Z_gen_001191_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2256.23       2901.46       3498.98       2943.26       377.122        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00385003     0.0135982     0.0301297     0.0134384    0.00655297       10     
reco:pandora:StandardPandora                            2253.78       2898.02       3489.67       2939.97       376.013        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000685663    0.0614649     0.343347      0.0122042     0.0988317       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000696179    0.0114999     0.0390756    0.00687154     0.0112278       10     
reco:pandoracalo:Calorimetry                          0.000472504    0.0197397     0.0741857    0.00285739     0.0279177       10     
reco:pandorapid:Chi2ParticleID                        0.000102632   0.000522701   0.00208883    0.000146578   0.00070402       10     
reco:cvnmap:CVNMapper                                  0.0676097     0.0911628     0.121775      0.0884607     0.0158871       10     
reco:cvneva:CVNEvaluator                                1.36514       2.46239       7.88365       1.90487       1.8557         10     
reco:energyrecnumu:EnergyReco                         0.00686944     0.101285      0.185256      0.119075      0.0623144       10     
reco:energyrecnue:EnergyReco                          0.00285276     0.0120659     0.0367764     0.0113066    0.00998122       10     
reco:energyrecnc:EnergyReco                            0.0114436     0.0494257     0.150524      0.0174974     0.0531932       10     
[art]:TriggerResults:TriggerResultInserter            2.5136e-05    4.81904e-05   0.000134647   3.69445e-05   3.04671e-05      10     
end_path:out1:RootOutput                               9.025e-06    1.68462e-05   4.1797e-05    1.4193e-05    9.31359e-06      10     
end_path:out1:RootOutput(write)                        0.414752      0.611507      0.860105      0.644972      0.134401        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10937.9 MB
  Peak resident set size usage (VmHWM): 2909.35 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 51.317 | hit correction : 1.41094
hit initial : 146.705 | hit correction : 1.4124
hit initial : 42.2308 | hit correction : 1.41257
hit initial : 201.817 | hit correction : 1.41207
hit initial : 194.342 | hit correction : 1.41166
hit initial : 244.21 | hit correction : 1.41128
hit initial : 300.689 | hit correction : 1.41104
hit initial : 221.71 | hit correction : 1.41096
 *** Total Charge : 1980.35
 Looking at view: 1 | Number of hits in view: 8
hit initial : 27.9018 | hit correction : 1.41101
hit initial : 145.53 | hit correction : 1.41249
hit initial : 28.4001 | hit correction : 1.41261
hit initial : 194.308 | hit correction : 1.4121
hit initial : 155.738 | hit correction : 1.41172
hit initial : 139.904 | hit correction : 1.41144
hit initial : 142.686 | hit correction : 1.41124
hit initial : 147.929 | hit correction : 1.41107
hit initial : 201.966 | hit correction : 1.41097
hit initial : 192.11 | hit correction : 1.41093
 *** Total Charge : 1942.88
 Looking at view: 2 | Number of hits in view: 10
Begin processing the 8th record. run: 5027 subRun: 0 event: 11908 at 11-Dec-2025 22:56:34 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.711
True Electron Energy: 19.3731
Completed MCParticle Loop
True Neutrino Energy: 28.3277
 Size of all hits: 0
Begin processing the 9th record. run: 5027 subRun: 0 event: 11909 at 11-Dec-2025 22:56:41 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  29
Added hit list length: 29 | Max was set to: -3.40282e+38
True Electron Energy: 21.124
Completed MCParticle Loop
True Neutrino Energy: 24.4313
 Size of all hits: 29
hit initial : 88.8786 | hit correction : 1.15127
hit initial : 376.366 | hit correction : 1.15129
hit initial : 460.509 | hit correction : 1.1513
hit initial : 287.403 | hit correction : 1.15127
hit initial : 205.238 | hit correction : 1.15117
hit initial : 200.831 | hit correction : 1.15109
hit initial : 217.381 | hit correction : 1.15104
hit initial : 291.709 | hit correction : 1.15104
hit initial : 269.73 | hit correction : 1.1511
hit initial : 134.538 | hit correction : 1.15119
hit initial : 27.8458 | hit correction : 1.15126
 *** Total Charge : 2947.54
 Looking at view: 0 | Number of hits in view: 11
hit initial : 50.7104 | hit correction : 1.15126
hit initial : 157.069 | hit correction : 1.15119
hit initial : 237.058 | hit correction : 1.15109
hit initial : 266.225 | hit correction : 1.15103
hit initial : 255.32 | hit correction : 1.15104
hit initial : 280.881 | hit correction : 1.15112
hit initial : 260.949 | hit correction : 1.15124
hit initial : 205.285 | hit correction : 1.15134
hit initial : 233.996 | hit correction : 1.15131
hit initial : 324.86 | hit correction : 1.15127
hit initial : 222.557 | hit correction : 1.15126
hit initial : 46.9913 | hit correction : 1.15127
 *** Total Charge : 2926.21
 Looking at view: 1 | Number of hits in view: 12
hit initial : 10.4508 | hit correction : 1.15146
hit initial : 507.684 | hit correction : 1.1513
hit initial : 581.772 | hit correction : 1.1511
hit initial : 1052.17 | hit correction : 1.15114
hit initial : 142.181 | hit correction : 1.15125
hit initial : 273.766 | hit correction : 1.15127
 *** Total Charge : 2956.26
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 10th record. run: 5027 subRun: 0 event: 11910 at 11-Dec-2025 22:56:49 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.569
True Electron Energy: 11.3834
Completed MCParticle Loop
True Neutrino Energy: 16.8942
 Size of all hits: 0
11-Dec-2025 22:56:56 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.156589      0.208254      0.307797       0.18296      0.0472029       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000828245   0.00183662    0.00958304    0.000987419    0.0025848       10     
end_path:ana:AnalyzeEvents         0.155552      0.206128      0.306635      0.181624      0.0459045       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8775.42 MB
  Peak resident set size usage (VmHWM): 1567.14 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 14:58:55 UTC       justIN version: 01.06.00