justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258024.0@dunegpschedd02.fnal.gov

Jobsub ID258024.0@dunegpschedd02.fnal.gov
Workflow ID11012
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 15:05:50
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 09:54:16
From worker nodeHostnamedunegli-7832153-0-fnpc18009.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:30:58
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T022010Z_gen_003946_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (29790s)
CPU time8h (29222s = 98%)
Max RSS bytes3079909376 (2937 MiB)
Outputting started2025-12-06 09:47:28
Output files
Finished2025-12-06 09:54:16
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

, 0.74168, 0.0227471, 
Output 3: 0.955595, 0.0325489, 0.00453961, 0.00731675, 
Output 4: 0.771673, 0.199067, 0.0234995, 0.00576022, 
Output 5: 0.547219, 0.393059, 0.0496546, 0.0100676, 
Output 6: 0.0315489, 0.0920653, 0.0805137, 0.795872, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 39460 at 06-Dec-2025 09:15:13 UTC
Boundary wire vector sizes: 6798, 6688, 6627
minwire 0: 38
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 669, 1168
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0991695, 
Output 1: 0.151432, 0.0838364, 0.203159, 0.561572, 
Output 2: 0.0436793, 0.0848826, 0.860831, 0.010607, 
Output 3: 0.975956, 0.021245, 0.000963832, 0.00183481, 
Output 4: 0.5053, 0.444866, 0.0423004, 0.00753369, 
Output 5: 0.348057, 0.43359, 0.157997, 0.0603566, 
Output 6: 0.076437, 0.182437, 0.120384, 0.620742, 

06-Dec-2025 09:46:26 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:46:26 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/97/ac/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T022010Z_gen_003946_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1833.43       2948.28       4807.21       3011.8        1005.64        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00541566     0.0663696     0.556957     0.00953664      0.16365        10     
reco:pandora:StandardPandora                            1831.06       2931.86       4733.17       3009.08       981.654        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000516555    0.0438246     0.240375     0.00553027     0.0713546       10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0004871    0.00933416     0.0635304     0.0034956     0.0182243       10     
reco:pandoracalo:Calorimetry                          0.000215427    0.010998      0.0822456    0.000508435    0.0242121       10     
reco:pandorapid:Chi2ParticleID                        9.4859e-05    0.000655861   0.00500884    0.000140265   0.00145503       10     
reco:cvnmap:CVNMapper                                  0.0365148     0.0617004     0.102882      0.062584      0.0201428       10     
reco:cvneva:CVNEvaluator                                1.82987       2.77172       9.18229       1.96695       2.1642         10     
reco:energyrecnumu:EnergyReco                         0.00473354      12.9441       65.3373      0.117557       25.7342        10     
reco:energyrecnue:EnergyReco                          0.00161368     0.0067662     0.0105047    0.00896268     0.003848        10     
reco:energyrecnc:EnergyReco                           0.00969717     0.0404628     0.121069      0.0133646     0.0426587       10     
[art]:TriggerResults:TriggerResultInserter            2.2252e-05    5.41362e-05   0.00023897     3.702e-05    6.20587e-05      10     
end_path:out1:RootOutput                               9.178e-06    1.67865e-05   4.5435e-05    1.4572e-05    9.91146e-06      10     
end_path:out1:RootOutput(write)                        0.373702      0.463811      0.899353      0.401814      0.158621        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21781.1 MB
  Peak resident set size usage (VmHWM): 3079.91 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 *** Total Charge : 2400.37
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 10th record. run: 5027 subRun: 0 event: 39460 at 06-Dec-2025 09:47:23 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  64
Added hit list length: 64 | Max was set to: -3.40282e+38
True Neutron Energy: 939.571
True Electron Energy: 39.8336
Completed MCParticle Loop
True Neutrino Energy: 59.3832
 Size of all hits: 64
hit initial : 36.6461 | hit correction : 1.01518
hit initial : 29.7305 | hit correction : 1.01521
hit initial : 112.028 | hit correction : 1.03156
hit initial : 149.708 | hit correction : 1.03157
hit initial : 87.0524 | hit correction : 1.03779
hit initial : 177.175 | hit correction : 1.03776
hit initial : 49.6484 | hit correction : 1.02102
hit initial : 130.541 | hit correction : 1.02101
hit initial : 55.376 | hit correction : 1.02102
hit initial : 164.71 | hit correction : 1.02046
hit initial : 253.348 | hit correction : 1.02034
hit initial : 96.0355 | hit correction : 1.02014
hit initial : 45.9786 | hit correction : 1.01963
hit initial : 297.364 | hit correction : 1.0194
hit initial : 132.773 | hit correction : 1.01918
hit initial : 457.916 | hit correction : 1.01905
hit initial : 577.235 | hit correction : 1.01842
hit initial : 743.802 | hit correction : 1.01816
hit initial : 77.0568 | hit correction : 1.01813
hit initial : 140.148 | hit correction : 1.0172
hit initial : 123.109 | hit correction : 1.01695
hit initial : 166.147 | hit correction : 1.01669
hit initial : 411.635 | hit correction : 1.01671
hit initial : 71.0092 | hit correction : 1.01656
 *** Total Charge : 4679.63
 Looking at view: 0 | Number of hits in view: 24
hit initial : 134.649 | hit correction : 1.021
hit initial : 229.876 | hit correction : 1.02042
hit initial : 272.268 | hit correction : 1.0201
hit initial : 104.511 | hit correction : 1.02045
hit initial : 81.961 | hit correction : 1.01981
hit initial : 109.788 | hit correction : 1.01941
hit initial : 171.003 | hit correction : 1.01926
hit initial : 249.922 | hit correction : 1.0191
hit initial : 256.506 | hit correction : 1.01884
hit initial : 249.234 | hit correction : 1.01858
hit initial : 281.709 | hit correction : 1.01839
hit initial : 395.936 | hit correction : 1.01823
hit initial : 382.209 | hit correction : 1.0181
hit initial : 276.035 | hit correction : 1.01788
hit initial : 253.233 | hit correction : 1.0175
hit initial : 290.46 | hit correction : 1.01719
hit initial : 340.952 | hit correction : 1.0168
hit initial : 287.793 | hit correction : 1.0166
hit initial : 93.2112 | hit correction : 1.01654
 *** Total Charge : 4543.4
 Looking at view: 1 | Number of hits in view: 19
hit initial : 259.383 | hit correction : 1.03156
hit initial : 69.2627 | hit correction : 1.03152
hit initial : 382.895 | hit correction : 1.01671
hit initial : 221.575 | hit correction : 1.01651
hit initial : 26.087 | hit correction : 1.01616
hit initial : 212.713 | hit correction : 1.02102
hit initial : 28.7874 | hit correction : 1.02088
hit initial : 17.1241 | hit correction : 1.02038
hit initial : 706.534 | hit correction : 1.02031
hit initial : 37.0261 | hit correction : 1.01971
hit initial : 115.739 | hit correction : 1.01957
hit initial : 506.438 | hit correction : 1.01925
hit initial : 153.476 | hit correction : 1.01901
hit initial : 257.294 | hit correction : 1.0188
hit initial : 327.187 | hit correction : 1.01852
hit initial : 406.017 | hit correction : 1.01832
hit initial : 477.378 | hit correction : 1.01813
hit initial : 290.197 | hit correction : 1.01792
hit initial : 215.442 | hit correction : 1.01761
hit initial : 261.644 | hit correction : 1.01731
hit initial : 238.565 | hit correction : 1.01703
 *** Total Charge : 5311.62
 Looking at view: 2 | Number of hits in view: 21
06-Dec-2025 09:47:27 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0893184     0.108318      0.158851      0.105082      0.0187955       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000475507   0.000709228   0.00140891    0.000651739   0.000259514      10     
end_path:ana:AnalyzeEvents         0.0882414     0.107358      0.156798      0.104315      0.0184943       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19474.8 MB
  Peak resident set size usage (VmHWM): 1536.63 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:09:30 UTC       justIN version: 01.05.03