justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267094.0@dunegpschedd01.fnal.gov

Jobsub ID267094.0@dunegpschedd01.fnal.gov
Workflow ID11113
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 23:04:41
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-12 03:08:44
From worker nodeHostnamedunegli-7875989-0-fnpc19118.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-11 23:05:34
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T180452Z_gen_001363_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time4h (14575s)
CPU time4h (14457s = 99%)
Max RSS bytes2912022528 (2777 MiB)
Outputting started2025-12-12 03:08:30
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11113/1/001/analysisLowE_20251211T230535Z.root
Finished2025-12-12 03:08:44
Saved logsjustin-logs:267094.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t 1: 0.0777581, 0.0760185, 0.147437, 0.698786, 
Output 2: 0.0377786, 0.143586, 0.792649, 0.0259867, 
Output 3: 0.979883, 0.00821594, 0.00190696, 0.00999439, 
Output 4: 0.401323, 0.525719, 0.0646807, 0.00827677, 
Output 5: 0.252611, 0.403652, 0.259197, 0.0845406, 
Output 6: 0.033847, 0.0761913, 0.0865142, 0.803448, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 13630 at 12-Dec-2025 02:43:08 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
Boundary wire vector sizes: 6805, 6672, 6702
minwire 0: 23
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.107635, 
Output 1: 0.028046, 0.0361941, 0.160919, 0.774841, 
Output 2: 0.312082, 0.163282, 0.477508, 0.0471278, 
Output 3: 0.97131, 0.0250182, 0.0010227, 0.00264947, 
Output 4: 0.0792782, 0.88709, 0.0323064, 0.00132508, 
Output 5: 0.789763, 0.19596, 0.0103114, 0.00396527, 
Output 6: 0.0312061, 0.0942944, 0.0876454, 0.786854, 

12-Dec-2025 03:07:47 UTC  Closed output file "matt_reco2.root"
12-Dec-2025 03:07:47 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/01/6c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T180452Z_gen_001363_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1407.19       1444.47       1475.83       1442.04       20.5025        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00105954     0.0122677     0.0788991    0.00328699     0.0228247       10     
reco:pandora:StandardPandora                            1405.8        1442.96       1474.49       1440.78       20.5637        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0018701     0.0139727     0.0614686    0.00402471     0.0181148       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00157927    0.00395216     0.0122528     0.003219     0.00294462       10     
reco:pandoracalo:Calorimetry                          0.000274932    0.0092556     0.0245085    0.00640694    0.00946891       10     
reco:pandorapid:Chi2ParticleID                        5.5951e-05    0.000214726   0.000879556   9.39305e-05   0.000240655      10     
reco:cvnmap:CVNMapper                                  0.0232498     0.0322027     0.0434149     0.0336952    0.00666883       10     
reco:cvneva:CVNEvaluator                               0.822414       1.02869       2.69491      0.839048      0.555734        10     
reco:energyrecnumu:EnergyReco                         0.00466435     0.0604711     0.0915141     0.0715848     0.0305257       10     
reco:energyrecnue:EnergyReco                          0.000855936   0.00467667    0.00746619    0.00673453    0.00301154       10     
reco:energyrecnc:EnergyReco                           0.00655093     0.0321456     0.130852     0.00710817     0.0415089       10     
[art]:TriggerResults:TriggerResultInserter             1.148e-05    2.0277e-05     7.151e-05    1.5585e-05    1.71897e-05      10     
end_path:out1:RootOutput                               5.56e-06      8.235e-06     2.351e-05     6.425e-06    5.12035e-06      10     
end_path:out1:RootOutput(write)                        0.289026       0.31161      0.375578      0.297382      0.0315056       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21678.1 MB
  Peak resident set size usage (VmHWM): 2912.02 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 26.3262 | hit correction : 1.05899
hit initial : 71.2879 | hit correction : 1.0592
hit initial : 39.6203 | hit correction : 1.05811
hit initial : 13.82 | hit correction : 1.05809
hit initial : 13.2999 | hit correction : 1.05869
hit initial : 53.8827 | hit correction : 1.05868
hit initial : 107.318 | hit correction : 1.0586
hit initial : 188.069 | hit correction : 1.05856
hit initial : 226.486 | hit correction : 1.05843
hit initial : 231.824 | hit correction : 1.05827
hit initial : 230.427 | hit correction : 1.05817
hit initial : 235.448 | hit correction : 1.05816
hit initial : 253.32 | hit correction : 1.0583
hit initial : 283.972 | hit correction : 1.05856
hit initial : 340.328 | hit correction : 1.05886
hit initial : 292.711 | hit correction : 1.05912
 *** Total Charge : 2813.43
 Looking at view: 1 | Number of hits in view: 17
Begin processing the 8th record. run: 5026 subRun: 0 event: 13628 at 12-Dec-2025 03:08:21 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 14.2362
Completed MCParticle Loop
True Neutrino Energy: 19.7439
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 13629 at 12-Dec-2025 03:08:24 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  34
Added hit list length: 34 | Max was set to: -3.40282e+38
True Neutron Energy: 942.055
True Neutron Energy: 940.209
True Electron Energy: 46.7989
Completed MCParticle Loop
True Neutrino Energy: 62.3026
 Size of all hits: 34
hit initial : 37.9915 | hit correction : 1.43382
hit initial : 198.327 | hit correction : 1.43222
hit initial : 185.244 | hit correction : 1.43228
hit initial : 44.7382 | hit correction : 1.43232
hit initial : 80.9988 | hit correction : 1.43999
hit initial : 169.567 | hit correction : 1.43998
hit initial : 119.843 | hit correction : 1.43794
hit initial : 84.4913 | hit correction : 1.43999
hit initial : 347.734 | hit correction : 1.4375
hit initial : 125.065 | hit correction : 1.43749
hit initial : 240.033 | hit correction : 1.43665
hit initial : 120.07 | hit correction : 1.43602
hit initial : 127.535 | hit correction : 1.43499
hit initial : 338.349 | hit correction : 1.43444
hit initial : 235.177 | hit correction : 1.43372
 *** Total Charge : 3525.17
 Looking at view: 0 | Number of hits in view: 15
hit initial : 224.381 | hit correction : 1.44256
hit initial : 92.215 | hit correction : 1.44247
hit initial : 66.8245 | hit correction : 1.44004
hit initial : 155.6 | hit correction : 1.44
hit initial : 54.047 | hit correction : 1.4373
hit initial : 103.887 | hit correction : 1.43992
hit initial : 43.2092 | hit correction : 1.43514
hit initial : 53.1575 | hit correction : 1.43439
hit initial : 93.7353 | hit correction : 1.43245
hit initial : 251.234 | hit correction : 1.43243
hit initial : 130.607 | hit correction : 1.43207
hit initial : 18.9032 | hit correction : 1.43985
hit initial : 74.239 | hit correction : 1.43787
hit initial : 174.74 | hit correction : 1.4378
hit initial : 294.013 | hit correction : 1.43736
hit initial : 334.418 | hit correction : 1.43669
hit initial : 132.555 | hit correction : 1.43588
hit initial : 305.144 | hit correction : 1.43448
hit initial : 208.413 | hit correction : 1.43379
 *** Total Charge : 4038.85
 Looking at view: 1 | Number of hits in view: 19
Begin processing the 10th record. run: 5026 subRun: 0 event: 13630 at 12-Dec-2025 03:08:27 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.569
True Electron Energy: 6.03809
Completed MCParticle Loop
True Neutrino Energy: 9.76245
 Size of all hits: 0
12-Dec-2025 03:08:29 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0670674     0.0869304     0.160524      0.0772094     0.0267377       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000368133   0.00491745     0.0443682    0.000429678    0.0131514       10     
end_path:ana:AnalyzeEvents         0.0665283     0.0818253     0.115606      0.0764444     0.0154486       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.7 MB
  Peak resident set size usage (VmHWM): 1533.13 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 15:00:04 UTC       justIN version: 01.06.00