justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257901.60@dunegpschedd02.fnal.gov

Jobsub ID257901.60@dunegpschedd02.fnal.gov
Workflow ID11024
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 10:59:38
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 09:45:03
From worker nodeHostnamedunegli-7988749-0-fnpc18001.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-05 20:23:13
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T213307Z_gen_004185_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time13h (47693s)
CPU time13h (46914s = 98%)
Max RSS bytes3048067072 (2906 MiB)
Outputting started2025-12-06 09:38:06
Output files
Finished2025-12-06 09:45:03
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

97, 
Output 4: 0.275794, 0.632217, 0.0828967, 0.00909133, 
Output 5: 0.457795, 0.374826, 0.126823, 0.0405563, 
Output 6: 0.0463399, 0.114175, 0.102665, 0.736821, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 41850 at 06-Dec-2025 08:47:23 UTC
Boundary wire vector sizes: 6945, 6825, 6672
minwire 0: 29
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1650, 2149
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0824283, 
Output 1: 0.0580379, 0.0458321, 0.113338, 0.782792, 
Output 2: 0.0625382, 0.253644, 0.658844, 0.0249737, 
Output 3: 0.945711, 0.040511, 0.00610422, 0.00767402, 
Output 4: 0.307585, 0.546061, 0.131747, 0.0146076, 
Output 5: 0.79495, 0.185633, 0.0165194, 0.00289703, 
Output 6: 0.0226264, 0.070409, 0.0799129, 0.827052, 

06-Dec-2025 09:36:46 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:36:46 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c0/2a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T213307Z_gen_004185_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2956.69       4731.29       5687.7        5119.63       885.038        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00476413     0.0154642     0.0383501     0.0110913     0.0107179       10     
reco:pandora:StandardPandora                            2953.69       4726.45       5682.59       5114.76       883.513        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000574323    0.0685234     0.387133      0.0269476     0.110693        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000489744    0.0133099     0.0834002    0.00488693     0.0237326       10     
reco:pandoracalo:Calorimetry                          0.000270911    0.0125035     0.0969286    0.000827576    0.0287673       10     
reco:pandorapid:Chi2ParticleID                        8.7245e-05    0.00109006    0.00890289    0.000184848   0.00260966       10     
reco:cvnmap:CVNMapper                                  0.0595587     0.106234      0.199942      0.0900942     0.0420314       10     
reco:cvneva:CVNEvaluator                                2.01604       3.70772       9.55032       3.56793       2.12122        10     
reco:energyrecnumu:EnergyReco                          0.0138505     0.0846863     0.154149      0.102402      0.0576875       10     
reco:energyrecnue:EnergyReco                          0.00263464     0.0076202     0.0191097    0.00428483    0.00623333       10     
reco:energyrecnc:EnergyReco                            0.0101365     0.0774824     0.160304      0.0581022      0.06308        10     
[art]:TriggerResults:TriggerResultInserter            2.5037e-05    6.10159e-05   0.000159811   4.74045e-05   3.74057e-05      10     
end_path:out1:RootOutput                              1.1391e-05    2.25566e-05   4.8291e-05    1.87255e-05   1.08716e-05      10     
end_path:out1:RootOutput(write)                        0.470294      0.740905       1.2974       0.735551      0.216008        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21776.5 MB
  Peak resident set size usage (VmHWM): 3048.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 241.594 | hit correction : 1.07177
hit initial : 106.014 | hit correction : 1.0718
hit initial : 51.8633 | hit correction : 1.07332
hit initial : 45.2563 | hit correction : 1.07334
hit initial : 42.6945 | hit correction : 1.07394
hit initial : 434.248 | hit correction : 1.07398
hit initial : 44.0133 | hit correction : 1.07398
hit initial : 327.817 | hit correction : 1.07452
hit initial : 308.353 | hit correction : 1.07526
hit initial : 50.1903 | hit correction : 1.07575
hit initial : 64.7366 | hit correction : 1.0759
hit initial : 523.845 | hit correction : 1.07658
hit initial : 68.9199 | hit correction : 1.07662
hit initial : 58.6679 | hit correction : 1.07732
hit initial : 110.187 | hit correction : 1.07734
hit initial : 607.726 | hit correction : 1.07794
hit initial : 92.2993 | hit correction : 1.07837
hit initial : 104.279 | hit correction : 1.07845
hit initial : 431.035 | hit correction : 1.07955
hit initial : 241.219 | hit correction : 1.07976
hit initial : 42.9838 | hit correction : 1.08027
hit initial : 49.9921 | hit correction : 1.08038
 *** Total Charge : 5550.15
 Looking at view: 0 | Number of hits in view: 32
hit initial : 73.1416 | hit correction : 1.05278
hit initial : 212.894 | hit correction : 1.05279
hit initial : 120.114 | hit correction : 1.05283
hit initial : 38.2354 | hit correction : 1.05344
hit initial : 153.973 | hit correction : 1.05348
hit initial : 140.32 | hit correction : 1.05352
hit initial : 21.6734 | hit correction : 1.07111
hit initial : 11.1671 | hit correction : 1.07112
hit initial : 127.51 | hit correction : 1.07175
hit initial : 234.93 | hit correction : 1.07177
hit initial : 59.2087 | hit correction : 1.0718
hit initial : 28.7405 | hit correction : 1.0733
hit initial : 48.5596 | hit correction : 1.07332
hit initial : 30.2606 | hit correction : 1.07345
hit initial : 219.972 | hit correction : 1.07392
hit initial : 114.7 | hit correction : 1.07398
hit initial : 514.785 | hit correction : 1.07402
hit initial : 29.0621 | hit correction : 1.07527
hit initial : 102.841 | hit correction : 1.07564
hit initial : 22.3825 | hit correction : 1.07576
hit initial : 102.782 | hit correction : 1.07624
hit initial : 14.3106 | hit correction : 1.07652
hit initial : 34.1495 | hit correction : 1.07737
hit initial : 618.459 | hit correction : 1.07763
hit initial : 588.725 | hit correction : 1.07866
hit initial : 104.584 | hit correction : 1.07892
hit initial : 16.7012 | hit correction : 1.0796
hit initial : 88.4037 | hit correction : 1.08013
 *** Total Charge : 4149.38
 Looking at view: 1 | Number of hits in view: 28
hit initial : 36.4223 | hit correction : 1.04986
hit initial : 237.084 | hit correction : 1.04992
hit initial : 40.4931 | hit correction : 1.05001
hit initial : 58.8536 | hit correction : 1.05107
hit initial : 39.9969 | hit correction : 1.0527
hit initial : 327.604 | hit correction : 1.05278
hit initial : 108.101 | hit correction : 1.05316
hit initial : 196.589 | hit correction : 1.05344
hit initial : 99.725 | hit correction : 1.05367
hit initial : 188.976 | hit correction : 1.07165
hit initial : 245.442 | hit correction : 1.07186
hit initial : 13.4669 | hit correction : 1.07324
hit initial : 277.44 | hit correction : 1.07339
hit initial : 17.4748 | hit correction : 1.07391
hit initial : 456.354 | hit correction : 1.07394
hit initial : 268.785 | hit correction : 1.0743
hit initial : 311.375 | hit correction : 1.07486
hit initial : 578.362 | hit correction : 1.07582
hit initial : 39.8665 | hit correction : 1.07597
hit initial : 459.961 | hit correction : 1.0771
hit initial : 390.505 | hit correction : 1.07776
hit initial : 363.134 | hit correction : 1.07853
hit initial : 337.807 | hit correction : 1.07918
hit initial : 371.077 | hit correction : 1.0797
hit initial : 222.168 | hit correction : 1.08029
hit initial : 17.9432 | hit correction : 1.08049
 *** Total Charge : 6112.36
 Looking at view: 2 | Number of hits in view: 26
06-Dec-2025 09:38:05 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.125613      0.195081      0.407741      0.149632      0.0942838       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00066248    0.000852928   0.00213871    0.000705275   0.000431088      10     
end_path:ana:AnalyzeEvents         0.124625      0.193965       0.40684      0.148731      0.0942954       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.6 MB
  Peak resident set size usage (VmHWM): 1539.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:09:14 UTC       justIN version: 01.06.00